TW201730627A - Light field display metrology - Google Patents

Abstract

Examples of a light field metrology system for use with a display are disclosed. The light field metrology may capture images of a projected light field, and determine focus depths (or lateral focus positions) for various regions of the light field using the captured images. The determined focus depths (or lateral positions) may then be compared with intended focus depths (or lateral positions), to quantify the imperfections of the display. Based on the measured imperfections, an appropriate error correction may be performed on the light field to correct for the measured imperfections. The display can be an optical display element in a head mounted display, for example, an optical display element capable of generating multiple depth planes or a light field display.

Description

Optical metrology system for detecting light field defects generated by a display [Reciprocal Reference Related Applications]

The present application is filed in the U.S. Patent Application Serial No. 62/250, 925, filed on Nov. 4, 2015, entitled LIGHT FIELD ERROR CORRECTION, citing U.S. Patent Application Serial No. 62/278,779, filed Jan. Priority is LIGHT FIELD ERROR CORRECTION, citing U.S. Patent Application No. 62/250,934, filed on November 4, 2015, entitled AUTOMATED CALIBRATION IMAGE PROJECTION AND CAPTURE FOR DISPLAY CALIBRATION, citing U.S. Patent Application No. 62/ 278,824, filed January 14, 2016, entitled DYNAMIC CALIBRATION OF A DISPLAY BASED ON EYE-TRACKING, and US Patent Application No. 62/278,794, filed on January 14, 2016, entitled CHROMATIC BALANCING A DISPLAY HAVING VARYING CHROMATICITY ACROSS A FIELD OF VIEW. The contents of all U.S. applications are incorporated herein by reference.

The present invention discloses a virtual reality, augmented reality imaging and Visualization systems, and more particularly, metrology systems for measuring, calibrating the optical properties of imaging and visualization systems. At the same time, this paper also reveals a dynamic calibration of virtual reality, extended reality imaging and visualization system based on eye tracking technology.

Modern computing and display technologies have made it easier to develop systems for so-called "virtual reality" and "augmented reality" experiences, where digitally reproducing images or parts of them makes them seem to be viewed Presented to the user for a real situation. Virtual reality (or "VR"), the general context involved is the visual input of the actual real environment rendered by digital or non-transparent virtual image information; augmented reality (or "AR"), generally involved A situation is a visualization that is rendered with digital or virtual image information to enhance the real environment around the user; or a hybrid reality "MR" that involves merging the real world and the virtual world to create a new environment in which objects in the real world Coexist with the objects of the virtual world and interact instantly. It turns out that human visual perception systems are complex, so how to use virtual reality, augmented reality and mixed reality technology to present comfortable, natural feelings, rich virtual maps in other virtual or real world image elements. Image components are extremely challenging. The system and method of the present invention reveals how to solve various difficulties arising from virtual reality, augmented reality, and mixed reality.

Embodiments of the imaging system include projections for the eyes of the viewer a projection device for an image, the image further comprising a light field of a light source of the virtual object, wherein the virtual object is configured to be projected as if it were at one or more desired depths of focus, and a light field measurement for detecting a defect in the light field Device. The light field metering device can be configured to capture one or more images corresponding to a portion of the light field, analyze the one or more captured images to identify one or more perceived depths of focus to correspond to portions The light field focuses depth, creating a depth map based at least in part on the identified depth of focus, and comparing the created depth map to the one or more expected depths of focus. The system can generate a correction method for dynamically calibrating the spatial and/or color defects of the wearable display system.

The details of one or more embodiments described in the subject matter of the specification are set forth in the description and the description below. Other features, aspects, and advantages will be apparent from the description, drawings and claims. It should be noted that the summary of the invention and the following detailed description are not intended to limit or limit the scope of the inventive subject matter.

100‧‧‧Augmented reality scene

110‧‧‧Real environment park setting

120‧‧‧Concrete platform

130‧‧‧Robot statue

140‧‧‧ Cartoon virtual character

200‧‧‧Display system

204‧‧‧ Viewers, wearers, users

208‧‧‧ display

212‧‧‧Frame

216‧‧‧Speaker

220‧‧‧Operating coupling

224‧‧‧Local Processing and Data Modules

228‧‧‧Remote processing module

232‧‧‧Remote Data Repository

236, 240‧‧‧ communication lines

302, 304‧‧‧ eyes

306‧‧‧ Plane

400‧‧‧Display system

405‧‧‧Wave laminate assembly

410‧‧‧ eyes

420, 422, 424, 426, 428‧‧ ‧Band

430, 432, 434, 436‧‧ ‧ features (lens)

438‧‧‧Compensation lens layer

440, 442, 444, 446, 448‧ ‧ image forming devices

450‧‧‧ Controller

452, 454‧‧‧ imaging system

456‧‧‧World

466‧‧‧User input device

460, 462, 464, 466, 468‧ ‧ light extraction optics

500‧‧‧ eye tracking camera

505‧‧‧Light source

Edge of 510‧‧

515‧‧‧Output beam

604‧‧‧Main Plane Waveguide

612‧‧‧Distributed planar waveguide

608, 616‧‧‧Diffractive Optical Elements (DOE)

620‧‧‧Color light source

624‧‧‧ fiber optic

628‧‧‧ hollow tube

632‧‧‧ cantilever

636‧‧‧Drive Electronics

640‧‧‧ wire

644‧‧‧ components

648‧‧‧Mirror

702‧‧‧ calibration pattern (checkerboard checkered pattern)

704‧‧‧Light field imagery

802‧‧‧ expected location

804‧‧‧Detection location

806‧‧‧ line

900‧‧‧ Expected image location

900a‧‧‧Display image location

901‧‧‧ translation vector

902, 904‧‧‧ central location

906‧‧‧Display image

907‧‧‧Rotation

908‧‧‧ center point

910‧‧‧ position

912‧‧ images

913‧‧‧Zoom

914, 922‧‧‧ Expected images

916, 920‧‧‧ display images

917‧‧‧Zoom

918‧‧‧ Expected images

1002‧‧‧Drop depth plane

1004‧‧‧Expected depth plane

1006‧‧‧Rotation axis

1102, 1104‧‧‧ area

1202‧‧‧Location

1204‧‧‧ mode

1206, 1208‧‧‧ luminance values

1302, 1304‧‧‧ intensity distribution

1400‧‧‧ Drawing

1402‧‧‧Red layer

1404‧‧‧Blue layer

1406‧‧‧Green layer

1408‧‧‧ average brightness

1410‧‧‧Maximum brightness (average + maximum error)

1412‧‧‧ Minimum brightness (average-maximum error)

1500‧‧‧Curve

1502‧‧‧ red layer

1504‧‧‧Blue layer

1506‧‧‧Green layer

1600‧‧‧flow chart

1602‧‧‧ Camera calibration

1604‧‧‧ Spatial error

1604a‧‧‧XY Center

1604b‧‧‧Gathering rotation

1604c‧‧‧Gathering zoom

1604d‧‧‧ Spatial mapping

1606‧‧‧Color error

1606a‧‧ ‧ Brightness flattening

1606b‧‧‧Color balance

1702, 1710‧‧‧ objects

1708‧‧‧Distance

1706, 1712‧‧‧ rays

1800‧‧‧Measurement system

1802‧‧‧ display

1804‧‧‧Light

1806‧‧‧ camera

1808‧‧‧ Controller

1900‧‧ images

1902, 1904‧‧‧ areas

1910‧‧Deep map

1912‧‧‧Measured depth of focus

1914‧‧‧Focus of depth

1916, 1918‧‧‧ Area

1920‧‧ depth map

1922‧‧‧ Expected depth position

1924‧‧‧ Depth of focus

2001‧‧‧Program

2002‧‧‧Set focus depth

2004‧‧‧ Capture images of virtual target patterns at depth of focus

2006‧‧‧More depth of focus

2008‧‧‧Change the depth of focus

2010‧‧‧ Identify the depth of focus of the target pattern in different areas

2012‧‧‧Create a depth map based on the identified depth of focus

2014‧‧‧ depth map compared to expected depth of focus

2016‧‧‧Performance error correction

2150‧‧‧ method

2160‧‧‧Get the monitor image

2170‧‧‧Compatible with global transformation parameters

2180‧‧‧Conforming to local transformation parameters

2190‧‧‧Transform parameters are stored in the memory associated with the display

2200‧‧‧ calibration system

2202‧‧‧ display

2204‧‧‧ calibration pattern

2206‧‧‧Light field

2208‧‧‧ camera

2302‧‧‧Feature points

2304‧‧‧Check box

2306‧‧ ‧ pixels

2308‧‧‧ arrow

2400‧‧‧ Process

2402‧‧‧Projected image

2404‧‧‧Image capture of images

2406‧‧‧Computational distortion

2408‧‧‧More locations

2410‧‧‧Projecting images in new locations

2412‧‧‧ Generate distortion map

2414‧‧‧Performance error correction

2500‧‧‧ display

2503i1, 2503i2, 2503i3‧‧‧ wavelength

2505‧‧‧Band

2505a, 2505b‧‧‧ main surface

2507‧‧‧Light input coupling components

2509‧‧‧Light output coupling components

2511‧‧‧Light distribution components

2513‧‧‧wavelength selection filter

2600‧‧‧ Dynamic Calibration System

2602‧‧ points

2602a, 2602b‧‧‧ position

2610‧‧‧Dynamic calibration processor

2700‧‧‧ method

2710‧‧‧ Tracking eyes to determine eye position

2720‧‧‧Access to the calibration of the display based on the determined eye position

2730‧‧‧ Calibration applied to displays to correct for space and/or color defects in displays

2805‧‧‧Processing process

2810‧‧‧ Eye Agent Camera Calibration System

2820‧‧‧ Generate calibration for each eye proxy camera grid point

2830‧‧‧ Can be stored in memory and stored in memory (calibration of dynamic calibration LUT)

2840‧‧‧ Eye Tracking System

2850‧‧‧Extracting calibration from the reservoir based on the perspective of the eye

2860‧‧‧ Calibration applied to the display

2870‧‧‧Users viewing calibrated displays

FIG. 1 is a schematic diagram of a user viewing an augmented reality scene using an augmented reality (AR) device.

2 is a schematic diagram of an implementation of a wearable display system.

3 is a schematic diagram of simulating three-dimensional imaging using multiple depth planes.

4 is a schematic diagram of the implementation of a laminated waveguide for outputting image information to a user.

Fig. 5 is a schematic view showing the implementation of an outgoing beam outputted from a waveguide.

6 is a schematic diagram of an optical system of a waveguide device for coupling light to or coupling light from a waveguide device and a control subsystem for generating a multifocal volume display, image or light field.

Figure 7 is a schematic illustration of the implementation of the distortion produced when the projected calibration pattern of the system is displayed.

8 is a schematic diagram of an implementation of a vector field generated by one or more captured images of a projected offset between a desired position of a point in a projected light field and an actual display position.

Figure 9A is a schematic diagram of the implementation of the in-plane translational space error.

Figure 9B is a schematic diagram of the implementation of the aggregated rotational space error.

Figure 9C is a schematic diagram of the implementation of the aggregated scaling space error.

Figure 9D is a schematic diagram of another implementation of the aggregated scaling space error.

Figure 9E is a schematic diagram of the implementation of the corrected residual spatial error for performing XY translation, rotation, and scaling.

FIG. 10A is a schematic diagram of an implementation of viewing multiple depth planes at different depths.

10B-10D are schematic diagrams showing the implementation of the out-of-plane spatial error when observing the projected depth plane.

FIG. 11 is a schematic diagram of an implementation of capturing images of a test image after projection.

FIG. 12A is a histogram of the intensity generated by the captured image of the projected test image.

Figure 12B is the intensity distribution generated by the captured image of the projected test image. schematic diagram.

Figure 13 is a histogram of the intensity of the mode, median, and mean difference.

FIG. 14A is a schematic diagram showing the intensity of red-green-blue (RGB) generated by the captured image of the projected test image.

Figure 14B is a graphical representation of a graph mapping the maximum color imbalance error.

Figure 15 is a color corrected RGB intensity diagram of a display system having red, green and blue layers of different intensities.

Fig. 16 is a flow chart showing an implementation of performing image correction processing on a display system.

Fig. 17A is a schematic view showing an implementation of observing an object using a normal light field.

Fig. 17B is a schematic view showing the implementation of observing an object using a defective light field.

Figure 18 is a schematic illustration of an implementation of a measurement system for measuring the light field quality of a display.

Figure 19A is a schematic illustration of an embodiment of an image captured by a camera at a particular depth of focus.

Fig. 19B is a schematic view showing the depth of the measurement of the depth of focus.

Figure 19C is a schematic diagram showing the depth of one or more generated images.

Figure 20 is a flow chart showing the implementation of measuring the quality of the virtual target pattern generated by the light field display.

21 is a flow chart of a method implementation for calibrating a display.

Figure 22 is a schematic illustration of the implementation of a calibration system using a calibration pattern.

Figure 23A is a schematic illustration of an implementation of a checkerboard calibration pattern.

Figure 23B is a schematic diagram of the implementation of a single pixel calibration pattern.

Figure 24 is a flow diagram showing the process for calibrating the projected light field.

25A is a top plan view showing an implementation of a display including a waveguide, a light input coupling element, a light re-partitioning element, and a light output coupling element.

Figure 25B is a schematic cross-sectional view showing the display of Figure 25A along axis A-A'.

Figure 26 is an illustration of an embodiment of a dynamic calibration system for a display and may apply calibration to correct the spatial and/or color error of the grid reference position (shown by dots).

27 is a flow chart showing a method of dynamically calibrating a display according to eye tracking.

28 is a flow diagram of the interaction of a factory calibration system and a dynamic calibration system associated with a particular display.

Throughout the drawings, reference numerals may be reused to indicate the correspondence between the reference elements. The drawings are provided to illustrate the embodiments described herein and are not intended to limit the scope of the disclosure.

In order for a three-dimensional (3D) display to produce true depth perception, and more specifically, simulated perception of surface depth, it may be desirable for each point in the field of view of the display to produce an modulating response corresponding to its virtual depth. If the adjustment response of the display point does not correspond to the virtual depth of the point, as determined by the convergence of the binocular depth cues and stereo vision, the human eye may experience adjustmental conflicts, resulting in unstable imaging, eye fatigue, headache, and Almost completely absent without adjustment messages Deficit surface depth.

The virtual reality and augmented reality experience may be provided by a display system having a display in which an image corresponding to multiple depth planes is provided to the viewer. The images on each depth plane may be different (eg, providing a slightly different rendered scene or object) and may be individually focused by the viewer's eyes to help provide the user with the depth needed for eye adjustment The cues focus different image features of the scenes located on different depth planes and/or focus on different image features located on different depth planes. As discussed elsewhere herein, such depth cues provide trusted depth perception.

■3D display

Figure 1 depicts an illustration of an augmented reality scene with certain virtual reality objects and certain actual reality objects being viewed by a person. 1 depicts an augmented reality scenario 100 in which a user using AR technology sees a real environment park setting 110 owner, trees, a background of a building, and a concrete platform 120. In addition to these items, users using AR technology can also feel that he "sees" the robotic statue 130 standing on the concrete platform 120 of the real environment, and a flying hornet cartoon avatar 140, but these components do not exist. real world.

In order for a three-dimensional (3D) display to produce a true depth perception, and more specifically, a simulated perception of surface depth, it may be desirable for each point in the field of view of the display to produce a modulating response corresponding to its virtual depth. If the adjustment response of the display point does not correspond to the virtual depth of the point, determined by the convergence of the binocular depth cues and stereo vision, the human eye may experience adjustment conflicts, resulting in unstable imaging and eye fatigue. Work, headache, and almost no surface depth without adjustment messages.

The experience of virtual reality, augmented reality, and mixed reality can be provided by a display system having a display in which an image corresponding to multiple depth planes is provided to the viewer. The images on each depth plane may be different (eg, provide a slightly different rendered scene or object) and may be individually focused by the viewer's eyes to help provide the user with the eye conditioning needed The depth cues focus different image features of the scenes located on different depth planes and/or focus on different image features located on different depth planes. As discussed elsewhere herein, such depth cues provide trusted depth perception.

2 is a schematic illustration of a wearable display system 200 of an embodiment that can be used to present a VR, AR or MR experience to a display system wearer or viewer 204. Display system 200 includes a display 208, as well as various mechanical and electronic modules and systems for supporting the operation of display 208. The display 208 can be coupled to a frame 212 to form a display system that can be worn by a user, wearer or viewer 204 and to position the display 208 in front of the eyes of the wearer 204. Display 208 can be a light field display. In some embodiments, a speaker 216 is coupled to the frame 212 and to the ear canal position of the user (in some embodiments, another speaker not shown is also located near the ear canal of the user for providing stereo/can Forming sound control). The display 208 is operatively coupled 220, such as by wire or wirelessly to a local data processing module 224, which can be mounted in a variety of configurations, such as fixedly coupled to the frame 212, fixedly Attached to a helmet or cap worn by the user, embedded in the earpiece or otherwise detachably connected to the user 204 (eg, disposed within the backpack structure, disposed within the integrated belt structure).

The local processing and data module 224 can include a hardware processor and digital memory, such as non-volatile memory (e.g., flash memory), both of which can be used to assist in processing, caching, and storing data. The data includes data a from the sensor (which may be, for example, operatively coupled to the frame 212) or otherwise connected to the user 204), such as an image capture device (eg, a camera). , a microphone, an inertial measurement component, an acceleration sensor, a compass, a GPS device, a radio, and/or a gyroscope; and/or b) obtaining and/or processing its use of the remote processing module 228 and/or a remote data repository 232, after such processing or extraction Communication lines 236 and/or 240 are operatively coupled to remote processing module 228 and remote data repository 232, such as via a wired or wireless communication line, such that the remote modules 228, 232 can transfer useful resources to the local Processing and data module 224. In addition, remote processing module 228 and remote data repository 232 can be operatively coupled to each other.

In some embodiments, the remote processing module 228 can include one or more processors configured to analyze and process data and/or image information. For example, video information captured by the image capture device. The video data may be stored locally in the local processing and data module 224 and/or the remote data repository 232. In some embodiments, remote data repository 232 can include a digital data storage device, which can be other network configurations that are through an internet or "cloud" resource architecture. In some implementations In the example, all data storage and all operations are performed in the local processing and data module 224, and are allowed to be completely autonomous through the remote module.

It is challenging for the human visual system to be complex and to provide a true depth of perception. Without being bound by theory, it is believed that due to the combination and adjustment of a convergence, the viewer of the object can perceive that the object is "three-dimensional". The mutual convergent motion between the two eyes (i.e., the rolling motion that causes the eye's line of sight to focus on an object to bring the pupil closer to or away from each other) is closely related to the focusing (or "focusing") of the lens of the eye. Under normal conditions, changing the focus of the lens of the eye, or adjusting the eye, changing the focus from one object to another at different distances will autonomously cause a match within the convergence to change to the same distance. It is called "adjusting the convergence of the convergence". Similarly, under normal conditions, changes in the convergence will trigger a matching change within the adjustment. The display system provides a better match between adjustment and convergence to create a more realistic or comfortable 3D imaging simulation.

3 is a schematic diagram of a method of simulating a three-dimensional imaging using multiple depth planes. Referring to Figure 3, the eyes 302 and 304 are used to adjust objects of various different distances from the eyes 302 and 304 to the z-axis so that the objects are focused. It is assumed that under special adjustment conditions, eyes 302 and 304 will focus on objects at different distances along the Z-axis. Thus, a particular adjustment state can be said to be associated with one of the planes 306 of a particular depth plane having an associated focal length, such that when the eye is in a state of adjustment of the depth plane, an object or portion of the particular depth The object will be focused. In some embodiments, three-dimensional imaging can be simulated to provide different rendered images for each of the eyes 302 and 304, and also provide different rendered images corresponding to each Depth planes. Although separately shown for clarity of illustration, it should be understood that, for example, as the distance along the z-axis increases, the fields of view of eyes 302 and 304 may overlap. Moreover, although shown in plan for ease of illustration, it should be understood that the contour of the depth plane may be curved in physical space such that all features in the depth plane are aligned with the eye in a particular adjusted state. Without being bound by theory, it is believed that a human eye can often be interpreted as a finite depth plane to provide depth perception. Thus, a high-confidence simulation of depth perception can be achieved through the eye to present different images to correspond to each of these limited depth planes.

■Waveguide stacking components

4 is a schematic diagram of a laminated waveguide for outputting image information to a user. A display system 400 includes a waveguide stack, or waveguide stack assembly 405, which can use a plurality of waveguides 420, 422, 424, 426, 428 to provide three-dimensional perception to the eye 410 or brain. In some embodiments, display system 400 can correspond to system 200 of FIG. 2, as shown in perspective in FIG. 4, showing portions of system 200 in greater detail. For example, in some embodiments, waveguide stack assembly 405 can be integrated into display 208 of FIG.

With continued reference to FIG. 4, the waveguide stack assembly 405 can also include a plurality of features 430, 432, 434, 436 between the waveguides. In some embodiments, the feature 430, 432, 434, 436 can be a lens. In some embodiments, the features 430, 432, 434, 436 may not be lenses. Instead, they may be spacers (eg, a cover layer and/or a structure for forming an air gap).

The waveguides 420, 422, 424, 426, 428 and/or the plurality of transparent The mirrors 430, 432, 434, 436 can be configured to transmit image messages to the eye with varying degrees of wavefront curvature or divergence of light. Each planar waveguide can be associated with a particular depth plane and can be configured to output an image message corresponding to the depth plane. Image forming devices 440, 442, 444, 446, 448 can be used to inject image information into waveguides 420, 422, 424, 426, 428, each image forming device being used to diverge incident light across each corresponding waveguide And output toward the eye 410. Light exits the output surface of image forming devices 440, 442, 444, 446, 448 and is injected into the input end edges of corresponding waveguides 420, 422, 424, 426, 428. In some embodiments, a single beam of light (eg, a collimated beam) may be injected into each waveguide to output a complete pseudo-collimated beam region that is directed from a particular angle (and divergence) toward the eye. 410 and corresponds to a depth plane associated with a particular waveguide.

In some embodiments, image forming devices 440, 442, 444, 446, 448 are discrete displays, each of which produces image information for injection into a corresponding waveguide 420, 422, 424, 426, 428. In some other embodiments, image forming devices 440, 442, 444, 446, 448 are outputs of a single multi-display and may be, for example, a delivery catheter through one or more light pipes (eg, fiber optic cables) The image information is sent to each image forming device 440, 442, 444, 446, 448.

A controller 450 controls the operation of the waveguide stack assembly 405 and image forming devices 440, 442, 444, 446, 448. In some embodiments, controller 450 includes programming (eg, instructions in a non-transitory computer readable medium), It regulates the time and provides image information to the waveguides 420, 422, 424, 426, 428. In some embodiments, controller 450 can be a single, unitary device, or a distribution system that is connected by wired or wireless communication conduits. Controller 450 can be part of processing module 224 or 228 (shown in Figure 2) in some embodiments. In some embodiments, the controller can communicate with an inward facing imaging system 452 (eg, a digital camera), an outwardly facing imaging system 454 (eg, a digital camera), and/or a user input device 466. An inward facing imaging system 452 (eg, a digital camera) can be used to capture an image of the eye 410 to, for example, determine the size and/or orientation of the pupil of the eye 410. The outwardly facing imaging system 454 can be used to image a portion of the world 456. The user can enter commands to the controller 450 via the user input device 466 to interact with the display system 400.

The waveguides 420, 422, 424, 426, 428 can be configured to transmit light within each corresponding waveguide through total internal reflection (TIR). Each of the waveguides 420, 422, 424, 426, 428 may be planar or have other shapes (eg, curved) having a main top and bottom surface and an edge extending between the main top and the bottom surface. In the schematic, the waveguides 420, 422, 424, 426, 428 can each include light extraction optics 460, 462, 464, 466, 468 configured to extract light from the waveguide by redirecting light, at each Corresponding waveguide propagation, outputting image information to the eye 410 outside of the waveguide. The extracted light may also be referred to as outcoupling light, and the light extraction optical element may also be referred to as an outcoupling optical element. An extraction beam is output by the waveguide to a position that is the position at which light transmitted within the waveguide impinges on the light redirecting element. The light extraction light The learning elements 460, 462, 464, 466, 468 may, for example, have reflective and/or diffractive optical properties. When it is illustrated that the bottom major surfaces of the waveguides 420, 422, 424, 426, 428 are provided for ease of description and illustration, in some embodiments, the light extraction optical elements 460, 462, 464, 466, 468 can be set At the top and/or bottom major surface, and/or may be disposed directly within the volume of the waveguides 420, 422, 424, 426, 428. In some embodiments, light extraction optical elements 460, 462, 464, 466, 468 can be formed within a layer of material that is attached to a transparent substrate to form the waveguides 420, 422, 424, 426, 428. In other embodiments, the waveguides 420, 422, 424, 426, 428 can be a single wafer material and the light extraction optical elements 460, 462, 464, 466, 468 can be on a surface and/or in the material of the sheet. Formed internally.

With continued reference to FIG. 4, as discussed herein, each of the waveguides 420, 422, 424, 426, 428 is configured to output light to form an image corresponding to a particular depth plane. For example, the waveguide 420 closest to the eye can be configured to transmit a collimated beam of light that, when injected into such a waveguide 420, transmits a collimated beam of light to the eye 410. The collimated beam can be the representation of the optical infinity focal plane. Prior to reaching the eye 410, the next upward waveguide 422 can be configured to transmit collimated light through the first lens 430 (eg, a concave lens). The first lens 430 can be configured to create an incremental convex front curvature, thus causing the eye/brain interpretation light to come from the next upward waveguide 422 as if from the first focal plane, making the infinitely brighter light closer The interior is projected towards the eye 410. Similarly, before reaching the eye 410, the third upward waveguide 424 transmits his output light through the first lens 430 and the second lens. 432. The combined optical power of the first lens 430 and the second lens 432 can be configured to create another incremental convex front curvature, thus causing the eye/brain interpretation light to come from the third waveguide 424 as if from a second focal plane Even an infinite source of light will be projected closer to the inside toward the person than from the source of the next upward waveguide 422.

Other waveguides (e.g., waveguides 426, 428) and lenses (e.g., lenses 434, 436) are also of the same configuration, with the topmost waveguide 428 passing through all of the lenses to transmit the output, all of which will be at the top Between the waveguide 428 and the eye, and the closest focal plane to the person is a representation of the total focal power. To compensate for the stacked lenses 430, 432, 434, 436 when the viewed/interpreted light is from the world 456, the laminated waveguide assembly 405 on the other side will include a compensation lens layer 438 for placement on top of the stack to compensate for the low The total power of the stacked lenses 430, 432, 434, 436 is described below. A waveguide/lens configuration can be utilized to provide more perceptual focus planes. The focus aspects of the light extraction optics 460, 462, 464, 466, 468 and lenses 430, 432, 434, 436 of the waveguides 420, 422, 424, 426, 428 may be static (ie, non-dynamic or electroactive). However, in some alternative embodiments, they may be dynamic and use electrically active properties.

With continued reference to FIG. 4, the configured light extraction optical elements 460, 462, 464, 466, 468 redirect light from the respective waveguides and output collimated light having a moderate amount of divergence or a specific depth plane, Both are associated with the waveguide. As a result, the waveguides have different associated depth planes and have different configurations of light extraction optics that output a different amount of divergent light depending on the associated depth plane. In some embodiments, as discussed herein, light extraction optical element 460, The volume or surface features of 462, 464, 466, 468, which can be configured to output a light source at a particular angle. For example, the light extraction optics 460, 462, 464, 466, 468 can be volume holograms, surface holograms, and/or diffraction gratings. A light extraction optical element, such as a diffraction grating, is disclosed in U.S. Patent Application Serial No. 2015/0178939, filed on Jun. 25, 2015, the entire disclosure of . In some embodiments, features 430, 432, 434, 436 may not be lenses; rather, they may be simple spacers (eg, a cover layer and/or a structure for forming an air gap).

In some embodiments, light extraction optical elements 460, 462, 464, 466, 468 have diffractive features or "diffractive optical elements" (DOEs) that form a diffraction pattern. Preferably, the diffractive optical element has a relatively low diffraction efficiency, so that only a portion of the light of the beam is deflected away from the DOE toward each of the intersections toward the eye 410, and the light of the remaining beams continues to pass through the full Internal reflection moves within the waveguide. The light carrying the image information is thus divided into a number of associated outgoing beams that exit from multiple locations of the waveguide, and these specific quasi-beams bounce within the waveguide and then scatter a fairly uniform pattern from the exit to Eye 410.

In some embodiments, one or more of the DOEs can be switched to an on or off state. They are actively diffracted in the open state, whereas they are not significantly diffracted in the off state. For example, a switchable DOE can include a polymer dispersed liquid crystal layer in which the droplets comprise a diffraction pattern in the host medium, and The droplet index of refraction can be switched to substantially match the refractive index of the host material (in which case the grating does not significantly illuminate the incident light) or the droplet can be switched to an index that does not match the refractive index of the host medium. (In this case, the grating actively diffracts the incident light).

In some embodiments, the number and distribution and/or depth of field of the depth plane may be dynamically changed based on the pupil size and/or orientation of the viewer's eye. In some embodiments, an inward facing imaging system 452 (eg, a digital camera) can be used to capture an image of the eye 410 to determine the size and/or orientation of the pupil of the eye 410. In some embodiments, the inward facing imaging system 452 can be attached to the frame 212 (shown in FIG. 2) and can be in communication with the processing modules 224 and/or 228, which can be processed by the processing modules 224 and/or 228. Image information from the inward facing imaging system 452 to determine, for example, the pupil diameter and/or orientation of the eyes of the user 204.

In some embodiments, an inward facing imaging system 452 (eg, a digital camera) can observe the motion of the user, such as eye movements and facial movements. The inward facing imaging system 452 can be used to capture images of the eye 410 to determine the pupil size and/or orientation of the eye 410. The inward facing imaging system 452 can be used to obtain images to determine the direction in which the user is looking (eg, eye posture) or biometrics for the user (eg, via iris recognition). The images obtained by the inward facing imaging system 452 can be analyzed to determine the user's eye posture and/or mood, which can be determined by the display system 400 as to which audio or visual content should be presented to the user. Display system 400 may also use a sensor such as an inertial measurement element (IMU), accelerometer, gyroscope, etc. to determine a head pose (eg, head position or head orientation). The head gesture can be used alone or in combination with an eye gesture to interact with the audio track and/or current audio content.

In some embodiments, a camera can be used for each eye to determine the pupil size and/or orientation of each eye, respectively, thereby allowing the image information to be presented to each eye to be dynamically clipped for delivery to the eye. In some embodiments, at least one camera can be used for each eye to independently and separately determine the pupil size and/or eye posture of each eye, allowing for dynamic cropping of image information to be presented to each eye. Transfer to the eye. In some other embodiments, only the pupil diameter and/or orientation of the monocular 410 is determined (eg, only a single camera is used for each pair of eyes), and the two eyes of the viewer 204 are assumed to be similar.

For example, the depth of field can be inversely proportional to the change in the size of the viewer's pupil. As a result, as the size of the pupil of the viewer's eye decreases, the depth of field increases, making a plane indistinguishable because the position of the plane exceeds the depth of focus that the eye can discern, and as the size of the pupil decreases and the corresponding depth of field increases More focused. Likewise, the number of spaced apart depth planes used to present different images to the viewer may decrease as the pupil size decreases. For example, without adjusting the adjustment of the eye away from one depth plane to another depth plane, the viewer may not be able to clearly perceive the details of the first depth plane and the second depth plane at one pupil size. However, the two depth planes can simultaneously focus on the user at the same pupil size without changing the adjustment.

In some embodiments, the display system can be based on the certainty of the pupil size and/or orientation, or based on receiving an electrical message of a particular pupil size and/or orientation. The number indicates to change the number of waveguides that receive image information. For example, if the user's eyes are unable to distinguish between two depth planes associated with the two waveguides, the controller 450 can be configured or programmed to stop providing image information to one of the waveguides. This can advantageously reduce the processing burden on the system, thereby increasing the responsiveness of the system. In embodiments where the DOE for the waveguide can be switched between on and off states, the DOE can be switched to the off state when the waveguide receives image information.

In some embodiments, it may be desirable for the diameter of the exiting beam to satisfy a condition having a diameter that is less than the diameter of the observer's eye. However, in view of the variability of the viewer's pupil size, meeting this condition can be challenging. In some embodiments, the size of the exiting beam is varied by reacting to the deterministic size of the observer's pupil size such that the condition is met over a wide range of pupil sizes. For example, as the pupil size decreases, the size of the exiting beam can also be reduced. In some embodiments, a variable aperture can be used to vary the exit beam size.

Display system 400 can include an outward facing imaging system 454 (eg, a digital camera) that images a portion of world 456. The portion of the world 456 can be referred to as a field of view (FOV), and the imaging system 454 is sometimes referred to as an FOV camera. The entire area viewed or imaged by the viewer 204 may be referred to as a region of interest (FOR). FOR may include a 4π sphericity around the solid angle of display system 400. In some embodiments of display system 400, FOR can include substantially all of the solid angles around user 204 of display system 400, as user 204 can move their head and eyes to see objects surrounding the user ( On the front, back, top, bottom or side of the user). From an outward facing imaging system 454 The acquired image can be used to track gestures made by the user (eg, hand or finger gestures), detect objects in the world 456, objects in front of the user, and the like.

Display system 400 can include user input device 466 through which a user can enter commands to interact with display system 400. For example, user input device 466 can include a touchpad, a touch screen, a joystick, a multi-degree of freedom (DOF) controller, a capacitive sensing device, a game controller, a keyboard, a mouse, a D-pad, a stick, A haptic device, a totem (eg, having functionality such as a virtual user input device), and the like. In some cases, a user may use a finger (eg, a thumb) to press or slide on a touch input device to provide input to display system 400 (eg, to provide user input to a user interface provided by display system 400). The user can hold the user input device 466 during use of the display system 400. And the user input device 466 can communicate with the display system 400 in a wired or wireless manner.

Figure 5 is an embodiment of an outgoing beam output from a waveguide. Referring to the waveguide as shown, it should be understood that other waveguides within the laminated waveguide assembly 405 have similar functions, with the laminated waveguide assembly 405 including a plurality of waveguides. When light source 505 is injected into input edge 510 of waveguide 420, it is transferred within waveguide 420 using TIR. Light source 505 at the point will strike on DOE 460, and a portion of the source will exit the waveguide as exit beam 515. The exiting beams 515 are substantially parallel, as shown, they can also be redirected at an angle to the eye 410 (e.g., to form a divergent exit beam) depending on the depth plane associated with the waveguide 420. It should be understood that substantially parallel outgoing beams can be used to present There is now a waveguide for a light extraction optical element that externally couples light to form an image disposed on a depth plane from the eye 410 to a large distance (eg, light infinity). Other waveguides or other sets of light extraction optics can output a more divergent exit beam pattern, requiring the eye 410 to adjust the diverging exit beam image to a closer distance, focusing it on the retina, and the brain will Interpreted as if the light came from a distance closer to the eye 410 than the infinity of the light.

6 is another embodiment of a display system 400 that includes a waveguide device for coupling light to or from an optical coupler subsystem that couples light from the waveguide device and a control subsystem. Display system 400 can be used to generate a multi-focal moment volume, image or light field. Display system 400 can include one or more primary planar waveguides 604 (only one shown in FIG. 6) and one or more DOEs 608 associated with each of at least a few of the primary waveguides 604. The planar waveguide 604 can be similar to the waveguides 420, 422, 424, 426, 428 discussed in FIG. The optical system may use a distributed waveguide device to propagate light along a first axis (vertical or Y-axis of Figure 6) and to expand the effective exit pupil of light along a first axis (eg, the Y-axis). The distributed waveguide device may, for example, include a distributed planar waveguide 612 and at least one DOE 616 (shown by double dashed lines) associated with the distributed planar waveguide 612. The distributed planar waveguide 612 can be similar or identical to the primary planar waveguide 604 in at least some aspects, having a different orientation thereto. Likewise, at least one DOE 616 can be similar or identical to DOE 608 in at least some aspects. For example, distributed planar waveguide 612 and/or DOE 616 may be composed of the same material as main planar waveguide 604 and/or DOE 608, respectively. The optical system shown in Figure 6 can be integrated into the wearable Figure 2 Wearing the display system 200.

The propagated light and the exit pupil extended light are optically coupled from the distributed waveguide to one or more of the primary planar waveguides 604. The primary planar waveguide 604 is along the second axis and is orthogonal to the first axis (eg, horizontal or X-axis see FIG. 6). Notably, the second axis can be a non-orthogonal axis relative to the first axis. The primary planar waveguide 604 extends the effective exit path of light along a second axis (eg, the X-axis). For example, the distributed planar waveguide 612 can propagate and spread light along the vertical or Y-axis and pass the light to the primary planar waveguide 604, which in turn propagates and spreads light along the horizontal or X-axis.

Display system 400 can include one or more colored light sources (eg, red, green, and blue lasers) 620 that can be optically coupled into the proximal end of a single mode fiber 624. The end portion of the optical fiber 624 may be connected or received in series by a hollow tube 628 of piezoelectric material, and the end protrudes from the hollow tube 628 to form a flexible cantilever 632 that is fixedly swingable. Piezoelectric tube 628 can be associated with four quadrant electrodes (not shown). For example, the electrodes can be plated on the outer, outer or outer circumference or diameter of the hollow tube 628. The core electrode (not shown) is also located in the core, center, inner circumference or inner diameter of the tube 628.

The drive electronics 636, for example, drive the corresponding two electrodes via wires 640 to bend the piezoelectric tubes 628 on two separate axes. The fiber 624 protrudes from the four end tips to have a mechanical resonance mode. The resonant frequency may depend on the diameter, length, and material properties of the fiber 624. The piezoelectric tube 628 is vibrated in the vicinity of the first mode of mechanical resonance of the fiber cantilever 632, causing the fiber cantilever 632 to vibrate, and a large off angle can be scanned.

The two-dimensional (2-D) scanned region is filled and the tip end of the fiber cantilever 632 is biaxially scanned by intense resonant vibration on both axes. The light emitted from the fiber cantilever 632 forms an image by modulating the intensity of the light source 620 in synchronization with the scanning of the fiber cantilever 632. A description of such an arrangement is provided in U.S. Patent Application Serial No. 2014/0003762, the entire content of which is incorporated herein by reference.

Element 644 of the optocoupler subsystem collimates light emerging from scanning fiber cantilever 632. The collimated light is reflected by mirror 648 into a narrowly distributed planar waveguide 612 comprising at least one diffractive optical element (DOE) 616. The collimated light propagates vertically (through the view of FIG. 6) along the distribution plane waveguide 612 by total internal reflection and thus crosses the DOE 616 in turn. More preferably, DOE 616 has low diffraction efficiency. This causes a portion (e.g., 10%) of the light to be diffracted toward the edge of the larger principal plane waveguide 604 at the DOE 616 at each intersection, and a portion of the light continues to pass through the TIR along its original trajectory along the distribution plane waveguide 612. The length of the move.

At each intersection that intersects the DOE 616, additional light is diffracted toward the entrance of the main waveguide 612. By splitting the incident light into a plurality of output coupling sets, the exit pupil of the light is vertically expanded by the DOE 616 in the distributed planar waveguide 612. The vertically extended light coupled from the distribution plane waveguide 612 enters the edge of the main planar waveguide 604.

Light entering the primary planar waveguide 604 propagates along the primary planar waveguide 604 via the TIR level (associated with the view of Figure 6). When light propagates horizontally along the length of at least a portion of the main planar waveguide 604 via TIR, the light is at multiple points with the DOE 608 Cross. The DOE 608 can advantageously be designed or configured to have a phase distribution that is the sum of a linear diffraction pattern and a radially symmetric diffraction pattern to produce deflection and focus of the light. The DOE 608 can advantageously have low diffraction efficiency (e.g., 10%) such that only a portion of the beam of light will deflect with the DOE 608 of each intersection toward the eye of the view, while the remaining light continues to propagate through the main plane waveguide 604 through the TIR.

At each intersection between the propagating light and the DOE 608, a portion of the light is diffracted toward the adjacent faces of the main planar waveguide 604, allowing light to exit the TIR and exit from the surface of the main planar waveguide 604. In some embodiments, the radially symmetric diffraction pattern of DOE 608 has a focus level of diffracted light, and the optical wavefront (eg, imparting curvature) shape of the two individual beams directs the beam at a symmetrical angle to the designed focus level. .

Thus, these different paths can pass light through multiple DOEs 608 at different angles, the focus is horizontally coupled out of the main planar waveguide 604, and/or produces a different fill pattern in the exit pupil. Different fill patterns at the exit pupil can advantageously be used to create a light field display having multiple depth planes. Each layer in the waveguide assembly or a set of layers (eg, 3 layers) in the stack can be used to produce a corresponding color (eg, red, blue, green). Thus, for example, a first set of three adjacent layers can be employed to produce red, blue, and green light at a first depth of focus, respectively. A second set of three adjacent layers can be used to produce red, blue, and green light, respectively, at the second depth of focus. Multiple sets can be employed to produce a complete 3D or 4D color image light field with various depths of focus.

■Other components of the AR system

In many embodiments, the AR system can include other components than the wearable display system 200 (or optical system 400). An AR device can, for example, include one or more haptic devices or components. A haptic device or component can be used to operate and provide tactile sensation to the user. For example, when touching virtual content (eg, virtual objects, virtual tools, other virtual constructs), the haptic device or component can provide a tactile perception of pressure and/or texture. Haptic perception can reflect the physical response presented by a virtual object, or can reflect the sensation of an imaginary object or character (eg, a dragon) presented by the virtual content. In some embodiments, the haptic device or component can be worn by a user (eg, a user wearable glove). In some embodiments, the haptic device or component can be held by a user.

The AR system can, for example, include one or more physical objects that can be manipulated by a user to allow for input or interaction with the AR system. These physical objects are referred to herein as totems. Some totems can take the form of inanimate objects, such as a piece of metal or plastic, walls, or the surface of a table. Alternatively, some totems may take the form of an animated entity, such as a user's hand. As described herein, a totem may not actually have any physical input structure (eg, keys, triggers, joysticks, trackballs, rocker switches). Instead, the totem can simply provide a solid surface, and the AR system can present the user interface such that the user appears to be on one or more surfaces of the totem. For example, an AR system can make an image of a computer keyboard and trackpad appear to be on one or more surfaces of a totem. For example, the AR system can present a virtual computer keyboard and virtual touchpad on the surface of a thin rectangular aluminum plate that acts as a totem. The rectangular plate itself does not have any physical keys or touchpads or sensors. However, AR The system can detect user interactions or interactions or touches with rectangular panels as selected or entered via a virtual keyboard and/or virtual trackpad.

Examples of haptic devices and totems that can be used with the AR devices, HMDs, and display systems of the present invention are described in U.S. Patent Publication No. 2015/0016777, the entire disclosure of which is incorporated herein by reference.

■Example of performing error correction on the display system

As noted above, a display system can include a stacked waveguide assembly, such as shown in the figures. Referring to Figures 4-6, there are multiple display layers of substrate material with a diffraction grating to redirect light that produces a digitized light field that impinges on the eye. In some embodiments, the waveguide assembly includes a substrate layer for each color, each depth. For example, a dual depth planar RGB display can have a total of six waveguide layers. The display system can be an embodiment of the wearable display system 200.

In the stacked waveguide assembly, there is a possibility that a series of potential phenomena causing deterioration of image quality may be introduced. This can include ghosting (multiple images), distortion, misalignment (between color or depth), and color intensity variations on the field of view. In addition, certain types of artifacts that may occur under other types of conditions, such as when illuminated with laser light relative to the LED (eg, spots, strips, Newtonian stripes), or when the density of the output coupled beam is less than At a certain amount (for example, the wavefront is sparse, it can be perceived as if it passes through a screen door or fence).

Due to defects in the optics of the light field display, the ideal three-dimensional grid becomes distorted in the render engine when displayed through the optics. In order to identify and correct the distortion between the desired image and the actually displayed image, A display system can be used to project a calibration pattern, such as a checkerboard pattern.

FIG. 7 is an embodiment of distortion that may occur when the pattern 702 is calibrated via projection of the display system. The calibration pattern 702 can be adapted to perform any type of pattern of spatial or color calibration (eg, a checkerboard pattern comprising a plurality of checkerboard tiles). The calibration pattern 702 can include any type of test or calibration pattern, such as a geometric pattern or a random pattern. Projection calibration pattern 702 produces a generated light field image 704. Distortion present in image 704 may include spatial distortion (eg, when the visible pixel is not within its intended field of view) and color distortion (eg, when the color value of the visible pixel is different than desired). For example, checkerboard pattern 702 may be offset from their intended position in image 704 (eg, spatial error). Additionally, some of the checkerboard tiles in image 704 may appear in other colors (eg, purple (eg, color difference)) instead of the checkerboard squares displayed in black and white. The display error can be measured using a light field metrology system that can include an image of a digital camera positioned to acquire a calibration pattern projected by the display. In some embodiments, multiple images of the calibration image corresponding to the offset to different locations may be retrieved to obtain a more detailed texture of the desired location and the actual location. Digital cameras can be used to focus at different depths of focus to determine which depth of image (eg, features on the displayed calibration pattern) displayed on different regions is focused on.

According to some embodiments, capturing multiple images at different depths of focus to determine the depth of different regions of the displayed image is described in more detail below in connection with Figures 17-20. Different types of calibration patterns that can be used in various embodiments are described in more detail below in conjunction with FIGS. 22 through 24.

■ Spatial error

Spatial errors can include several different manifestations. For example, spatial misalignment includes translation or rotation of the display layer. Spatial errors can also involve nonlinear spatial distortion that varies over the field of view (FOV) of the depth plane of the display.

The spatial error can be a sign of mechanical or optical defects within the display system. By clarifying the measured spatial error, metrics of the optomechanical quality of the quantized system and metrics of the proposed improved method can be derived. For example, a spatial error indicative of depth plane rotation may suggest that the display mechanically rotate relative to the desired position. The planar scaling of each color can be suggested as a lens system that does not adequately achromatic.

In order to identify spatial errors, the light field measurement system includes an image capture device, such as a digital camera, that can be used to capture one or more images projected by the display system (eg, a projection of a calibration pattern) and produce an image representing the actual display and The vector field of the deviation of the desired image. The vector field may be a three-dimensional vector field comprising an in-plane deviation in the xy plane of the display and an out-of-plane deviation in the z-direction (depth), or a two-dimensional vector field that is only offset in the xy plane. In some embodiments, a vector field can be generated by each depth plane or each color plane of the display system. In some embodiments, the depth can be measured in diopter, indicating the back focal length of the layer in meters.

8 is an implementation of a vector field generated from one or more captured images of a mapping offset between an expected position of a point in a projected light field and its actual display position. The points in the projected light field may correspond to features in the calibration image (eg, calibrating the center and corner of the checkerboard square). Each vector in the vector field is represented in the light field Distortion between the expected position in the middle and its corresponding actual position. In this embodiment, the distortion vector field is 2D. In the illustrated vector field, the first color and marker type (eg, "O" 802 for the intended location) is used to mark the intended location of a feature, while the second color is used (eg, "X" 804 is used for detection). position). Each pair of respective desired and displayed positions is connected by line 806, which may include an arrow indicating the direction in which correction is needed to correct the detected display position to the desired position.

Using vector fields, local or global distortion information can be extracted (eg, in-plane translation, aggregate scaling, aggregate rotation, mean pixel curvature, or diopter error, as described below). For example, a distortion map can be generated from the determined vector field. The distortion map can be used to analyze the distribution of pixel position error values (eg, vector magnitude) over the generated vector field. The distortion map may be a histogram of the frequency at which the pixel position error is displayed (eg, the pixel position error magnitude is plotted for the frequency at which the error magnitude appears in the vector field). Other types of graphs can be used to analyze other properties of the vector field (eg, distortion direction).

Spatial errors can be roughly divided into in-plane and out-of-plane spatial errors. The in-plane spatial error refers to the spatial error along a particular depth plane (e.g., according to the xy plane of the coordinate system shown in Figure 6) at a particular depth (measured on the z-axis). A vector field (eg, as shown in FIG. 8) may derive one or more metrics for spatial errors of different categories. Each of these metrics can be defined on a substrate of each layer (eg, for each individual display layer corresponding to a particular combination of color and depth (eg, a red-3 diopter display layer, a green-1 diopter display) Layers, etc.) or each display substrate (eg, quantify the overall fidelity of the display in concise parameters) degree).

■planar space error

In some embodiments, the in-plane spatial error can be divided into a plurality of different elements, each element corresponding to a different type of error. These components may include translational errors, rotational errors, scaling errors, or nonlinear spatial errors. Each of these error elements can be corrected individually or sequentially.

■In-plane translation error

Figure 9A is an embodiment of an in-plane (xy) translational spatial error (also referred to as the xy center). The xy translation error refers to the x and/or y pixels of the display image center of the display layer being offset from their intended position, and the xy translation error is intended to inform the mechanical or display alignment. In FIG. 9A, the expected image position 900 (in this embodiment, represented by a red rectangle) is converted to display image position 900a (represented by a green shape having a non-linear line). Using the center position 902 of the display image position 900a and the center position 904 of the expected image position 900, one or more shifts (along the determined translation vector 901) are performed to correct an xy translation error such that the displayed center Position 902 is aligned with expected center position 904 (mechanical alignment by display, software correction of displayed image, or a combination of both). A measure of one or more of the measured xy translational spatial errors may include a translational error measured on a substrate of each layer, the center of the measurement layer corresponding to an expected or reference location (eg, the optical axis of the display) or at each The displays measure the maximum translational offset, which identifies the maximum translation between any two display layers to quantify the overall translational registration.

■ Aggregate rotation error

Figure 9B is an embodiment of an aggregated rotational spatial error. Aggregate rotation refers to the expected position of the displayed image image near its center relative to the image. Although spatial distortion cannot always be fully described using simple affine rotation, an aggregate rotation metric can be used to provide a rotation angle by which pixel position errors (between display and expected image position) are minimized . Aggregate rotation metrics are intended to inform the machine or display alignment. As shown in FIG. 9B, the displayed image 906 can be rotated about the center point 908, with the specified amount of rotation 907 to the position 910 corresponding to the expected position to correct the focus rotation. (either by mechanical alignment of the display, by software correction of the displayed image, or both). The reported metrics may include the rotational error measured at each layer, identifying the relationship of the measured orientation to the expected or reference orientation (eg, relative to the horizontal axis of the display) and the maximum rotational offset measured at each display, The maximum translation is recognized between any two display layers to quantify the overall translational registration.

■ Aggregation scaling error

Figure 9C is an embodiment of an aggregated scaling spatial error. The aggregate zoom indicator shows the overall zoom factor of the image near its center relative to the desired image. Although spatial distortion cannot be fully described with simple affine scaling, the aggregate scaling measurement can indicate a scaling factor by which pixel position errors are minimized. The aggregate scaling metric is intended to inform the optical design or display alignment. As shown in FIG. 9C, the size of the displayed image 912 can be scaled with the specified amount of zoom 913 to match the size of the desired image 914 to correct the aggregated zoom space error. Report metrics for aggregate scaling can include scaling errors, measurements per layer, and their measurement images (eg, The physical target in the calibration settings and the maximum zoom offset measured by each display, which indicates the maximum zoom between any two display layers to quantify the overall translational registration.

Figure 9D is another embodiment of an aggregated scaling spatial error. The display image 916 appears to be smaller than the expected image 918. To correct the scaling error, display image 916 is scaled up by zoom amount 917 to match the size of expected image 918.

■Pixel distortion error

Figure 9E is an embodiment of the residual spatial error after correction for xy translation, rotation, and scaling has been performed. The residual error (also known as pixel warping or spatial mapping) indicates the average residual Euclidean pixel position error in the xy spatial distortion distribution after translation, rotation, and scaling (eg, as shown in Figures 9A-9D) imparting nonlinearity to the display system Or a measure of non-affine distortion characteristics and is used to inform display design and quality control. The reported metric of pixel distortion can include average pixel distortion (MPW), measuring each layer, representing the xy average residual Euclidean pixel position error after translation, rotation, and scaling, reference ideal grid and maximum average pixel distortion (maximum MPW) , which represents the maximum MPW value between display layers to quantify the overall distortion. In some embodiments, the display image 920 can be aligned with the desired image 922 by using the spatial mapping performed by the processing module (eg, module 224 or 228) to correct the remaining pixel distortion.

■Out-of-plane spatial error

Digital light field display systems, such as those shown in Figures 4-6, are capable of producing depth planes that appear to be at different depths (in the z-direction) from the observer (see, For example, Figure 3). In some embodiments, the depth plane corresponds to a plane that appears to be at a different distance from the viewer. As is common in optics, rather than the distance of the depth plane from the display, the inverse distance measured in diopter (m-1) can be used to reference different depth planes. For example, the display can have two depth planes at depths of 3 diopters (1/3 m) and 1 diopters (1 m). Due to defects in the display system, the diopter profile through the depth plane will not be as expected. For example, an image displayed on a depth layer may have a diopter profile of an incorrect distance or a zoom across the display FOV.

Out-of-plane spatial error (also known as diopter error) is a measure of the diopter (depth) error of the depth plane to inform optical, mechanical, and waveguide alignment or errors in the design. The metric of the reported diopter error may include indicating the amount of error between the expected depth plane and the measured depth plane, and the maximum diopter error, indicating the maximum depth error among the depth planes.

Figure 10A is an embodiment of viewing multiple depth planes at different depths. In an embodiment, three different depth planes are exposed, but the display system may contain more or fewer depth planes. Additionally, each depth plane may correspond to a plurality of waveguide layers (eg, RGB color layers).

10B-10D are embodiments of types of out-of-plane spatial errors that may occur when viewing the projected depth plane shown in FIG. 10A. For example, the projected depth plane can be displaced to a different depth such that it appears larger or smaller than the expected depth (Fig. 10B). The depth plane in which the direction deviates presents a (large amount) volume rotation from the expected depth (Fig. 10C). Depth planes can exhibit uneven contours of grating defects Sex (Fig. 10D). The depth plane can exhibit the combination of errors presented in Figures 10B-10D.

Figure 10E is another embodiment of the out-of-plane spatial error. The misaligned projected depth plane 1002 is related to the expected depth plane 1004. In an embodiment, the misalignment includes depth plane rotation. To correct the out-of-plane spatial error, an axis of rotation 1006 can be defined and rotated about the identified axis of rotation 1006 on the projected depth plane 1002 such that the projected depth plane 1002 is substantially aligned with the intended depth plane 1004. Although the axis of rotation 1006 is taken as an axis parallel to the intended depth plane 1004 (eg, a vertical axis), it should be understood that the axis of rotation can be in any direction.

Although the diopter error is different from the in-plane spatial error associated with in-plane distortion, the diopter error may potentially affect the in-plane spatial error, such as due to incorrect assumptions in pixel depth, which will result in view-dependent spatial distortion. For example, for a defect depth plane having a depth region that is different from the expected depth, the pixels may be unevenly offset relative to the viewer position, resulting in a distorted image distortion.

In some embodiments, the present invention describes that correction techniques for in-plane spatial errors (eg, xy center, aggregate scaling, aggregate rotation, and spatial mapping) can be extended to three dimensions. For example, Centration can be performed in three dimensions by discriminating the position of the center point of the display plane on the xyz coordinate system and moving the plane (eg, along the x, y, and z axes) such that the center point and the expected position Align.

■ Spatial error quantization based on distortion vector field

As shown herein with reference to Figure 8, a multi-dimensional (e.g., 2D or 3D) distortion vector field can be generated using the displacement of the measured image features from the expected location to the display location. The distortion vector field can be calculated with each layer of a multi-layer display (eg, a display including stacked waveguide assemblies 405). The distortion vector field can be used to capture and characterize the light field distortion projected by the display. For example, a vector analysis operation can be performed on the distortion vector field to determine some spatial errors. The light field metrology system can calculate such vector operations as part of an image analysis obtained by a metrology camera (eg, a digital camera or a light field camera) for a calibration pattern (eg, a checkerboard) for display projection. Such vector analysis techniques are not limited to light field displays and can be applied to the calibration of any multi-dimensional metric or any type of display.

Given a multi-dimensional distortion vector field, the curl of the vector field can be calculated to determine the local rotation. The curl mean over the area in the display FOV provides a measure of the aggregate rotation error in that area. In discrete depth plane embodiments of the light field display, the curl calculation of the distortion vector field can provide information about the in-plane or out-of-plane rotation of the layer.

The divergence of the distortion vector field can be calculated to determine the scaling error. In embodiments having multiple layers (e.g., RGB color layers), a full color image can be produced at each depth, which can be used to provide information related to zoom calibration.

Vector integral theorems (eg, Stokes theorem or divergence theorem (Gaussian theorem)) can be applied to the distortion vector field to calculate the curl and divergence of the vector field over the region in the FOV of the display (eg, the region's Rotate or gather zoom). The Euclidean average of the vectors in the distortion vector field can be calculated to obtain Information relating to the non-affine spatial transformation introduced by the distortion. .

■Quantification of color error

When the color value of the visible pixel is different from the expected color value, a color difference will occur. To evaluate the color difference, the display system can be used to project a calibration image. The calibration image can be the same calibration image used to perform spatial error correction, or can be a different calibration image. For example, the calibration image may include a solid image of a particular color (eg, red) at a particular luminance level (eg, maximum brightness). An image capture device (for example, one or more cameras) can be used to capture the output from the projected calibration image. Figure 11 is an embodiment of a captured image of a projected image after projection. Although the calibration image can be constant in the luminance level of the entire image, the brightness of the displayed calibration image will vary across the field of view of the display due to the presence of color errors. For example, certain regions 1102 of the captured image may be of high luminance level, while other regions 1104 may exhibit lower luminance levels resulting in dark areas or dark bands on the display. In some embodiments, the calibration image can include a color calibration pattern instead of a single color.

In some embodiments of the display, the observed luminance topology may depend on the wavelength. For example, for red, green, and blue, there may be different luminance variations such that the projected image appears to be different from the desired color (representing an imbalance between the red, green, and blue components). For example, a projected white calibration image may appear purple, with a green luminance level lower than the red and blue luminance levels. Additionally, the luminance change may also be based on the observer position (eg, the camera is moved, the dark band at region 1102 appears to be moving to a different location in the FOV). this phenomenon It may be challenging to maintain color uniformity and white balance on the FOV (especially when luminance or color balance depends on the viewer's position) and ultimately affect the color accuracy of the content being displayed.

Each display layer in the display system is associated with chrominance characteristics, measured color and luminance characteristics, measured brightness or intensity. Therefore, the chromatic aberration can be roughly classified into a luminance flatness error and a chromaticity uniformity error.

■ Brightness flatness

A measure of luminance flatness can be used to quantify how much luminance variation each display layer exhibits. Generally, in stacked waveguide assemblies, as each display layer is produced by a different waveguide in the stack (see, for example, waveguide assembly 405 in Figure 4), different display layers may have different luminance variations across the field of view.

Measuring the brightness flatness of the display layer, which can be used to capture some or all of the pixels of the image to determine the luminance value (also known as the intensity value). Although the present invention primarily relates to luminance values for pixels, in other embodiments, luminance values may be determined for regions of multiple pixels (eg, N x M pixel grids) rather than for a single pixel. In some embodiments, each determined luminance value is assigned a luminance unit of one or more luminance value ranges. For example, for an 8-bit color display system, 256 colors corresponding to 8 bits can be used.

From the determined luminance values, a plurality of metrics of luminance flatness can be calculated by the metering system. For example, a mode that represents the most common pixel luminance value across the display area can be calculated. From this mode, it can be determined that the half pixel density (HPPR) indicates that the luminance range or multiple luminance bins (bin) are adjacent to cover 50% of the pixels. Density mode. A small HPPR indicates that the brightness of the display layer is uniform across the display entity. The luminance value can also be referred to as an intensity value. For the purposes of this patent, luminance and intensity terms can be used interchangeably.

Figure 12A is an intensity histogram produced by a captured image of the projected calibration image (e.g., as shown in Figure 11). The intensity histogram is plotted as a luminance value for the frequency of occurrences in the captured image (eg, the number of pixels having a luminance value). This mode will have the highest luminance value in the image (for example, at position 1202).

Figure 12B is an intensity distribution diagram of the captured image of the projected calibration image. In this intensity profile, this pattern will appear in the luminance value 1204 (and in this embodiment has 236 colors). From this mode, a deviation range centered on mode 1204, which represents a range between luminance value 1206 and luminance value 1208, is determined to be a pixel density that covers 50% of the image. The HPPR is determined based on the calculated range of deviations (eg, the difference between the luminance value 1206 and the luminance value 1208).

For an ideal display layer, the resulting input illumination (eg, HPPR = 0), the intensity values of the field are the same. Deviation from this ideal characteristic will reveal the pixel intensity value distribution (range) away from the mode value. The distribution of the HPPR measurement attempt will use the metric to indicate the away mode. A substantially uniform luminance can have a small HPPR, for example, a HPPR that is smaller than the range of luminance values of the mode (e.g., 255 of an 8-bit color). For example, a substantially uniform (eg, flat) luminance display may be less than about 10%, less than about 5%, less than about 1%, or less than about 0.1% HPPR ratio over the total color range.

HPPR can be viewed as a variation of a quartile that is used to measure the distribution away from the median rather than away from the pattern. The median value of the pixel intensity value is not directly related to the desired plane-intensity response of the display layer. Figure 13 is an embodiment of intensity histograms 1302, 1304 for differences in mode, median, and mean (μ). The median values of the two distributions 1302, 1304 are the same in this embodiment. The two distributions 1302, 1304 have a standard deviation σ of 0.8 and 2, respectively. As shown in Figure 13, if the intensity distribution of the image is near normal (eg, intensity distribution 1302), the mode, median, and average will be very similar. Conversely, if the intensity distribution is not close to the normal distribution (eg, intensity distribution 1304), the mode, median, and average of the intensity distribution are substantially different from each other.

For each display layer of the display, luminance flattening is to reduce the variation in luminance across the display field of view. Since the luminance intensity of a pixel generally cannot exceed its maximum value, luminance flattening is generally a step of reducing the overall luminance, in which the luminance of the pixel is compressed in a specific section of the layer such that the luminance of the layer is as flat as possible.

For example, luminance flattening can be performed such that the pixel luminance has a maximum at the pixel luminance of the lowest luminance value, reducing the luminance of the display layer to a substantially minimum luminance. Alternatively, the pixel luminance may be configured to be a maximum at the selected luminance value, the maximum value being greater than the pixel luminance value of the lowest luminance value. This may not minimize the overall luminance because pixels below the selected luminance value are still present and the remaining luminance present is not uniform. In some embodiments, reducing the luminance value of a pixel or group of pixels includes identifying a luminance value by which to reduce a pixel or group of pixels. In other embodiments, reducing the luminance value of the pixel or group of pixels includes discriminating the scaling A factor that reduces the luminance value of a pixel or group of pixels to a minimum luminance value or a threshold luminance value by a scaling factor.

In some embodiments, if the initial luminance flattening of the display layer is good (eg, HPPR is below a threshold), the luminance value can be reduced to a minimum luminance value to provide a flat luminance field. On the other hand, if the luminance flatness is poor (eg, HPPR exceeds a threshold) or a low minimum luminance value (eg, the minimum threshold is not reached), the selected maximum luminance value may be selected. The luminance flattening can be performed in a software module (eg, processing module 224, 228).

When the execution luminance flattening of each display layer is not the same, the luminance level is lowered. However, different luminance levels for different layers in the same color group (eg, RGB layer groups) will result in a loss of white balance, which can be handled by correcting the chromaticity uniformity of the display.

■ Chromaticity consistency

Chromaticity generally refers to the color component of a display that is independent of luminance. As noted above, the display layer in the display system can include a red display layer, a green display layer, and a blue display layer, although it should be understood that in other embodiments, other numbers, types, or colors of display layers or combinations of displays can be used. Floor. In the following embodiments, the RGB color layer will be described for illustrative purposes, but this is not a method of limiting color balance (which can be applied to any display color combination).

If the luminance changes corresponding to the red, green, and blue display layers are the same, the chromaticity is maintained on the display. Conversely, if the luminance changes on the corresponding red, green, and blue display layers are different, the chromaticity of the displayed image will be the desired different. For example, for a white calibration image, if the red and blue layers have a higher brightness than the green layer, the area of the white calibration image will appear purple. These deviations from the expected white can be referred to as off grayscale.

A measure of chroma consistency can be used to capture the grayscale of the image. The metric includes an average color error indicating the average of red, green, and blue across the FOV bias for red, green, and blue, respectively. The smaller the average color error, the closer the gray level of the image appears. The average color error can be normalized to a dimensionless value by dividing by the range of average colors or colors (eg, 8-bit 255 colors). In various embodiments, if the average color error is less than 10%, less than 5%, less than 1%, or some other threshold, then it can be considered that the display has achieved chroma consistency.

Figure 14A is an embodiment of a red-green-blue (RGB) intensity map generated from a captured image of a projected test image. The red and blue layers 1402 and 1404 have substantially similar luminances to each other, and both the red layer 1402 and the blue layer 1404 have a higher luminance than the green layer 1406. As a result, the projected area of the white test image will appear purple (red plus blue, see Figure 11B for an example).

Figure 14B is a plot 1408 that maps the maximum color imbalance error. The average luminance 1410 can be identified as the average luminance value of the red, green, and blue color layers. The "Average + Maximum Error" surface 1412 represents the maximum luminance values for the red, green, and blue layers, while the "Average Maximum Error" surface 1414 represents the minimum luminance values for the red, green, and blue layers.

Figure 15 is different in the display field of view as shown in Figure 14A. Color-corrected RGB intensity map of the display system for intensity red, green, and blue layers. As shown below and in graph 1500, in this embodiment, in most displays, the maximum R and B luminance values have been lowered to a level lower than the G luminance value to provide chroma consistency. .

As shown in FIG. 14A, before the color correction, the luminances of the red and blue layers are much higher than the luminance of most of the green layers on the FOV, which causes a large area of the captured image of the white calibration image to appear purple. During color correction in this embodiment, for each point of the depth plane, the lowest luminance value of the color layer associated with the depth plane (eg, red, green, and blue) will be confirmed, and the luminance of each color layer will be set. The value is the lowest luminance value at that point. For example, as shown in FIG. 15, the color luminance of the red and blue layers 1502 and 1504 is lowered to match the color luminance of the green layer 1506 (eg, comparing the RGB intensity map of FIG. 14A with the RGB intensity map of FIG. 15). As a result, the luminances of the red and blue layers are corrected, so they can match the intensity of the green layer, thereby reducing the amount of grayscale off of the projected image.

■Image correction processing

Image calibration refers to the characteristics of the display device associated with previously defined image quality metrics (see description of Figure 7-15). Image correction refers to the corrective measures taken to improve image quality. The image quality metric notification is used to improve or optimize the corrective action taken by the display device image quality metric. Therefore, image correction is closely related to each image quality metric.

16 is a flow chart 1600 of an embodiment of performing image correction processing on a display system. At block 1602, a camera for capturing a projected image is calibrated (eg, For example, camera 1806 of metrology system 1800 described below. Camera calibration includes the camera's accuracy characteristics when capturing and representing actual visual/display messages. To ensure that any metrics measured from the captured image are from the display system, rather than from the camera-associated error, the camera used for image correction should be calibrated prior to image correction.

In some embodiments, camera calibration includes performing at least one of flat field correction (eg, ensuring that the intensity response of the camera is consistent across the FOV), lens distortion correction (eg, discriminating and compensating for lens distortion), or pixel scaling (eg, The relationship between the pixel size on the image capture of the camera and the pixel size of the display system is discriminated). In some embodiments, a display to camera pixel mapping can be applied to perform a transfer between display pixel values and camera pixel values. The display to camera pixel mapping may map the first halftone space to the second halftone space according to a first global nonlinear gamma function, a local, pixel related coupling function that maps the color pixel values to the first halftone space, And a second global non-linear gamma function that maps the second halftone space to the pixel intensity in the camera color space. Details of the embodiment of the display to camera pixel mapping described below with reference to FIG.

At block 1604, spatial error correction can be performed on the display system. Spatial error correction includes capturing one or more images of the projected light field using a calibrated camera that can be used to generate a distortion vector field between the image position displayed by the display and the desired image position. In some embodiments, each display layer produces a separate vector field. Using the generated vector field, one or more spatial corrections can be performed, which can include XY centers (block 1604a), aggregate rotation (block 1604b), aggregation Scale (1604c) or spatial map (block 1604d). In some embodiments, each of these corrections is performed on a substrate of each layer.

The XY center is used to indicate the translational space error of the center of the display image of the display layer with respect to the desired image position. Executing the XY center may include identifying the displayed image center point and moving the image along the measured translation vector such that the center point corresponds to the desired center position. An embodiment of the XY center correction described with reference to Figure 9A.

Aggregate rotation refers to the overall rotation error between the displayed image and the expected position. Performing the collective rotation may include identifying a center point of the displayed image and an amount of rotation of the rotated image around the identified center point (eg, a pixel position error to a minimized position relative to the desired image position). An embodiment of the aggregate rotation correction described with reference to Figure 9B.

The aggregate zoom is the overall scaling error between the displayed image and the desired image. Performing the aggregate zoom may include identifying a center point of the displayed image and scaling the image near the determined center point by a specified factor (eg, a factor that the pixel position error is minimized relative to the desired image position). Embodiments of the aggregate scaling described with reference to Figures 9C and 9D.

While xy centering, gather rotation, and gather scaling can be used to correct linear or affine spatial errors, the display image of the display layer can also contain additional nonlinear or non-affine spatial errors. Used to perform spatial mapping to correct any residual errors (eg, non-linear or non-affine errors) that have remained after performing XY centering, gather rotation, and gather scaling correction. This spatial mapping can also be referred to as pixel distortion. The embodiment as depicted in Figure 9E.

In some embodiments, the spatial error can be separated into an in-plane spatial error and an out-of-plane spatial error (sometimes referred to as a diopter error). For example, the display layer may be corrected for in-plane spatial errors before the out-of-plane spatial error is corrected, and vice versa. Alternatively, the intra-plane spatial error and the out-of-plane spatial error may be corrected together.

Block 1606 is for performing color error correction on the display system. The color error correction can include luminance flattening 1606a or color balance 1606b. In some embodiments, when each color cluster (eg, each RGB cluster) performs color balance, luminance flattening will be performed on the substrate of each layer.

Luminance flattening can refer to reduced luminance variations throughout the display layer. In some embodiments, luminance flattening includes reducing the luminance of all pixels in the displayed FOV to a minimum luminance value. Alternatively, all pixel luminances in the display's FOV having greater than maximum or threshold luminance are reduced to equal the maximum/threshold, while luminance pixels having less than the maximum/threshold may remain unchanged. In some embodiments, the luminance value can be scaled according to the distance between the luminance and the threshold luminance value. An embodiment of luminance flattening as depicted in Figures 12A and 12B.

Color balance may include reducing the off-gray effect caused by intensity mismatch between different color layers in a color cluster (eg, RGB cluster). Color balance can be performed by reducing the luminance of the color layer of each location region in the depth plane to match the color layer luminance in the color cluster having the lowest luminance at the location region. For example, for each pixel in the FOV, red, green in each location area The luminance of the blue color layer and the blue color layer are set at the lowest of the three color layers of the region. In some embodiments, the luminance above the threshold luminance value is reduced to be equal to the threshold luminance value, or reduced to the minimum luminance value in the color cluster of the location region, taking the maximum value therein. In some embodiments, the luminance can be scaled according to the distance between the luminance and the threshold luminance values. An embodiment of color balance as depicted in Figures 14A and 15 is shown.

In some embodiments, image calibration (to quantify the image quality metric) will be performed for each display system during the manufacturing process. The information associated with the image quality metric and the corrections available to improve or optimize the display system can be stored in a non-transitory memory (eg, data module 224 or remote data repository 232) associated with the display system. During use of the display system, image correction information can be applied to the display to perform appropriate corrections to provide an improved or optimized image of the user of the display system after reducing or eliminating image errors in the display. For example, the local or remote processing module 224, 228 can provide improved images to the user using image corrected instant information. An embodiment of the calibration process as depicted in Figures 27 and 28.

■Example of depth plane metrology

Embodiments of the display system of the present invention produce a light field (as depicted in Figures 1-6). Therefore, just as a physical (physical) object will produce a light field impinging on the eye at a certain distance from the wearer of the display, and a virtual object placed at a certain depth will also produce a (digitized) light field, the light The field will focus the virtual object at the desired depth. This allows for a convergence-adjustment match and presents a more convincing mixed reality.

Due to the resulting light field defects (eg, due to defects in the waveguide of the waveguide assembly 405), even if the content creator places the virtual object in a depth region that is specific to the viewer in the rendering engine, the virtual object will be different The depth region of the depth is expected to be focused. This will cause a convergence-adjustment mismatch. In some cases, different parts of the virtual object will appear to focus in different depth regions. These mismatched depths will correspond to a pattern of out-of-plane spatial errors, as in the embodiment shown in Figures 10A-10E.

Accordingly, the present invention describes an embodiment of a metering system that can measure the quality of the light field produced by the display. A portion of the metering system can map the topology and quality of the light field produced by the display and can provide information that will guide the evaluation of the quality of the light field produced by the display. A portion of the metering system captures the vector light field (eg, direction and amplitude) produced by the display and allows analysis of focus and depth defects in the display. The information generated by the metering system of the present invention will be utilized in the spatial and color calibration techniques of established light field displays. The metering system of the present invention is particularly applicable to embodiments of light field displays (e.g., embodiments of display systems 200, 400), but is not intended to be limiting, as the metering system can be used in other embodiments as well. To measure light from any type of display. Embodiments of the measurement system can be used to determine the 3D distortion field and can direct valid spatial calibration information from the display. The metering system can also be used for binocular calibration and monocular RGB and depth plane calibration.

Figure 17A is an embodiment of viewing an object 1702 from an eye 304 having a normal light field. The object 1702 will correspond to a light field that is substantially free of defects. Physical object or virtual object. The point on the object 1702 diverges the associated ray 1706 from a single point (the ray 1706 corresponds to a point on the object 1702 that appears to diverge from a single point), causing the point of the object 1702 to be focused from the eye 304 to a distance 1708. .

Figure 17B is an embodiment of viewing an object 1710 using a defective light field. The object 1710 can correspond to a virtual object, such as a virtual object produced using a display system (eg, display system 400 as shown in Figures 4 and 6). Reuse of the resulting light field defects, such as defects in the waveguides 420, 422, 424, 426, 428, 604, is intended to correspond to the ray 1712 at a particular point on the object 1710 to exhibit a different point divergence, or to exhibit a divergence that is different than expected. As a result, the object 1710 will be out of focus at a distance 1708. In addition, different portions of the object 1710 will be focused at different depths or distances.

The measurement system can also be used to measure the quality of the light field produced by the display. FIG. 18 is an embodiment of a measurement system 1800 for measuring the light field quality of display 1802. The light field produced by the display 1802 will have light 1804 directed at the camera 1806. The display 1802 can correspond to a stacked waveguide assembly (eg, stacked waveguide assembly 405, as shown in FIG. 4). Although the ray 1804 is parallel in the embodiment, this is intended to be illustrative only, as the ray 1804 can also be projected in different directions (eg, diverging) to convey one or more virtual objects presented by the light location. Different depths. Additionally, due to defects in display 1802, light 1804 will also be non-parallel (see Figure 17B).

In some embodiments, camera 1806 can be used to capture at least a portion of the generated light field to measure, for example, a virtual object represented in the light field. Perceive depth. The camera 1806 is for focusing at a particular depth or distance (hereinafter also referred to as "focus depth"). In some embodiments, this can be done using a lens with a small depth of focus (DOF). For example, the DOF can be less than the Z-distance, where defects on the display typically cause the depth of focus to deviate from the expected depth of focus (eg, as shown in Figure 19C, less than the distance between the peak 1924 of the depth map and the expected focus) 1922). In other embodiments, the DOF can be less than a multiple of the distance between the camera and the display, wherein the factor can be less than about 0.1, less than about 0.01, less than about 0.001, and the like. Camera 1806 can be used to capture a particular portion of the light field or the entire light field. The camera 180 is configured to capture a portion of the light field associated with a particular virtual object and present it using the light field. The camera 1806 is positioned to capture images substantially similar to the perception of the eye 304. The camera 1806 and the display 1802 can be moved relative to one another to map out the light field. For example, the relative motion may be parallel to the display 1802 (eg, along the X direction shown in FIG. 18 or perpendicular to the Y direction of X and Z (not disclosed)) or perpendicular to the display 1802 (eg, as shown in FIG. 18) Z direction). In other embodiments, scanning optics (not disclosed) can be used to relatively scan the camera 1806 and the display 1802. In some embodiments, camera 1806 can be used to capture the generated portion of the light field to determine a deformation map (eg, as shown in FIG. 8) that can be used to discriminate spatial errors in the projected image (eg, as illustrated The in-plane spatial error shown in 9A-9E or the out-of-plane spatial error shown in Figures 10A-10E). Additionally, camera 1806 can be used to discriminate luminance or chromatic aberration in the generated light field (e.g., as shown in Figures 11-15).

In some embodiments, camera 1806 can be moved to different parties Oriented upwards. For example, although the camera 1806 is exposed to face the display 1802 orthogonally, the camera 1806 can also be rotated (eg, rotated along the Y axis or X-axis) such that the display 1802 is oriented at a different angle, thereby allowing the camera 1806 measures the light field produced by display 1802 in different directions or orientations.

In various embodiments, camera 1806 can be a digital camera, such as a short focus digital camera. In other embodiments, camera 1806 can be a light field camera.

Camera 1806 is coupled to controller 1808 for controlling the depth of focus of camera 1806, the field of view of camera 1806, the exposure time, the relative movement of camera 1806 and display 1802, and the like. In some embodiments, controller 1808 can correspond to controller 450 as shown in FIG. Controller 1808 can include a hardware processor and a non-transitory data store.

19A is an embodiment of an image 1900 captured by a camera at a particular depth of focus (eg, camera 1806). Image 1900 can include one or more regions 1902 of focus, and one or more regions 1904 that are out of focus. Since the camera 1806 can be used to focus at different depths of focus, the focused or unfocused regions of the image can be changed. For example, if the camera is changed to focus at a different depth of focus, region 1902 can appear out of focus, while portions of region 1904 can enter focus. The perceived depth of each region of the light field can be determined by capturing multiple images of the light field at a plurality of different depths of focus. For example, each pixel of the image captured by the camera will correspond to a particular depth of focus associated with the depth of focus, where a portion of the light field will correspond to the focus of the pixel. The generated light field region and the perceived depth may be mapped to form a depth map or a graphic. In addition, depth maps or graphics are also The projected depth of focus can be specifically prepared for the display, thus comparing the expected depth of focus of the virtual object presented in the light field with the actual measured depth of focus.

19B is an embodiment of a depth map to reveal an embodiment in which measurement of the depth of focus is performed using measurement system 1800. Figure 1910 plots the measured focal depth 1912 of the light field generated along a line that emits a light field from display 1802 (e.g., along the horizontal X-axis of the light field, as shown in Figure 18). In some embodiments, the depth of focus of camera 1806 can be scanned across a plurality of different depths of focus to produce graphics 1910. For example, camera 1806 can be focused at a focal depth 1914 (represented by a horizontal dashed line). In an ideal display, the light field produced by the display will cause the actual measured depth of the virtual object to be exactly the expected depth, but in actual display, the two may not be identical due to defects in the display. Thus, when the measured depth of focus is close to any region of the light field of the depth of focus 1914 (eg, region 1916), it can be perceived as being focused, while having a measured depth of focus that is significantly different from the region of the light field of the depth of focus 1914 (eg, region 1918) can be perceived as out of focus.

Figure 19C is an embodiment of a depth map for capturing one or more generated images. The depth map 1920 includes an expected depth location 1922 and a measured depth map 1924. The depth position 1922 is the image generated by the focus of the display 1802 (indicated by a horizontal plane in FIG. 19C), and the measured depth map 1924 exhibits a depth of focus (Z) and is also focused to generate an image. The expected depth of focus 1922 and measured depth of focus 1924 are compared to allow for defects in the light field produced by display 1802 to be discerned and quantized on the field of view (FOV) of the display.

For example, the expected depth of focus of the light focused at the horizontal position (X ₀ , Y ₀ ) is Z ₀ , and the measured depth of focus at that position is Z, then (ZZ ₀ ) is at the position (X ₀ , Y ₀ The display at the location focuses on the measure of the defect. In some embodiments, the actual horizontal position (X, Y) of the light focus can be measured. In an embodiment, the vector measurement of the actual focus position relative to the expected focus position (X, Y, Z) - (X ₀ , Y ₀ , Z ₀ ) may present a defect in the light field produced by the display. The display defects of this vector measurement provide errors in 3D in-plane and out-of-plane (eg, diopter). In some embodiments, only the 2D vector error measurement (X, Y) - (X 0, Y 0) measurements (and calibrated) within the error plane. In some cases, the focus error of the display can be determined on a pixel-by-pixel basis. However, due to the large number of pixels in many displays (eg, millions of pixels), focus error data can be determined only for portions of the display or groups of pixels that are sampled for the display (eg, a 10×10 or a span across the display) 100x100 samples). The checkerboard pattern does not need to be square, but can also be designed to conform to the pixel structure of the display.

FIG. 20 is a flowchart showing an implementation of the program 2001 for measuring the quality of the virtual target mode generated by the light field display. Program 2001 may be performed by metering system 1800, such as by controller 1808. In some embodiments, the virtual target mode is a checkerboard pattern with an array of alternating light and dark regions. The checkerboard pattern can be used to sample portions of the display (eg, 10x10 or 100x100 or other sized checkers) or can correspond to the number of pixels of each size display. In other cases, one (or more) of the pixel groups may be turned on and off in sequence and an image of the open pixel may be acquired to obtain pixel-by-pixel data. Checkerboard pattern (sequence of open/closed pixels) can include bright and dark areas Random sequence or geometric pattern of bright and dark areas or any other type of calibration pattern. Embodiments of the checkerboard pattern and pixel switch sequence described below with reference to Figures 22-23B. At block 2002, the initial focus depth can be set. In some embodiments, this includes the depth of the focusing lens on the camera. The initial depth of focus may correspond to any depth of the virtual target mode. For example, the initial depth may correspond to a minimum or maximum depth associated with the virtual target mode.

At block 2004, an image of the virtual target pattern is captured at the selected depth of focus. In some embodiments, the image may include a focused portion and an out-of-focus portion. In some embodiments, the range of images may be concentrated on a particular virtual object associated with the virtual target mode. In other embodiments, the image may correspond to an entire light field comprising a plurality of virtual objects. The image may include pixel-by-pixel focus depth information across the virtual target pattern.

At block 2006, the next step is performed if there is an additional depth of focus for the captured image. If it is determined that there is an additional depth of focus, then at block 2008, a new depth of focus is selected. In some embodiments, the number of depths of focus may be at least partially exhibited by the display system at different depths (eg, the number of depth planes 306 as shown in FIG. 3 or the waveguides in the waveguide assembly as shown in FIG. Quantity). In some embodiments, if the image is focused on a particular virtual object, the range of depth of focus may be associated with one or more depths of the virtual object (eg, the minimum depth and maximum depth associated with the virtual object).

If there is no additional depth of focus for the captured image, another step is performed. The step block 2010 is used to analyze the captured image of the virtual target pattern. Image to identify depth Z or lateral position (X, Y), where different regions of the target pattern are actually focused. For example, each captured image of the virtual target pattern may correspond to focus and partial out of focus of a portion included in a particular depth of focus. In some embodiments, each image may be divided into one or more regions corresponding to the light field regions. Autofocus techniques can be used to determine at which depth each region is focused. In some embodiments, each region may correspond to a pixel.

At block 2012, a depth map is established based, at least in part, on the measured depth of focus (or lateral position). The depth map will include any type of data structure or visualization that maps the depth of focus to the light field location. For example, the depth map may include depth information of the captured image of one or more pixels (eg, a Z-axis focus depth or a measurement of the Z-axis focus depth (X and/or Y position) in conjunction with the lateral focus position). In some embodiments, the pixels may correspond to a pixel cloud associated with the target virtual object. In this way, the depth map can specify the actual perceived depth of the virtual object when viewed through the display optics.

At block 2014, the depth map may compare one or more desired depths of focus to each other, where the desired depth of focus may correspond to the depth presented at one or more virtual objects. Defects and/or deviations in the light field can be found by examining the difference between the actual perceived depth of the virtual object and the depth of focus in which the virtual object is intended to appear.

At block 2006, error correction may be performed based at least in part on the comparison of the depth map and the desired depth of focus. This error correction can compensate for defects in the light field display or image content projected by the display.

The program 2001 can be repeated for each of the waveguide assemblies 405 of the light field display to map the defects of each waveguide. In some cases, there may be multiple waveguides corresponding to multiple depth planes and multiple waveguides corresponding to multiple colors (eg, red (R), green (G), and blue (B)). For example, some displays have three color planes for each depth plane, so a waveguide assembly with two depth planes can have 2x3 = 6 waveguides. Camera 1806 can be a combination of multiple color sensitive cameras or cameras, each sensitive to a subset of colors. The depth of focus information obtained by metering system 1800 can be used to determine the spatial distribution of focus errors and the distribution of color (color) defects of the display.

In some embodiments, a light field camera can be used to capture the light field produced by display 1802 (eg, using a digital camera with scanning focus) instead of multiple images captured at multiple different depths of focus. The captured light field can analyze focus and/or depth defects. By analyzing the vector of rays in the captured light field, the depth of focus of each region can be determined. The identified depth of focus and one or more expected focus depths can then be compared and appropriate error corrections can be performed (as in block 2016). For example, the actual focus position (X, Y, Z) of the vector meter corresponds to the expected focus position (X ₀ , Y ₀ , Z ₀ ) and can be determined as: vector error = (X, Y, Z) - (X ₀ , Y ₀ , Z ₀ ), used to describe defects in the light field generated by the display.

■ Embodiment method of color balance display

As mentioned above, some implementations of full-color displays project red (R), green (G), and blue (B) wavelengths by combining displays, and A tristimulus response is produced on the subject's retina. An ideal display has a spatially uniform luminance for the three color layers; however, due to hardware defects, the actual display will have some amount of variation in the luminance of the field of view. If there are different variations for different color layers, the change will produce uneven chromaticity on the field of view (FOV) of the display (eg, as shown in Figure 11). Embodiments of a method of correcting color variations and attempting to uniform chromaticity on the FOV are described. For example, the intensity of the various color layers (eg, R, G, and B) of the display can be adjusted such that the white point of the display is substantially uniform at the FOV.

In some embodiments, the light field metrology system described herein can be used for color balance of a display. For example, a digital color camera can capture an image of the display (eg, using a metrology system 1800 as shown in FIG. 18) from which the color response of the display can be determined for some or all of the pixels of the display. In many displays, there are three color layers (eg, R, G, and B), however, the invention is not limited to RGB or 3-color displays. The method of the invention can be applied to any number of color layers (e.g., 2, 3, 4, 5, 6, or more) and any color (e.g., cyan, magenta, yellow, black).

Figure 14A (before color calibration) and Figure 15 (after color calibration) are embodiments of measurement color balance for a particular implementation of an RGB display. 14A and 15 are distribution plots (1400, 1500, respectively) including R, G, and B intensities (vertical axes) across display pixels (horizontal axes). Figure 14B includes a curve 1408 for the maximum color imbalance (vertical axis) of the display pixels (horizontal axis), showing the average and average values before color correction plus or minus the maximum error.

As noted above, Figure 14A presents an uncalibrated display having a distinctly non-uniform color response on the pixels of the display. The red and blue reactions are the same, and the R and B intensities protrude toward the right side of the drawing 1400. The green response is typically less than the R or B reaction and will therefore decrease toward the right side of the plot 1400. Figure 15 reveals that after applying the color calibration described below, the calibrated display will have a uniform color response on the display pixels.

Embodiments of the color balance system and method of the present invention are for providing a technique for adjusting the intensity of at least some of the color layers in a multi-color display such that the white point of the display is substantially uniform across the display FOV. In various implementations, the display can be a light field display. For example, a display may have the ability to present a color image to a viewer in multiple depth planes. Embodiments of the color balance system and method can be applied to color balance display 208 (FIG. 2), display system 400 (FIGS. 4-6), and display 2500 (FIGS. 25A, 25B, 26).

The human eye does not perceive the level of light in a linear manner. For example, the human eye is more sensitive to changes in dark tones or similar tonal hue than an ideal, linear display, which allows the human visual system to operate over a wide range of luminance levels. Real-world displays also do not provide accurate linear brightness response. In addition, digital images are typically encoded to indicate a more uniform tone level in perception. Human visual perception, display output, and image coding are typically shaped to follow an approximate power law relationship with respect to brightness or color gradation. For example, the output level and the increased power gamma: V _out . The input level is proportional. This nonlinearity, power law, and characteristic are often referred to as gamma correction, gamma coding, or simply gamma.

In some embodiments, if the luminance flatness of each color layer in the display is nearly uniform across the FOV of the display, the color balance may include the zoom intensity of the various color layers used to achieve uniform color balance across the display. If the luminance variation across the FOV of the display is less than 1%, less than 5%, less than 10% in various embodiments, the display may have appropriate luminance flatness. Due to the gamma response of the display and human visual perception, in some cases, such direct scaling may have certain drawbacks.

If the color layer of the display does not have substantial luminance flatness, the color balance may include not only the zoom intensity of each color layer. For example, color balance can attempt to separately balance white points at each pixel (or set of pixels) of the display. In some such embodiments, the color balance of the FOV across the display can be achieved without flattening the luminance across the FOV. Luminance flattening may be performed in addition or alternatively to achieve color balance.

The purpose of the color balance display is to allow the viewer of the display to perceive a uniform color balance across the FOV of the display. To measure and adjust the color balance of the display, use a calibration camera (rather than the human eye) to record the image of the display output. It can be assumed that the camera is a representation of human perception of the display output, and if the camera image of the display is color balanced, the viewer's perception of the display will also be color balanced.

In some embodiments, the following mode is for converting pixel values of the display color layer to color pixel values measured by the calibration camera. In the following embodiments, there are three color layers, which are assumed to be R, G, and B; however, this is for illustration The purpose, not the limit. In other cases, any number and hue of color layers can be used in embodiments of color balancing techniques. In addition, proper scaling between the display and the pixel size of the camera can be considered before applying this mode.

In equation (1), [Rd, Gd, Bd] is the intensity value of the RGB image transmitted to the display. In many cases (eg, standard RGB or sRGB), the intensity values are between 0 and 255. The gamma 1{} represents a first nonlinear gamma function (having an exponent γ ₁ ) which maps the display gradation to the midtone representation [R1 G1 B1]. Coupling () represents a function that maps the chrominance value [R1 G1 B1] to the second halftone representation [R2 G2 B2]. The coupling() function can be a linear function, for example, a 3x3 matrix (in the case of 3 color layers). In other embodiments, the coupling () function can be non-linear. The gamma 2{} is a second nonlinear gamma function (having an index γ ₂ ) that maps the second halftone expression [R2 G2 B2] to the pixel intensity [Rc Gc Bc] recorded by the calibration camera.

In some embodiments, the first and second gamma functions are global functions on the FOV of the display (eg, the indices γ ₁ and γ ₂ on the FOV are constant). The coupling () can be a local (pixel dependent) function that varies from pixel to pixel throughout the FOV. The per-pixel color mapping provided by the coupling() function allows each pixel to be color balanced.

To determine the function gamma 1{}, gamma 2{} and coupling (), one or more images of a series of displays can be captured by the camera and analyzed by a programmed analysis system to perform a repetitive optimization calculation. The method (for example, hill climbing algorithm, local search, simplex method, genetic algorithm, etc.) is used to provide a suitable fit of the gamma and coupling function of the display with a reasonable color balance. When the analysis system searches for a suitable fit of the gamma and the coupling function, the analysis system can use the feedback during the iterative process by capturing additional images (or multiple) of the display. For example, function gamma 1{}, gamma 2{}, and coupling () can be determined by repeatedly adjusting these functions to improve or optimize the color balance of the camera image of the FOV across the display. These functions can be adjusted repeatedly until the white point of the camera image acquired during the iterative process is substantially uniform at the FOV of the display. In various embodiments, a substantially uniform white point distribution is associated with a change in white point that is measured within the color system as a white point value that is less than 10%, less than 5%, or less than 1% across the FOV. . For example, a color space provided by the International Commission on Illumination (CIE) can be used. In some embodiments, the substantially uniform white point distribution may be associated with a change in white point that is less than a threshold amount according to a difference in color space (JND). In some embodiments, the gamma transfer function gamma 1{} and gamma 2{} are first repeatedly calculated, and then once the gamma function (eg, the indices γ ₁ and γ ₂ ) has been calculated, the calculation is performed again. Coupling () function.

A production process for calibrating a display in a manufacturing environment can automatically depict the display as it is transmitted along the production line. For example, at a suitable point in the production process, the calibration camera and analysis system of the present invention can perform repeated analysis to obtain a gamma transfer function and coupling function for a particular display, and the resulting The gamma and coupling functions are stored in a memory associated with the display. The display then has the ability to automatically perform color balance.

During use of a particular display, once the gamma transfer function gamma 1{} and gamma 2{} and coupling () functions are known for a particular display, the appropriate display pixel value [Rd Gd Bd] can be input to Equation (1) to achieve color balance output. For example, the gamma index and coupling() functions determined by a particular display can be stored in a suitable memory within the display and retrieved to transform the input image pixel color values to provide a color balanced output from the display. In some embodiments, the local processing and data module 224 of the wearable display system 200 can store gamma transfer and coupling functions, and the processing module can output an instant color balance image (FIG. 2) using equation (1). In other embodiments, controller 450 of display system 400 can perform color balancing in accordance with equation (1) and stored gamma and coupling functions (FIG. 4). In other embodiments, as described below, dynamic calibration processor 2610 of dynamic calibration system 2600 can perform color balance on display 2500 (FIG. 26) using equation (1) and stored gamma and coupling functions.

Embodiments of method 2700 or process flow 2805 for dynamically calibrating a display according to eye tracking as described with reference to FIGS. 27 and 28 are also described. The function of the image sensor to perform color balance and other error correction/calibration is described in more detail below. For example, the calibration taken at block 2720 of method 2700 can include gamma and coupling functions, and at block 2730, the color of the display can be corrected by using equation (1) and the extracted gamma and coupling functions. defect. As another example, block 2880 of process flow diagram 2805 can take out gamma and coupling functions and Apply them during calibration.

21 is a flow diagram of an implementation of a method 2150 for calibrating a display. The display can be a light field display. The display can be display 208 (FIG. 2), display system 400 (FIGS. 4-6), and display 2500 (FIGS. 25A, 25B, 26). Method 2150 can be performed by an analysis system (including a camera and an analysis program executed by a computer hardware, such as metering system 1800 shown in FIG. 18) as part of a production line of a display manufacturing process (eg, as described in reference to FIG. 28) Process 2805). Method 2150 can be performed as with camera calibration of a portion of block 1602 of flowchart 1600 described with reference to FIG. In some embodiments, method 2700 applies equation (1) to determine an appropriate transition between the display and the camera (assuming visual perception of the viewer presenting the display). At block 2160, the image of the display is captured by the camera. At block 2170, global transformation parameters that are transformed between the display and the camera are determined. Global transformation parameters may include parameters that are not varied across the FOV of the display (eg, not pixel related parameters). For example, global transformation parameters can include Gamma1{} and Gamma2{} functions. In some cases, method 2150 can return to block 2160 to obtain one or more additional images as part of an iteration, a feedback process for determining global transformation parameters. After obtaining a suitable fit to the global transform parameters, method 2150 moves to block 2180 where local (eg, pixel related) transform parameters are blended into the camera image. For example, the local transform parameters can include a coupling () function (eg, the function value at a pixel location across the FOV of the display). In some cases, method 2150 can return to block 2160 to obtain one or more additional images as part of an iteration for determining local transformation parameters Feedback process. In some embodiments, after obtaining additional images at block 2160, method 2150 can jump back to block 2180 to continue to fit the local transformation parameters instead of passing block 2170 because the global transformation parameters have been previously determined. After the appropriate fit of the local transformation parameters is matched to the camera image, method 2150 moves to block 2190 where the local transformation parameters and global transformation parameters are stored in a memory associated with the display (eg, local data module 71). As described above, at block 2720 of method 2700 for dynamically calibrating the display, local and global transform parameters can be accessed for partial calibration of the display, and at block 2730, local and global transform parameters and equations ( 1) Can be applied to produce color balance images from the display.

Although described with respect to the color balance of the display, the present systems and methods are not limited thereto and can be applied to correct other color (or spatial) defects of the display (eg, any color or spatial defects described above). For example, as described above, the display can exhibit a change in luminance flatness, and embodiments of the analysis techniques of the present invention can determine a luminance flatness calibration that is used to correct for defects in luminance flatness. Additionally or alternatively, the display may exhibit spatial imperfections, including in-plane translation, rotation, scaling or warping errors, and out-of-plane (eg, depth of focus) errors. Embodiments of the analysis techniques of the present invention may determine calibration for some or all of these spatial errors.

■ Display calibration example using calibration pattern

Defects in the display may cause the virtual object projected by the display to appear spatially distorted or chromatically distorted. In order to correct these distortions, you can first measure The distortion is calibrated and then any necessary error corrections are performed (e.g., using measurement system 1800 shown in Figure 18) to calibrate the display. Display calibration may include using a display to project a calibration pattern, such as a checkerboard pattern (eg, as shown in FIG. 7), and capturing the resulting image with a camera. The captured image can then be processed to determine the distortion at the feature point location of the calibration pattern by quantifying the error between the desired position of the pattern feature point and its measured position. For displays with separate color layers (eg, red (R), green (G), and blue (B)), this calibration can also correct color quality and image quality.

22 is an embodiment of a calibration system 2200 that uses a calibration pattern. Display 2202 can be configured to project calibration pattern 2204 into a generated light field 2206, which can be captured using an imaging device such as camera 2208. In some embodiments, display 2202 includes a stacked waveguide assembly (eg, as shown in FIG. 4 or FIG. 6) or other type of light field display. In some embodiments, camera 2208 (or display 2202) is configured to be movable such that system 2200 will be able to capture images of light field 706 from different lateral positions, depths, or angles. In some embodiments, calibration system 2200 can be similar to metering system 1800 of FIG. For example, display 2202, light field 2206, and camera 2208 can correspond to display 1802, light field 1804, and camera 1806 of metering system 1800.

In this embodiment, the calibration pattern 2204 includes a checkerboard pattern in which different regions have different (eg, alternating) optical characteristics, such as luminance (eg, light or dark), chromaticity, hue, saturation, color, and the like. The checkerboard pattern may be a regular pattern (eg, as shown in FIG. 22) or an irregular pattern. Calibration pattern 2204 includes A plurality of feature points for measuring the amount of distortion in the image captured by camera 2208. For example, the feature points of the checkerboard pattern include points on the border and corners between the check boxes of the checkerboard or points at the center of the check box. The calibration pattern 2204 can be the same size as or smaller than the display 2202. As the system 2200 moves over the display 2202, a smaller calibration pattern can be moved over the display 2202, and when measuring the distortion of the display 2202, the camera 2208 can acquire multiple images of the calibration pattern 2204. In some embodiments, the calibration pattern 2204 can be randomly sampled according to a sequence optimized for operation.

Due to errors in display 2202 (eg, defects in one or more waveguides or lenses), light field 2206 may contain defects that cause distortion of the pattern in the virtual object or light field. This may cause a deviation between the expected focus position (lateral or depth) of the feature points on the calibration pattern 2204 and the actual measured position in the image they are captured by the camera 2208. The deviation caused by the distortion is determined and measured by comparing the actual measurement position of the feature points of the calibration pattern 2204 with the expected position at which these feature points can be identified and measured. In some embodiments, the calibration pattern includes color information such that the color error of display 2202 can be quantified by system 2200. In some embodiments, a distortion map can be generated for error correction 2202 of the spatial or color error of the display (eg, as shown in FIG. 8).

In some embodiments, each of the calibration patterns 2204 corresponds to a single pixel of the display 2202, which may allow for direct measurement of display defects on a pixel by pixel basis. In other embodiments, each check box 2304 corresponds to a plurality of pixels (eg, an N x M pixel grid, at least one of N or M being greater than one). In some such embodiments, the coarse quality of the calibration pattern means that distortion information is obtained at the sample point and can be interpolated to obtain each pixel distortion information. For example, in the checkerboard pattern shown in FIG. 23A, distortion information may be measured for a pattern position corresponding to the feature point 2302 (eg, a border, a corner, or a point on the center of the inspection frame). Distortion information for other points in the checkbox region 2304 can be inferred or interpolated from the measured distortion values associated with the feature points 2302.

The board projection capture process identifies feature points (eg, the edges of the check box) to quantify errors in the expected position and measurement position for distortion calibration. The feature points may be sparse compared to the number of pixels in the display. For example, a high definition display can include millions of pixels (eg, 2.1 million pixels for a 1920x1080 pixel resolution), while the number of squares 804 in a calibration pattern can be almost less (eg, for 50x50, 100) ×100, 500×500 pattern). Thus, embodiments of system 2200 that use a single projection to extract approximate rate sampling measurements can be interpolated to estimate each pixel distortion.

In order to obtain accurate per-pixel distortion information for the display, embodiments of system 2200 can automatically obtain distortion information by implementing different or displacement calibration patterns. Different calibration patterns can be projected or the same pattern can be incrementally shifted such that the entire pixel space of display 2202 is measured. Automated image projection and capture or different displacement calibration patterns allow for accurate mapping of the distorted pixels for display 2202.

The system 2200 can be modified on a per pixel basis by automatically repeating the board projection, but with a calibration pattern of, for example, 1 pixel displacement. Good distortion information. For example, camera 2208 can obtain an image of the pattern each time the pattern is displaced. Due to each repeated image capture, the feature points of the projected calibration pattern are mapped to different pixel groups. The displacement of this calibration pattern can be repeated until an intensive sampling of the distortion field of the display is obtained. For example, the board may be projected and displaced through a plurality of locations corresponding to pixels of the checkbox of the board and allow for measurement of distortion information for each pixel of the display. In other embodiments, the displacement can be different than one pixel, for example, 2, 3, 4, 5, 8, 16, or more pixels. The displacement can be different for displays in different directions, for example, the x displacement does not need to be the same as the y displacement.

Although the present invention discloses an embodiment of a checkerboard pattern, it should be understood that other types of patterns may be used. For example, other geometric patterns can be used, random patterns can be used at random, or any other type of calibration or test pattern can be used. In some embodiments, a single pixel calibration pattern in the display is turned on during use. Figure 23B is an embodiment of a single pixel calibration pattern in which the single pixel 2306 has been turned on. The transfer function of each pixel of the display device to the viewer scene is quantized from the captured image of each resulting frame. After each image capture, the position of the displayed pixel 2306 can be moved across the display (eg, in the direction indicated by arrow 2308) over a set distance (eg, a single pixel). And by automatically scanning each pixel of the display to obtain a full quantification of the quality of the display device. In other embodiments, the shifting of the illuminated pixels can be a different number of pixels, such as 2, 3, 4, 5, 8, 16 or more pixels, which can be different for different lateral directions on the display, Or can be in each image撷 The pixels are illuminated to illuminate a plurality of pixels (instead of a single pixel as shown in Figure 23B).

Figure 24 is a schematic diagram of a flow 2400 for performing automatic display calibration. Flow 2400 can be performed, for example, as part of flows 2700 and 2805 described with reference to Figures 27 and 28. At block 2402, a calibration pattern is projected by the display. The calibration pattern can include any pattern having one or more feature points that can be produced by the display. In some embodiments, the calibration pattern comprises a checkerboard pattern. In other embodiments, other types of calibration patterns can be used, such as a single pixel pattern.

At block 2404, a camera or other type of image capture device is used to capture an image of the displayed calibration pattern. If there are errors or defects in the light field produced by the display, portions of the displayed calibration pattern may become distorted, with one or more of the feature points in the calibration pattern appearing at a different location than the desired location. The brightness or chromaticity of the image will also differ from the brightness or chromaticity desired for the calibration pattern.

At block 2406, a distortion of the error between the expected position of the feature point corresponding to the calibration pattern and the position of the captured feature point is determined. For example, for a single pixel calibration pattern, distortion information can be calculated for a particular pixel location of the pattern. For the checkerboard pattern, distortion information can be calculated for pixels corresponding to feature points of the checkerboard (eg, the edge, corner, or center of the check box). In some embodiments, a luminance or chromaticity error between the luminance or chrominance of the calibration pattern and the corresponding luminance or chrominance of the captured image of the calibration pattern is determined.

At block 2408, it is determined if there are any additional locations on the display that should have projected the calibration pattern. If it is determined that there is an additional location, then at block 2410, the calibration pattern is shifted and projected at the new location, and the calibration can be captured The image of the pattern (block 2404) is used to calculate the amount of distortion (block 2406). In some embodiments, the number of different locations used to display the calibration pattern is based on the calibration pattern used. For example, for a single pixel calibration pattern, the number of locations may correspond to the total number of pixels that the display can display. For a checkerboard pattern, the number of locations can be based on the number of pixels in each checkbox.

Once the calibration pattern has been displayed at all desired locations, at block 912, the calculated distortion can be aggregated and used to generate a distortion map of the distortion information for each pixel (or set of pixels) of the display. Distortion information may include spatial distortion due to focus errors (eg, in-plane or out-of-plane errors) or color errors (eg, luminance or chrominance errors). At block 2414, error correction can be performed on the display using the calculated distortion map. For example, distortion information (eg, a distortion map) may be stored by the data modules 224, 232 of the wearable display system 200 shown in FIG. The processing modules 224, 228 of the wearable display system 200 can use distortion information to correct for spatial or chromatic aberrations in the display 208 such that images perceived by the wearer 204 of the display system 200 are at least partially compensated.

In some embodiments, FIG. 24 is an embodiment for light field display to perform a process 2400 on each of the waveguide assemblies 405 of the light field display to calibrate each waveguide. In some cases, there may be multiple waveguides corresponding to multiple depth planes and multiple waveguides corresponding to multiple colors (eg, red (R), green (G), and blue (B)). For example, for some displays, there are three color planes per depth plane, so a waveguide assembly with two depth planes can have 2x3 = 6 waveguides. In addition, in addition to the pixel position, you can also calibrate the chromaticity And quality in order to correct the color (color) defects of the display. For example, camera 2208 can be a combination of multiple color sensitive cameras or cameras, each sensitive to a subset of colors, and used to capture an image of light field 2208 in which the captured color or luminance value of projection pattern 2204 can be identified. The deviation from the expected color or luminance value.

■Waveguide display embodiment

25A illustrates a top view of an embodiment of a display 2500 that includes a waveguide 2505, a light input coupling element 2507, a light redistribution element 2511, and a light output coupling element 2509. Figure 25B is a schematic cross-sectional view showing the display 2500 of Figure 25A taken along axis AA'.

Waveguide 2505 is part of waveguide stack 405 in display system 400 shown in FIG. For example, the waveguide 2505 can correspond to one of the waveguides 420, 422, 424, 426, 428, and the light output coupling element 2509 can correspond to the light extraction optical elements 460, 462, 464, 466, 468 of the display system 400.

Display 2500 is used to couple incident light of different wavelengths, as indicated by rays 2503i1, 2503i2, and 2503i3 (solid, dashed, and two-dot, respectively), to optical waveguide coupling element 2507 into waveguide 2505. Incident light incident on the waveguide 2505 can be projected from an image injecting device, such as one of the image injecting devices 440, 442, 444, 446, 448 shown in FIG. Light input coupling element 2507 is used to couple the wavelength of the incident light at an appropriate angle into waveguide 2505, which is transmitted through waveguide 2505 by the support of total internal reflection (TIR).

The light redistribution element 2511 is configured to be disposed in the optical path along which different wavelengths of light 2503i1, 2503i2, and 2503i3 are transmitted through the waveguide 2505. The light distributing element 2511 is configured to redirect light from a portion of the light input coupling element 2507 toward the light output coupling element 2509 and to increase the beam size of the interacting light in the direction of propagation. Therefore, the light distributing element 2511 can advantageously expand the exit pupil of the display device 2500. In some embodiments, the light distributing element 2511 can thus have the function of a vertical pupil dilator (OPE).

The light output coupling element 2509 re-extracts the input coupling light incident on the element 2509 at an appropriate angle (e.g., in the z direction) and efficiency to facilitate the xy plane of the waveguide 2505, contributing to different wavelengths and at different depth planes. A color image that allows viewers to perceive good visual quality. The light output coupling element 2509 can have a diverging optical power that provides light that is emitted through the waveguide 2505 such that the image formed by the light exiting the waveguide 2505 is (for the viewer) originating at a certain depth. Light output coupling element 2509 can amplify the exit pupil of display 2500 and can direct light to the viewer's eye as an exit pupil dilator (EPE).

The light input coupling element 2507, the light output coupling element 1009, and the light distribution element 2511 may comprise a plurality of gratings, such as an analog surface relief grating (ASR), a binary surface relief structure (BSR), a volume holographic optical element (VHOE), a digital surface Embossed structures and/or volume phase holographic materials (eg, holograms recorded in volume phase holographic materials) or switchable diffractive optical elements (eg, polymer dispersed liquid crystal (PDLC) gratings). In various embodiments, light input coupling element 2507 can include one or more optical prisms or optical elements including one or more diffractive elements and/or refractive elements. Various diffractive or grating structure groups are used for the waveguide by using a fabrication method such as injection compression molding, UV replication, or nanoimprinting of a diffraction structure.

The light input coupling element 2507, the light output coupling element 1009, or the light distributing element 2511 are not necessarily a single element (for example, as shown in FIGS. 25A and 25B), and such elements may also be plural elements. These elements are disposed on one (or both) of the major surfaces 2505a, 2505b of the waveguide 2505. As shown in FIGS. 25A and 25B, the light input coupling element 2507, the light output coupling element 2509, and the light distribution element 2511 are disposed on the main surface 2505a of the waveguide 2505.

In some embodiments, one or more wavelength selective filters may be coupled to light input coupling element 2507, light output coupling element 2509, or light distribution element 2511. The display 2500 shown in FIG. 25A includes a wavelength selective filter 2513 that is integrated into or on the surface of the waveguide 2505. The wavelength selective filter can be configured to filter out light of a portion of one or more wavelengths propagating along various directions in the waveguide 2505. The wavelength selective filter can be an absorption filter such as a ribbon absorber.

■ Dynamic calibration embodiment of AR or VR display according to eye tracking.

The display system is used to be calibrated (space and/or chrominance) to produce an improved quality image. In some near-eye displays (eg, stacked waveguide assembly 405 shown in FIG. 4 for display 208 shown in FIG. 2 or display 2500 described with reference to FIGS. 25A and 25B), the calibration is nominally fixed. The position of the eye (e.g., the wearer looking straight ahead through the display 208) can be quite accurate, but the orientation or position of the other eyes will not be accurate. Therefore, the calibration of the display will depend on the eye position or eye direction. If only a single (eg, reference) position is used for calibration, there will be no when the wearer is facing a different position (eg, away from the reference position) Corrected error.

The present invention also describes an embodiment of dynamic calibration of the wearable display system 400 using eye tracking, where spatial and/or color calibration may vary due to changes in eye position (or, in some cases, eye direction). Some of these calibrations provide a pre-calibration system that will result in maintaining high quality images for a wide range of eye movements. In some embodiments, calibration is performed on-the-fly through a hardware processor (eg, processing module 224, 228 of wearable display system 200 or controller 450 of display system 400) without adding special hardware.

Calibration can compensate (or correct) spatial and/or color (color) errors in the field of view of the display. For example, spatial errors may include in-plane translation, rotation, scaling or distortion errors as well as out-of-plane (eg, depth of focus) errors. The color difference may include luminance flatness or color uniformity error for each color (eg, R, G, and B) to be displayed.

26 is an embodiment of a dynamic calibration system 2600 for display 2500 for which calibration can be applied to correct for spatial and/or color errors of the grid reference position (shown by point 2602). The dynamic calibration system 2600 can include a display 2500, an internal-facing imaging system (eg, an eye-tracking camera 500), and a dynamic calibration processor 2610 (which retrieves and applies the calibration). Figure 26 is another embodiment of a display 2500 that includes an embodiment of the optical component described with reference to Figures 25A and 25B. Light output coupling element 2509 directs light to the viewer's eyes. When the viewer's eye is positioned at a different position 2602 relative to the light output coupling element 2509, the display for that particular eye position (point 2602 as disclosed in FIG. 26) The optical alignment of the display 2500 can be different. For example, if the eye is at position 2602a, near the center of light output coupling element 2509, and if the eye is above position 2602b, toward the upper left corner of light output coupling element 2509, and similar to any other implementation on optical element 2509. The calibration of position 2602 is not the same.

The field of view (FOV) of the display remains substantially the same as the user's eyes move relative to the display, but the spatial and/or chromatic distortion in the display can change as the eye translates relative to the display. Since the FOV includes an angular range in which the image is presented to the user, the calibration data (at a given location relative to the display) can generally take into account all orientations or viewing angles of the eye. For example, when a user directs his vision to a different angle (while maintaining the same position relative to the display), the user can only view images of different portions having the same overall distortion. Thus, at any given location, when the direction of the eye changes (eg, the eye gaze direction changes), the view of the eye typically remains within the FOV of the display, and the same calibration (for a given eye position) can be used for almost The direction of all eyes. Thus, certain embodiments of the calibration system utilize position-dependent calibration that is without additional orientation correlation.

Note that points 2602, 2602a, 2602b are for reference only and do not form part of light output coupling element 2509 or display 2500. Moreover, although nine locations 2602 in a 3x3 grid are illustrated in Figure 26, this is for illustrative purposes only, it should be understood that the number (or arrangement) of locations for calibration of display 2500 can be different from the map. 26 is shown. For example, in various embodiments, 1, 2 are used. 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 16, 20, 25, 100, 256 or more calibration positions. The calibration locations can be arranged in 2x2, 3x3, 4x4, 5x5, 6x6, 7x7, 9x9 or other size grids or other pattern or positional layouts.

Calibration of one or more locations on display 2500 can be determined using a light field metric system that measures errors in a calibration pattern (eg, a checkerboard) projected by the display. Calibration can depend on the location on the display, which is the location at which the display is viewed. For example, the metering system can sweep the eye-proxy camera relative to the display (eg, by relatively panning the camera and display) to simulate the range of positions of the user's eyes. As the camera scans relative to the display, at each sampling point 2602, the metrology system can establish a calibration (correction) to produce a set of calibrations compared to the eye proxy position.

Calibration of a particular display may be stored by the data modules 224, 228 of the wearable display system 200 as a look-up table (LUT) (or other valid data structure). In other embodiments, the analysis model can obtain calibration data from the measurement system and the fit analysis model can be stored by the wearable display system 200. Other modeling or data reference methods can be used to store the calibration. As noted above, the calibration may include spatial and/or color corrections generated for each calibration location of the display (e.g., an embodiment of a 3x3 grid of calibration locations of display 2500 shown in FIG. 26). Note that in various embodiments, in order to obtain calibration, the display is scanned (translated) relative to a fixed camera, the camera is scanned (translated) relative to a fixed display, or the camera and display are scanned (translated) relative to each other.

The field of view (FOV) of the eye agent camera is greater than the FOV of the display In an embodiment, the calibration camera is placed at a plurality of discrete locations relative to the display (eg, at location 2602 as indicated by the dots), and the one or more calibration images will provide sufficient information about the display defects to determine each Calibration of discrete locations. In some such embodiments, the camera can capture the full FOV of the display and may not need to change the orientation (eg, pointing direction) at each position 2602 of the camera. In other embodiments, the camera can be changed (at each location 2602) to obtain additional images to plot the FOV of the display (eg, when the FOV of the camera is less than the FOV of the display).

The calibration position may represent the eye position relative to display 2500. For example, the wearer of display 2500 consistently positions the display such that the wearer's eyes (in the xy plane) are generally near the center of light output coupling element 2509, for example, the wearer's eyes are above position 2602a. Thus, the calibration of position 2602a (near the center of optical element 2509) corresponds to light that propagates substantially perpendicular to display 2500 (eg, substantially along the z-direction) and can be applied to dynamic calibration processor 2610. If the wearer's eyes move up and to the left at position 2602b (near the upper left corner of optical element 2509), calibration of position 2602b can be applied to processor 2510. The eye tracking camera 500 can image the eye (eg, on-the-fly), and the dynamic calibration processor 2510 can use the eye tracking data to determine the position of the eye, select an appropriate calibration (based on the determined eye position), and apply the calibration monitor. In some embodiments, the eye position is determined from the corneal position and the gaze direction. Moreover, in other embodiments, eye orientation (eg, gaze direction) can be determined, and calibration of directional correlation can be used.

Embodiments of the wearable display system 200 can include embodiments of the dynamic calibration system 2600 shown in FIG. For example, the eye tracking camera 500 (described with reference to FIG. 4) can be secured to the frame of the wearable display system 200 and can dynamically measure the wearer's eye posture (eg, eye position or eye direction). Images from camera 500 may be used by dynamic calibration processor 2610 to determine the wearer's eye posture either immediately or near instantaneously. When operating a dynamically calibrated system, the eye tracking camera can notify the dynamic calibration processor 2610 about the current eye posture of the wearer, either immediately or near instantaneously. The dynamic calibration processor 2610 can acquire and apply appropriate calibrations (eg, appropriate calibration LUTs stored in the data modules 224, 228) based on the measured eye gestures (eg, position or orientation). In the event that the wearer does not directly view the stored calibration position or the wearer's eyes are not directly above the calibration position, the dynamic calibration processor may interpolate (or extrapolate) between calibrations close to the calibration position (eg, at least include The calibration position closest to the wearer's eye posture is determined to determine the appropriate calibration to apply to the wearer's current eye posture. Thus, display system 200 (with dynamic calibration system 2600) can correct for defects (space or color) in the display to provide a good quality color image of the wearer. As described herein, in some cases, the calibration depends on the position of the eye relative to the display, and not on the orientation of the eye (eg, the direction of gaze), but this is not a limitation.

Dynamic calibration processor 2610 is software stored in a storage (e.g., data module 224, 228), and software instructions are executed by one or both of processing modules 224, 228 or by controller 450. Thus, continuous adjustment calibration can produce high quality images over a wide range of input motions of the wearer's eyes.

In some embodiments, the calibration is stored in a reduced number of calibration locations (eg, a 2x2 or 3x3 grid) to reduce data storage. As noted above, the dynamic calibration processor can be interpolated or extrapolated to determine the calibration of the eye gesture that is not directly at the stored calibration location.

In some embodiments, the wearable display system 200 uses a single eye tracking camera to measure the pose of the wearer's single eye, and the dynamic calibration processor 2610 infers the pose of the wearer's other eye relative to the display system 200 (because The eyes usually point in the same direction). In other embodiments, the wearable display system 200 uses two eye tracking cameras (one for each eye) and independently measures the pose of each eye. In some embodiments, each display in the wearable system stores a separate calibration (in many cases, there will be two displays, each in front of each eye of the wearer, thus storing two calibrations). In other embodiments, a single calibration (eg, average calibration) is stored and used for all displays in the wearable system 200.

An eye tracking camera (or other type of inward facing imaging system) can image the periocular area of the user's face. The eyelid area may include an area around the eyes and eyes. For example, the eyelid region can include an eye (eg, an eye socket) and an area around the eye. The area around the eyes may include portions such as eyebrows, nose, cheeks, and forehead. The periocular region can have a variety of features, such as the shape of the eyebrows, the corners of the eyes, the features of the eyelids, and the like. In some embodiments, one or more of these features may be represented by key points, point clouds, or other types of mathematical features. The wearable device can identify these features in the image and use these features to determine the wearable display Shows the relative position between the system and the user's face. In some embodiments, the wearable display system 200 can calculate a relative position for each eye separately. For example, when the wearable device has one or two eye cameras, each eye camera is configured to image one eye of the user, the wearable device can calculate a relative position between the left eye and the wearable display system And another relative position between the right eye and the wearable display system. The wearable device can also track the relative positions of the corresponding eyes, respectively. Because the relative position between the left eye and the wearable display system can be different from the relative position between the right eye and the wearable display system (eg, when the wearable system is tilted to one side), the adjustment of the position of the virtual object is presented, The left eye display and the right eye display can be different.

Wearable display systems can use neural networks or visual keypoint techniques to calculate and track eye features, such as Scale Invariant Feature Transform Algorithm (SIFT), Accelerated Robust Feature Algorithm (SURF), Directional FAST, and Rotating BRIEF (ORB) ), Binary Robust Non-Variable Scalable Keypoint Algorithm (BRISK), Fast Retina Keypoint Algorithm (FREAK), etc. In some embodiments, a detector designed specifically for the particular facial feature can be used to track a particular facial feature. For example, various algorithms can be used to individually identify and track eye features, such as eye corners, nose features, mouth angles, and the like. It may be advantageous to track one or more of these periocular features separately, as they are prone to substantial motion when the user expresses himself or is speaking. Detectors associated with these periocular features can take into account the range of mobility. For example, when some facial features move in certain directions and are stable in other directions (eg, the eyebrows tend to move up or down without moving to the left or right). can The wearable system can statistically analyze the movement of facial features. These statistics can be used to determine the likelihood that a facial feature will move in a particular direction. In some embodiments, one or more facial features may or may not be tracked. For example, a wearable display system can ignore eye movements while tracking the position of the eye area.

The wearable display system can also use visual synchronization and map establishment (vSLAM) techniques such as timing Bayesian estimators (eg, Kalman filters, extended Kalman filters, etc.), beam adjustments, etc. to identify and track faces. Features. In some embodiments, the wearable device can be configured to allow for depth perception. For example, the wearable system can construct a dense map that encodes at least a portion of the face based on data retrieved by one or more cameras. The dense map replaces the keypoint map and may include a facial plaque or region that measures the 3D shape of the face. Plaques or regions may be associated with calculating the location of the HMD and the face of the user using techniques such as iterative nearest algorithm or similar algorithm.

In some embodiments, the image acquired by the eye camera may be a low resolution image because the wearable display system 200 may not require high quality images to track eye features. Additionally or alternatively, the resolution of the image obtained from the eye imager may be downsampled relative to its original resolution or resolution used in other applications (eg, eye tracking).

The wearable display system 200 can analyze images obtained by one or two eye cameras and use various techniques to determine the relative position between the display and the user of the display system. The relative position between the display and the user's eyes may be the normal resting position of the display system 200 relative to the user's face. Display system 200 The normal rest position can be determined during the initialization phase of the wearable system. For example, when the user first uses the wearable system, the wearable system can establish a facial model (eg, a map of the user's face) and determine the normal resting position of the display relative to the user's eyes based on the user's eyes. model.

When a user uses the wearable system 200, the wearable system can use various techniques to keep track of the relative position between the display and the user. For example, a wearable device can identify and track visual key points associated with eye contour features. The wearable system can also match a dense map of the area of the face identified in the acquired image relative to the user's face to calculate the position of the display relative to the face.

Thus, various eye tracking or facial imaging techniques (statically or dynamically) can be used to determine the relative position between the user's eyes and the display of the display system. Display system 200 can then select and apply appropriate spatial and/or color calibration to the display based at least in part on the determined relative eye position, as further described herein.

27 is a flow diagram of a method 2700 of dynamically calibrating a display based on eye tracking. Method 2700 can be performed by dynamic calibration system 2600. At block 2710, the user's eyes are tracked to determine the user's eye position relative to the display. For example, camera 500 of display system 2600 can determine the user's eye position. You can track one or two eyes. At block 2720, a calibration based on the determined eye position is taken. At block 2730, calibration is applied to the display to correct for spatial and/or color defects in the display. For example, the dynamic calibration processor 2610 can apply a correction to adjust the properties of the light injected into the waveguide of the display such that the desired beam is Display output. In some cases, light can be injected in slightly different colors or positions or orientations to adjust for display defects. For example, one or more of the RGB color values in the input image projected by the display can be modified by a corresponding RGB calibration (based on the location of the user's eyes) and the modified RGB values are sent to the display for projection. The net effect projection modified RGB values that are not fully displayed are projection images used to generate defects (space and/or color) that at least partially correct the display. In other cases, the actively controlled diffractive optical elements in the waveguide assembly can be adjusted by a dynamic calibration processor such that a beam of light is projected from the display that at least partially corrects defects in the display. In some embodiments, method 2700 acts as a feedback loop that is executed on-the-fly, such that eye tracking camera 500 monitors the user's eyes, and if a change in eye position is detected, a new calibration (for a new eye position) is used Calibrate the display. In some cases, if the change in eye position exceeds a threshold (eg, a portion of the spacing between grids at the calibration location), then a new calibration is applied. Some such embodiments may advantageously provide a calibrated display for viewing by a user. In some embodiments, the method 2700 can be performed occasionally (eg, when the user places the display over the user's eyes) or periodically (eg, to correct for accidental sliding between the display and the user's eyes) .

28 is a flow diagram 2805 of the interaction of a factory calibration system and a dynamic calibration system associated with a particular display. In an embodiment, an eye agent camera calibration system 2810 is used in a factory (manufacture) setup to determine the position-dependent calibration of the display being manufactured. At block 2820, the process analyzes one or more calibration images for each particular display being manufactured and generates for each eye Calibration of the agent location. At block 2830, the calibration is stored in a reservoir associated with a particular display such that each display can take a calibration customized for that particular display during the manufacturing process. For example, the calibration can be stored as a look-up table (LUT) in data module 224 or remote data repository 232 of display 208. A portion of this process flow 2805 can be performed once for each display during manufacturing to provide a customized calibration for each display.

In this embodiment, each display system (eg, wearable display system 200) can perform an instant calibration using the calibration stored at block 2830. For example, the eye tracking system 2840 of the display (which may include the eye tracking camera 500) may determine the position of the cornea of the eye and the direction of gaze of the eye to determine the position of the eye. At block 2850, the display system (eg, via dynamic calibration processor 2610) can extract the appropriate calibration from the storage based on the determined eye position. At block 2860, calibration is applied to the display (eg, via dynamic calibration processor 2610) to correct for spatial and/or chromaticity errors of the display. At block 2870, the wearer is able to view the image projected by the calibrated display. As the wearer's eye position changes relative to the display, the processing flow in the display system also updates the calibration, for example, on the fly.

Although an embodiment of the dynamic calibration system 2600 has been described in the context of a display in a wearable display system, this is not a limitation, and a dynamic calibration system (eg, an eye tracking camera and a dynamic calibration processor) can be used for any display. (Wearable or non-wearable), the calibration is only close to the nominal viewing position (eg, perpendicular to the center of the display). For example, a dynamic calibration system can be used Flat panel displays, liquid crystal displays, light emitting diode displays, microelectromechanical systems (MEMS) displays, and the like.

■ Perform other aspects of image correction

In a first aspect, a computer implemented method for performing image correction on a display is disclosed. The method is under the control of a display calibration system having a computer hardware and a camera, comprising: calibrating a camera, using the camera to capture an image of a light field projected by the display, the light field being associated with a display layer of the display; Part of generating a vector field from the captured image, the vector field including a vector corresponding to a deviation between a projected position of the point of the display layer and an expected position; using the generated vector field, performing: center correction, At least one of an aggregate rotation correction, an aggregation scaling correction, or a spatial mapping; determining, based at least in part on the captured image, a plurality of luminance values corresponding to the plurality of points on the display layer; and using the determined plurality of The luminance value is performed on the display: at least one of luminance flattening or color balance.

In a second aspect, the computer-implemented method of aspect 1, wherein performing center correction comprises: identifying a center point of the projected display layer; and determining a translation vector, wherein the translation vector corresponds to the identified center point and the expected center point position Translation error between.

In a third aspect, the computer-implemented method of aspect 1 or aspect 2, wherein performing the collective rotation comprises: identifying a center point of the projected display layer; and determining an amount of rotation, wherein the amount of rotation corresponds to the projection display layer surrounding the The rotation of the center point minimizes the amount of pixel error between the projected position and the expected position.

The computer-implemented method of any of aspects 1-3, wherein performing the aggregate scaling comprises: identifying a center point of the projected display layer; and determining a scaling amount, wherein the scaling amount corresponds to a projection around the center point The scaling of the display layer minimizes the amount of pixel error between the projected position and the expected position.

The computer-implemented method of any one of aspects 1 to 4, wherein performing the spatial mapping comprises identifying a non-linear transformation to align the projected position of the display layer with the desired position.

The computer-implemented method of any of aspects 1-5, wherein performing luminance flattening comprises: determining a minimum luminance value of the plurality of luminance values; and all luminances of the plurality of luminance values The value is reduced to the minimum luminance value.

The computer-implemented method of any one of aspects 1 to 5, wherein performing luminance flattening comprises: determining a threshold luminance value; and using all of the plurality of luminance values greater than the threshold luminance value The value is reduced to the threshold luminance value.

The computer-implemented method of any of aspects 1-7, wherein performing color balancing comprises: identifying a color cluster associated with the display layer, the color cluster comprising at least one additional display layer; Displaying, in each of the plurality of points on the layer, a luminance value corresponding to a point on the display layer and a luminance value corresponding to a point on the additional display layer; and each of the plurality of luminance values The luminance values are reduced to the lowest luminance values associated with their corresponding points.

The computer-implemented method of any of aspects 1-8, wherein performing the aggregate rotation correction comprises calculating a curl of the vector field.

In a tenth aspect, the computer of any of aspects 1-9 The method of performing, wherein performing the aggregate scaling correction comprises calculating a divergence of the vector field.

The computer-implemented method of any of aspects 1 to 10, wherein the display comprises a light field display.

The computer-implemented method of aspect 11, wherein the light field display comprises a stacked waveguide assembly.

The computer-implemented method of aspect 12, wherein the stacked waveguide assembly comprises two or more waveguides respectively corresponding to two or more depth planes.

In a fourteenth aspect, the computer-implemented method of aspect 13, wherein each depth plane is associated with a red display layer, a green display layer, and a blue display layer.

In a fifteenth aspect, a method of calibrating a display is disclosed. The method is controlled by a display calibration system including a computer hardware, comprising: taking out an image of a calibration pattern projected by the display; determining a spatial distortion between an expected position of the calibration point in the projected light field and an actual display position in the image. The spatial distortion is analyzed to determine a spatial calibration of the display; and the spatial calibration is stored in a non-transitory memory associated with the display.

In a sixteenth aspect, the method of aspect 15, wherein the spatial calibration is corrected for one or more of: an in-plane spatial error or an out-of-plane spatial error.

In a seventeenth aspect, the method of aspect 15 or aspect 16, wherein the spatial calibration corrects one or more of: translational error, Rotational error, scaling error, or pixel distortion.

The method of any of aspects 15-17, further comprising: determining color distortion from the image; analyzing the color distortion to determine a color calibration of the display; and storing the color calibration in the display Associated with non-transitory storage.

In a nineteenth aspect, the method of aspect 18, wherein the color calibration corrects luminance flatness or color uniformity of the display.

■Additional aspects of optical metrology systems

In a twentieth aspect, an optical metrology system for measuring defects in a light field produced by a display is disclosed. The optical metrology system includes: a display for projecting a target light field including a virtual object having an intended focus position; a camera for obtaining an image of the target light field; and a processor programming of executable instructions: fetching corresponding to One or more images of a portion of the light field; analyzing the one or more shadow images to identify a focus position corresponding to the measurement of the virtual object at a focus position; and at least a portion of the focus position and the desired focus based on the measurement The result of the comparison of the positions determines the defects in the light field.

The optical metrology system of aspect 20, wherein the display comprises a light field display.

The optical metrology system of aspect 20 or aspect 21, wherein the display comprises a waveguide stack configured to output light for projecting a virtual object to a particular depth plane.

In a 23rd aspect, the optical according to any of aspects 20-22 A metering system wherein the camera comprises a digital camera with a small depth of field.

In a twenty-fourth aspect, the optical metrology system of aspect 23, wherein the camera has a focus, and the system is operative to scan a focus of the camera over a focus range to obtain the one or more images.

The optical metrology system of any of aspects 20-22, wherein the camera comprises a light field camera.

The optical metrology system of any of aspects 20-25, wherein the virtual object comprises a checkerboard pattern, a geometric pattern, or a random pattern.

The optical metrology system of any of aspects 20-26, wherein the display comprises a plurality of pixels, and the target light field corresponds to less than a subset of all illuminated pixels.

The optical metrology system of any of aspects 20-27, wherein the measured in-focus position comprises a depth of focus.

The optical metrology system of aspect 28, wherein the measured focus position further comprises a lateral focus position.

In a 30th aspect, the optical metrology system of aspect 29, wherein the determined defect is based at least in part on an error vector between an expected focus position and a measured focus position.

The optical metrology system of any of aspects 20-30, wherein the determined defect comprises a spatial defect.

The optical metrology system of any of aspects 20-31, wherein the determined defect comprises a color defect.

The optical metrology system of any of aspects 20-32, wherein the processor is further programmed to determine, at least in part, an error correction of the display based on the determined defect.

In a 34th aspect, a method for measuring a defect in a light field is disclosed, the method comprising: extracting one or more images of a light field corresponding to a portion of a projection of the display, the partial light field having an expected focus position Analyzing the one or more images to confirm the measured in-focus position and corresponding to the in-focus position of the light field; and determining a defect in the light field based at least in part on the comparison of the measured in-focus position and the expected in-focus position .

In a 35th aspect, the method of aspect 34, comprising scanning a focus of the camera to obtain the one or more images.

In a 36th aspect, the method of aspect 34, comprising obtaining one or more images using a light field camera.

The method of any of aspects 34-36, further comprising projecting a light field image comprising a checkerboard pattern.

The method of any of aspects 34-37, further comprising determining, at least in part, an error correction of the light field based on the determined defect.

■ Calibrate other aspects of the display

In a 39th aspect, a calibration system for a display is provided. The calibration system includes: a camera for acquiring an image of the display; and a hardware processor in communication with the camera, the hardware processor configured to: receive an image of the display; determine calibration of the display; Calibration stored in relation to the display Connected to the storage.

In a 40th aspect, the calibration system of aspect 39, wherein the calibration comprises spatial calibration to correct for spatial defects in the display.

In a 41st aspect, the calibration system of aspect 39, wherein the calibrating comprises using chromaticity calibration to correct for color defects in the display.

The calibration system of any of aspects 39-41, wherein the display comprises a plurality of pixels in a field of view, and wherein to determine the calibration, the hardware processor is programmed to: determine global Transforming parameters that are independent of pixels in the field of view of the display; and determining local transform parameters that depend on pixels in the field of view of the display.

In a 43rd aspect, the calibration system of aspect 42, wherein the global transformation parameter comprises one or more non-linear gamma corrections.

In a 44th aspect, the calibration system of aspect 42 or aspect 43, wherein the local transformation comprises a linear function.

The calibration system of any of aspects 39 to 44, wherein the hardware processor is programmed to use the feedback from the image captured by the camera to solve the problem in an iterative manner in order to determine the calibration calibration.

The calibration system of any of aspects 39 to 45, wherein the calibration comprises color calibration, the display includes a plurality of color gradations that provide white points, and the calibration is determined, the hardware processor Programming is used to adjust the intensity of the tone scale such that the white point is substantially uniform across the field of view of the display.

In a 47th aspect, the calibration system according to aspect 46, wherein Determining the calibration, the hardware processor programming is to: resolve a first gamma correction that maps the gradation sent to the display to the first intermediate color representation; and resolves mapping the first intermediate color representation to the second intermediate color A one-pixel correlation coupling function of the expression; and a second gamma correction that maps the second intermediate color representation to the gradation recorded by the camera.

In a 48th aspect, the calibration system of aspect 47, wherein the hardware processor is programmed to preferentially resolve the pixel correlation coupling function and to solve the first gamma correction and the second gamma correction.

The calibration system of any of aspects 39 to 48, wherein the display comprises a light field display.

In a 50th aspect, the calibration system of any of aspects 39 to 49, wherein the display comprises a plurality of waveguide stackable waveguide assemblies.

In a 51st aspect, the calibration system of any of aspects 39 to 50, wherein the display is configured for a wearable display system.

In a 52nd aspect, a method for calibrating a display is provided. The method, under the control of a dynamic calibration system performed by a computer hardware, includes: removing a calibration of the display; determining, based at least in part on the extracted calibration, a calibration applied to the display to at least partially correct defects in the display; And applying the correction to the display.

The method of aspect 52, wherein the taking out the calibration comprises color calibration.

In a 54th aspect, the method of aspect 53, wherein the display The device includes a plurality of pixels in the field of view, and the color calibration includes a plurality of pixel independent nonlinear gamma corrections and pixel dependent coupling functions.

The method of any of aspects 52 to 54, wherein the display comprises a light field display.

In a 56th aspect, a wearable display is provided, comprising: a display; a memory configured to store the calibration; and a hardware processor programming, the method of performing any of aspects 14 to 17, and The non-transitory memory communication link.

■Other aspects of the calibration mode

In an 57th aspect, an optical system for calibrating a light field generated by a display, comprising: a display for projecting a target light field including a calibration pattern containing feature points; and a camera for obtaining an image of the target light field; a processor that programs executable instructions for: for each of the plurality of locations: causing the display to project the calibration pattern at a location in the plurality of locations; such that the camera obtains an image of the projected calibration pattern; Calculating a distortion of the feature point, wherein the distortion corresponds to an error between the expected position of the feature point and the measured position of the feature point or between a desired luminance or chromaticity of the calibration pattern and a measured brightness or chromaticity of the calibration pattern An error; and reacting to the determination of the next of the plurality of locations, translating the calibration pattern to display at the next location.

The optical system of aspect 57, wherein the calibration pattern comprises a checkerboard pattern.

In an 59th aspect, the optical system of aspect 57, wherein The number of locations corresponds to the number of pixels in the checkbox of the checkerboard pattern.

The optical system of aspect 57, wherein the calibration pattern comprises a single pixel pattern.

In the 61st aspect, the optical system of aspect 60, wherein the number of the plurality of locations corresponds to the number of pixels displayed.

The optical system of any of aspects 57-61, wherein the processor is further programmed to generate a distortion map based at least in part on the calculated distortion corresponding to the plurality of locations.

The optical system of any of aspects 57-62, wherein the processor is further programmed to determine an error correction for the display based at least in part on the calculated distortion corresponding to the plurality of locations.

The optical system of any of aspects 57-63, wherein the display comprises separate red, green, and blue color layers.

The optical system of any of aspects 57-64, wherein the display comprises a light field display.

The optical system of aspect 65, wherein the light field display comprises a stacked waveguide assembly.

The optical system of aspect 66, wherein the stacked waveguide assembly comprises two or more waveguides respectively corresponding to two or more depth planes.

The optical system of any of aspects 57-67, wherein the calculated distortion further comprises luminance distortion or color distortion.

In a 69th aspect, a method for calibrating a light field produced by a display is provided. The method includes, for each of a plurality of locations: causing a display to project a calibration pattern at a location in a plurality of locations; causing the camera to obtain an image of the projected calibration pattern; calculating a distortion of the feature point, wherein the distortion Corresponding to an error between the expected position of the feature point and the measured position of the feature point or an error between the expected luminance or chrominance of the feature point and the measured luminance or chrominance; The determination of the next position in the position shifts the calibration pattern to display at the next position.

The method of aspect 69, wherein the calibration pattern is a checkerboard pattern.

The method of aspect 70, wherein the number of the plurality of locations corresponds to the number of pixels in the checkbox of the checkerboard pattern.

The method of aspect 69, wherein the calibration pattern comprises a single pixel pattern, a random pattern, or a geometric pattern.

The method of aspect 72, wherein the number of the plurality of locations corresponds to the number of pixels displayed.

The method of any one of aspects 69-73, further comprising generating a distortion map based at least in part on the calculated distortion corresponding to the plurality of locations.

The method of any one of aspects 69-74, further comprising determining an error correction for the display based at least in part on the calculated distortion corresponding to the plurality of locations.

The optical system of any of aspects 69-75, wherein the display comprises separate red, green, and blue color layers.

The optical system of any of aspects 69-76, wherein the display comprises a light field display.

The optical system of aspect 77, wherein the light field display comprises a stacked waveguide assembly.

In the 79th aspect, the optical system of aspect 78, wherein the stacked waveguide assembly comprises two or more waveguides respectively corresponding to two or more depth planes.

The optical system of any of aspects 69-79, wherein the calculated distortion further comprises luminance distortion or chrominance distortion.

■ Perform other aspects of dynamic calibration

In an 81st aspect, a display system is provided. The display system includes: an eye tracking camera; a display; a non-transitory data storage for storing a plurality of calibrations of the display, each of the plurality of calibrations being associated with a calibration position of the display; and a hardware processor associated with the eye tracking camera communication, the display, and the non-transitory data storage, the hardware processor being programmed to: determine an eye-related position of the display for the user of the display; Taking, at least in part, one or more of the plurality of calibrations at an eye position; determining, at least in part, based on one or more of the plurality of calibrations, applying a calibration to the display to at least partially correct the display Defects; and apply calibration to the display.

In a 82nd aspect, the display system according to aspect 81, wherein the number of calibration positions is 2, 3, 4, 5, 6, 7, 8, 9, or more.

The display system of aspect 81 or aspect 82, wherein the calibration position is distributed across the display in a grid.

In a 84th aspect, the display system of aspect 83, wherein the grid comprises a 2x2, 3x3, 5x5 or 9x9 grid.

The display system of any one of aspects 81 to 84, wherein the one or more of the plurality of calibrations comprises a calibration associated with a calibration position that is closest to the eye position.

The display system of any one of aspects 81 to 85, wherein the hardware processor is programmed to interpolate or externally in the one or more of the plurality of calibrations in order to determine the correction Plug in.

The display system of any of aspects 81 to 86, wherein each of the plurality of calibrations corrects for a spatial defect of the display, a color defect of the display, or both a spatial defect and a color defect.

The display system of any of aspects 81 to 87, wherein the display comprises a light field display.

The display system of any of aspects 81 to 88, wherein the display comprises a stackable waveguide assembly comprising a plurality of waveguides.

The display system of any one of aspects 81 to 89, wherein the display is configured as a wearable display system.

In a 91st aspect, a head mounted display is provided, including aspects A display system according to any one of 81 to 90.

In a 92nd aspect, a method for calibrating a display is provided. The method, under the control of a dynamic calibration system executed by a computer hardware, includes: determining an eye position of a user of the display; taking out, based at least in part on the determined eye position, a calibration for the display, the calibration and proximity A determined calibration position of the eye position is associated; determining, based at least in part on the extracted calibration, a calibration applied to the display to at least partially correct defects in the display; and applying the correction to the display.

In a 93rd aspect, the method of aspect 92, wherein the removing the calibration comprises selecting one or more calibrations from the plurality of calibrations, wherein each calibration is associated with a different calibration location relative to the display.

The method of aspect 93, wherein the calibration position is disposed in a grid spanning the display.

The method of any one of aspects 92 to 94, wherein the calibration is for correcting a spatial defect of the display, a color defect of the display, or both the spatial defect and the color defect.

The method of any one of aspects 92 to 95, wherein determining the correction comprises interpolating or extrapolating between one or more calibrations associated with a calibration position proximate the eye gesture.

The method of any one of aspects 92 to 96, wherein the display comprises a light field display.

In a 98th aspect, a head mounted display is provided, including eye tracking The trace system and a hardware processor are programmed to perform the method of any of aspects 92 to 97.

■Additional aspects of optical metrology systems

In a 99th aspect, an optical metrology system for measuring defects in a light field produced by a display is provided. The optical metrology system includes: a display for projecting a target light field including a virtual object having an intended focus position; a camera for obtaining an image of the target light field; and a hardware processor programmed with executable instructions Taking one or more images corresponding to a portion of the light field; analyzing the one or more images to identify a focus position corresponding to the measurement of the virtual object at a focus position; and at least partially comparing the measured focus position And the expected focus position to determine a defect in the light field.

The optical metrology system of aspect 99, wherein the display comprises a waveguide stack configured to output light to project the virtual object to at least one depth plane.

The optical metrology system of any of aspects 99 to 100, wherein the camera comprises a digital camera having a small depth of focus.

The optical metrology system of aspect 101, wherein the camera has a focus, and the system is configured to scan a focus of the camera over a focus range to obtain the one or more images.

The optical metrology system of any of aspects 99 to 102, wherein the camera comprises a light field camera.

In a 104th aspect, the light according to any one of aspects 99 to 103 A metrology system, wherein the virtual object comprises a checkerboard pattern, a geometric pattern, or a random pattern.

The optical metrology system of any of aspects 99 to 104, wherein the display comprises a plurality of pixels, and the target light field corresponds to a subset of less than all of the illuminated pixels.

The optical metrology system of any of aspects 99 to 105, wherein the measured in-focus position comprises a depth of focus.

In a 107th aspect, the optical metrology system of aspect 106, wherein the measured focus position further comprises a lateral focus position.

The optical metrology system of any one of clauses 99 to 107, wherein the determined defect is based at least in part on an error vector between the expected focus position and the measured focus position.

The optical metrology system of any one of aspects 99 to 108, wherein the hardware processor is further programmed to determine an error correction for the display based at least in part on the determined defect.

The optical metrology system of any one of clauses 99 to 109, wherein the hardware processor is further programmed to apply to the display to camera pixel mapping to transmit pixel values of the display to pixel values of the camera.

The optical metrology system of aspect 110, wherein the display to camera pixel mapping comprises: a first gamma correction, mapping the gradation of the display to a first intermediate color representation; a pixel independent coupling function, which The first intermediate color representation is mapped to the second intermediate color representation; and a second gamma The horse correction maps the second intermediate color representation to the color scale recorded by the camera.

The optical metrology system of any of aspects 99 to 111, wherein the determined defect comprises a spatial defect.

In an eleventh aspect, the optical metrology system of aspect 112, wherein the spatial defect comprises one or more of an in-plane translation, rotation, scaling or distortion error or an out-of-plane or focus depth error.

The optical metrology system of any of aspects 99 to 113, wherein the determined defect comprises a color defect.

In an eleventh aspect, the optical metrology system of aspect 114, wherein the color defect comprises one or more of luminance flatness or color uniformity error associated with a color displayable by the display.

In a 116th aspect, an optical metrology system for performing image correction on a display is provided. The system includes a camera for capturing an image of a light field projected by the display, the light field being associated with a display layer of the display, and a hardware processor programming executable instructions for: based at least in part on capturing the camera An image generation vector field, the vector field comprising a vector corresponding to a deviation between a projected position of the point of the display layer and an expected position; computing, based at least in part on the vector field, at least one of: a center correction, an aggregation Rotation correction, an aggregate zoom correction or spatial mapping for a display; computing, based at least in part on an image captured by the camera, a luminance value corresponding to a plurality of points on the display layer; and calculating, based at least in part on The determined luminance value, a luminance flat correction or a one color balance correction, is used for the display.

In an eleventh aspect, the optical metrology system of aspect 116, wherein the display layer of the display comprises a color layer or a depth layer.

The optical metrology system of any of aspects 116 to 117, wherein the camera comprises a light field camera or a digital camera having a small depth of focus.

The optical metrology system of any one of aspects 116 to 118, wherein the hardware processor is programmed to determine the identified center point corresponding to the projection display layer and the projection for calculating the center correction The translation vector of the translation error between the center points of the display layer, as well as the expected center point position.

The optical metrology system of any one of aspects 116 to 119, wherein the hardware processor is programmed to determine an amount of rotation corresponding to a rotation of the projection display layer about the center point in order to calculate the aggregate rotation correction, For example, the amount of pixel error between the projected position and the expected position is reduced or minimized.

In an eleventh aspect, the optical metrology system of any one of aspects 116 to 120, wherein the hardware processor is programmed to calculate the curl of the vector field in order to calculate the total rotation correction.

The optical metrology system of any one of aspects 116 to 121, wherein the hardware processor is programmed to determine a scaling corresponding to the center of the projection display layer in order to calculate the aggregate scaling correction The amount of quantization, the amount of pixel error between the projected position and the expected position is reduced or minimized.

The optical metrology system of any one of aspects 116 to 122, wherein the hardware processor is Programming to calculate the divergence of the vector field.

The optical metrology system of any one of aspects 116 to 123, wherein the hardware processor is programmed to determine a non-linear transformation to align the projected position of the display layer with the intended location for calculating the spatial mapping .

The optical metrology system of any one of aspects 116 to 124, wherein the hardware processor is programmed to: determine a threshold luminance value; and calculate a greater than threshold luminance value for calculating a luminance flat correction The luminance values are reduced to a total amount of threshold luminance values.

The optical metrology system of any one of aspects 116 to 125, wherein the hardware processor is programmed to: identify a color cluster associated with the display layer, the color cluster comprising at least An additional display layer; for each point of the display layer, comparing the luminance value corresponding to the point on the display layer with the luminance value corresponding to the point on the additional display layer; and calculating each luminance value Decrease a total amount of the lowest luminance value associated with its corresponding point.

■ Dynamic display of other aspects of calibration

In a 127th aspect, a display system is provided. The display system includes: an eye tracking camera; a display; a non-transitory data storage configured to store a plurality of calibrations for the display, each of the plurality of calibrations being associated with a calibration position relative to the display And a hardware processor in communication with the eye tracking camera, the display, and the non-transitory data storage, the hardware processor being programmed to: determine, based on information from the eye tracking camera, An eye position relative to a user of the display; taking out, based at least in part on the determined eye position, one or more of the plurality of calibrations; calculating, based at least in part on the one or more of the plurality of calibrations A calibration is applied to the display to at least partially correct defects in the display; and the correction is applied to the display.

In a 128th aspect, the display system according to aspect 127, wherein the number of calibration positions is 2, 3, 4, 5, 6, 7, 8, 9, or more.

The display system of any of aspects 127 to 128, wherein the calibration position is distributed across the display in a grid.

In a 130th aspect, the display system of aspect 129, wherein the grid comprises a 2x2, 3x3, 5x5 or 9x9 grid.

The display system of any of aspects 127 to 130, wherein one or more of the plurality of calibrations comprises a calibration associated with a calibration position that is closest to an eye position.

The display system of any one of aspects 127 to 131, wherein the hardware processor is programmed to interpolate or extrapolate between one or more of the plurality of calibrations, in order to calculate the correction, at least Based in part on the calibration position of the one or more of the plurality of calibrations and the determined eye position.

The display system of any one of aspects 127 to 132, wherein the display comprises a first display associated with the first eye of the user and a second associated with the second eye of the user A second display, and the hardware processor is programmed to determine an eye position of the user relative to the first display, and to apply the determined eye position to calculate a correction for the second display.

The display system of any one of aspects 127 to 133, wherein the display comprises a first display associated with the first eye of the user and associated with a second eye of the user A second display, and at least some of the plurality of calibrations represent an average calibration for the first display and the second display.

The display system of any of aspects 127 to 134, wherein the display comprises a light field display.

The display system of any of aspects 127 to 135, wherein the display comprises a stackable waveguide assembly comprising a plurality of waveguides.

The display system of any one of aspects 127 to 136, wherein the display is configured as a head mounted wearable display system.

The display system of any one of aspects 127 to 137, wherein each of the plurality of calibrations is for correcting a spatial defect of the display, a color defect of the display, or the spatial defect and Both of these color defects.

The display system of aspect 138, wherein the spatial defect comprises one or more of an in-plane translation, rotation, scaling or distortion error or an out-of-plane or focus depth error.

In a 140th aspect, the display system of aspect 138, wherein the color defect comprises one or more of luminance flatness or color uniformity error associated with a color displayable by the display.

In a 141st aspect, a method for calibrating a display is provided. The method, under the control of a dynamic calibration system executed by a computer hardware, includes: determining an eye position of a user of the display; taking out, based at least in part on the determined eye position, calibration of the display, wherein the calibration is based on an association The calibration position and the determined eye position are selected; calculating, based at least in part on the extracted calibration, a calibration applied to the display to at least partially correct defects in the display; and applying the correction to the display.

The method of aspect 141, wherein the removing the calibration comprises selecting one or more calibrations from the plurality of calibrations, wherein each calibration is associated with a different calibration location relative to the display.

In a 143th aspect, the method of aspect 142, wherein the calibration position is disposed in a grid spanning the display.

The method of any one of aspects 142 to 143, wherein the calculating the correction comprises one or more of the plurality of calibrations based on the associated calibration position of the one or more of the plurality of calibrations Interpolate or extrapolate and determine the position of the eye.

The method of any one of aspects 141 to 144, further comprising removing an image of a user's eyes of the display and determining an eye position based at least in part on the image of the eye.

The method of any one of aspects 141-145, wherein calculating the calibration comprises calibrating a spatial defect of the display, a color defect or a spatial defect of the display, and a color defect.

In a 147th aspect, a wearable display system is provided Included: an inward facing imaging system; a display; a non-transitory data store, to store a plurality of calibrations of the display, each of the plurality of calibrations being associated with a calibration location relative to the display; and a A hardware processor is in communication with the inward facing imaging system, the display, and the non-transitory data storage, the hardware processor being programmed to: determine, use the inward facing imaging system, relative to a user of the display An eye position of the display; calculating, based at least in part on the determined eye position and one or more of the plurality of calibrations, a calibration applied to the display to at least partially correct one or more spaces in the display Defects or color defects in the display; and apply this correction to the display.

In a 148th aspect, the wearable display system of aspect 147, wherein the hardware processor is programmed to apply the correction via a feedback loop that monitors changes in eye position.

The 149th aspect, the wearable display system of any one of clauses 147 to 148, wherein the hardware processor is programmed to determine a change in an eye position relative to a previous eye, and if the change exceeds a threshold, calculate the Correction.

In a 150th aspect, the wearable display system of any of aspects 147 to 149, wherein the spatial defect comprises one or more in-plane translational, rotational, scaling or distortion errors or out-of-plane or depth-of-focus errors.

The 151st aspect, the wearable display system of any of aspects 147 to 150, wherein the color defect comprises one or more luminance flatness or color uniformity errors associated with a color displayable by the display.

■Conclusion

Each of the processes, methods, and algorithms described and/or depicted in the present disclosure can be executed by one or more physical computing systems, hardware computer processors, application specific integrated circuits (ASICs). Groups, and/or electronic hardware configured to perform specific and specific computer instructions are fully implemented or partially automated. For example, a computing system can include a general purpose computer (eg, a server), a special purpose circuit, or the like, programmed with a particular computer instruction or a special purpose computer. Code modules can be compiled and linked into an executable program, installed in a dynamic link library, or written in an interpreter language. In some implementations, certain operations and methods may be performed by circuitry that is specific to a given function.

Moreover, some of the functions implemented by the present invention are mathematically, computationally, or technically complex, such that a dedicated hardware or one or more physical computing devices (with appropriate dedicated executable instructions) are required to perform functions, for example, The results are provided due to the volume or complexity of the calculations involved or substantially instantaneously. For example, a video may include many frames, each having millions of pixels, and in particular requiring programmed computer hardware to process the video data to provide a desired image processing task or application for a commercially reasonable amount of time.

Code modules or any type of data can be stored on any type of non-transitory computer readable media, such as physical computer storage, including hard drives, solid state memory, random access memory (RAM), read-only memory Body (ROM), optical disk, volatile or non-volatile storage, combinations thereof and/or the like. Methods and modules (or data) can also be used as generated data signals (eg, as part of a carrier or other analog or digitally transmitted signal) in a variety of computer readable transmission media Up-conversion, including wireless-based and wired/cable-based media, and can take various forms (eg, as part of a single or multiplexed analog analog signal, or as multiple discrete digital packets or frames). The results of the disclosed processes or process steps may be stored, permanently or otherwise stored in any type of non-transitory tangible computer memory, or may be transferred via a computer readable transmission medium.

Any process, block, state, step, or functionality in the flowcharts described herein and/or depicted in the drawings should be understood to potentially represent a code module, a code segment, or a code portion that includes a particular function. One or more executable instructions (eg, logic or arithmetic) or steps. The various processes, blocks, states, steps or functions may be combined, rearranged, added, deleted, modified or otherwise altered in various ways. In some embodiments, some or all of the functions described herein may also be performed in additional or different computing systems or code modules. The methods and processes described herein are also not limited to any particular sequence, and the blocks, steps or states associated therewith may be performed in other sequences suitable for, for example, serial, parallel, or in some other manner. Tasks or events may be added to or removed from the disclosed embodiments. Moreover, the separation of various system components in the embodiments described herein is for illustrative purposes and should not be construed as requiring such separation in all embodiments. It should be understood that the described program components, methods, and systems can generally be integrated together in a single computer product or packaged into multiple computer products. Many implementation changes are possible.

Processes, methods, and systems can be implemented in a network (or distributed) computing environment. The network environment includes enterprise-wide computer networks, intranets, and local areas. Network (LAN), wide area network (WAN), personal area network (PAN), cloud computing network, crowdsourcing computing network, internet and global information network. The network can be a wired or wireless network or any other type of communication network.

The systems, methods, and devices of the present disclosure each have a number of innovative aspects, and any single one is not the only reason for the desired attributes disclosed herein. The various features and processes described above can be used independently of one another or can be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. The various exemplary embodiments of the invention are apparent to those skilled in the art, and the general principles defined by the invention may be applied to other embodiments without departing from the spirit and scope of the invention. Therefore, the claims are not intended to be limited to the embodiments shown herein, but are to be accorded

Certain combinations of features described in this specification in the context of separate embodiments are implemented in a single embodiment. Conversely, various features that are described in the context of a single embodiment can be implemented separately in various embodiments or in any suitable sub-combination. Moreover, although features may be described above as being functional in certain combinations, even initially claimed, one or more features from the claimed combination may be deleted from the combination in some cases and may be The claimed combination points to a sub-combination or a variant of a sub-combination. Moreover, no single feature or combination of features is essential or indispensable for each embodiment.

Conditional language as used herein, such as "may", "may", "such as" and the like, unless otherwise explicitly stated or otherwise understood in the context of use. It is intended, therefore, that the claims, the claims, and the claims Thus, such conditional language is generally not intended to imply that features, elements and/or steps are required in any manner for one or more embodiments, or one or more embodiments must be included for use with or without operator input. These features, elements and/or steps are determined to be or will be performed in any particular embodiment. The terms "including", "comprising", "having", etc. are meant to be used in an open-ended manner and do not exclude additional elements, features, acts, operations, etc. Furthermore, the term "or" is used in its <RTI ID=0.0> </ RTI> <RTIgt; </ RTI> <RTIgt; </ RTI> <RTIgt; </ RTI> <RTIgt; In addition, the articles "a", "an" and "the"

As used herein, a phrase referring to "at least one of" a list of items refers to any combination of these items, including a single member. As an example, "at least one of A, B or C" is intended to cover: A, B, C, A and B, A and C, B and C, and A, B and C. In addition, unless otherwise specified, the conjunction language, such as the phrase "at least one of X, Y, and Z", is generally understood in the context of the commonly used, to express that the item, item, etc. may be X, Y, or Z. . Thus, such conjunction language is generally not intended to imply that certain embodiments require at least one of X, at least one of Y, and at least one of Z.

Similarly, although operations are depicted in a particular order in the figures, this should be understood as not necessarily being performed in the specific order shown or in a sequential order. Such an operation, or the need to perform all of the illustrated operations, can achieve the desired result. Still further, the illustration may schematically depict the process of another example in the form of a flow chart. However, other operations not described may be introduced by the exemplary methods illustrated. For example, one or more additional operations can be any of the operations shown before, after, simultaneously, or between. Moreover, in other embodiments, operations may be rearranged or reordered. In some cases, multiple as well as parallel processing may be advantageous. Moreover, the separation of the various system elements in the above-described embodiments should not be construed as requiring such separation in all embodiments, but it should be understood that the described program elements and systems can generally be integrated into a single unit. The software product or package is a plurality of software products. In addition, other embodiments are within the scope of the following claims, and in some cases the operations recited in the claims can be performed in a different order and still obtain the desired results.

1800‧‧‧Measurement system

1802‧‧‧ display

1804‧‧‧Light

1806‧‧‧ camera

1808‧‧‧ Controller

Claims (28)

An optical metrology system for detecting a light field defect generated by a display, the optical metrology system comprising: a display for projecting a target light field including a virtual object having an intended focus position; and a camera for capturing the target An image of the light field; a hardware processor programmed to execute an instruction: extracting one or more images corresponding to a portion of the light field; analyzing the one or more images to identify that the virtual object is in a focus position The measured focus position; and at least a portion is to determine a light field defect based on a comparison of the measured focus position and the expected focus position. The optical metrology system of claim 1 wherein the display comprises a waveguide stack for outputting a light source to project the virtual object to at least one depth plane. The optical metrology system of claim 1 wherein the camera comprises a digital camera having a small depth of focus. The optical metrology system of claim 3, wherein the camera has a focus point and the system is to scan a focus of the camera over a focus range to obtain one or more images. The optical metrology system of claim 1 wherein the camera comprises a light field camera. The optical metrology system of claim 1, wherein the virtual object comprises a checkerboard pattern, a geometric pattern, or a random pattern. The optical metrology system of claim 1 wherein the display comprises a plurality of pixels and the target light field corresponds to less than a subset of the plurality of illuminated multi-pixels. The optical metrology system of claim 1, wherein the measured in-focus position comprises a depth of focus. The optical metrology system of claim 1 wherein the measured in-focus position further comprises a lateral focus position. The optical metrology system of claim 1, wherein the defect of the determined light field is at least a portion based on an error vector between the measured in-focus position and the expected in-focus position. The optical metrology system of claim 1 wherein the hardware processor programming is at least in part determining an error correction of the display based on the determined defect. The optical metrology system of claim 1, wherein the hardware processor is further programmed for pixel mapping of the display to the camera and transmitting pixel values of the display to pixel values of the camera. The optical metrology system of claim 12, wherein the display to the camera pixel mapping comprises: a first gamma correction, mapping the color gradation of the display to a first intermediate color representation; a pixel correlation coupling function mapping the first intermediate color representation to a second intermediate color representation; and a second gamma correction mapping the second intermediate color representation to the gradation recorded by the camera. The optical metrology system of any of claims 1 to 13, wherein the determined defect comprises a spatial defect. The optical metrology system of claim 14, wherein the spatial defect comprises one or more in-plane transitions, rotations, scaling or distortion errors or an out-of-plane or a focus depth error. The optical metrology system of any of claims 1 to 13, wherein the determined defect comprises a color defect. The optical metrology system of claim 16, wherein the color defect comprises one or more luminance flattening or a display can display a color-associated color uniformity error. An optical metrology system for detecting a defect in a light field generated by a display, and performing image correction on a display, the optical metrology system comprising: a camera for capturing a light field image projected by a display, the light field Associated with a display layer of the display; a hardware processor programming to execute an instruction: at least a portion is to generate a vector field based on the image captured by the camera, the vector field including a projected position and an expected position of the point of the display layer The vector corresponding to the deviation; At least a portion is to calculate at least one of a center correction, a focus rotation correction, an aggregation zoom correction, or a spatial mapping of the display according to the vector field; at least a portion is based on the image captured by the camera, calculated and displayed on the display layer And a plurality of corresponding luminance values; and at least a portion is to calculate a luminance flat correction or a color balance correction of the display according to the determined luminance value. The optical metrology system of claim 18, wherein the display layer of the display comprises a color layer or a depth layer. The optical metrology system of claim 18, wherein the camera comprises a light field camera or a digital camera having a smaller depth of focus. The optical metrology system of claim 18, wherein for calculating a center correction, the hardware processor is programmed to determine a translation vector corresponding to a translation error between a center point identified by the projection display layer and an expected center point position . The optical metrology system of claim 18, wherein the hardware processor is programmed to determine an amount of rotation corresponding to rotation of the projection display layer about a center point in order to calculate an aggregate rotation correction such that the projection position and the expected position are reduced. Or minimized. The optical metrology system of claim 18, wherein the hardware processor is programmed to calculate the curl of the vector field in order to calculate the aggregate rotation correction. The optical metrology system of claim 18, wherein the aggregation is calculated A scaling correction is programmed to determine an amount of scaling corresponding to the projected display layer scaling about the center point such that the projected position and the expected position are reduced or minimized. The optical metrology system of claim 18, wherein to calculate the aggregate scaling correction, the hardware processor is programmed to calculate a divergence of the vector field. The optical metrology system of claim 18, wherein to calculate the spatial map, the hardware processor is programmed to determine a non-linear transition and to align the projected position of the display layer with an expected position. The optical metrology system of claim 18, wherein to calculate the luminance flat correction, the hardware processor is programmed to: determine a threshold luminance value; and calculate each luminance value that is greater than a threshold luminance value, to a threshold The number of luminance values. The optical metrology system of claim 18, wherein to calculate the color balance correction, the hardware processor is programmed to: identify a color aggregate associated with the display layer, the color aggregate comprising at least one additional display layer; for the display layer Each point of the comparison, comparing a luminance value corresponding to a point on the display layer and a luminance value corresponding to a point on the additional display layer; and calculating to reduce each luminance value to a lowest luminance value associated with the corresponding point quantity.