KR20190129899A - Precision multi-view display - Google Patents

Abstract

Precision multi-view (MV) display systems can accurately and simultaneously display different content to different viewers over a wide field of view. The MV display system may include features that allow individual MV display devices to be tiled easily and efficiently to form larger MV displays. The graphical interface allows the user to graphically specify the viewing area and to associate content viewed in that area in a simple way. The calibration procedure enables specification of the content in the correct viewing position.

Description

Precision multi-view display

TECHNICAL FIELD The present invention relates to a multi-view (MV) display system, and more particularly to an scalable precision MV display system capable of easily providing any (eg, different) content at a designated location.

The patent or application file includes at least one drawing executed in color. A copy of the patent application publication with this patent or color drawing (s) will be provided by the office upon request and payment of the necessary cost.

Multi-view displays simultaneously provide different content to different viewers based on the position of each viewer relative to the display. The novel lenticular card is a simple example of a multi-view system. The lenticular card can reveal different images when viewed from different angles. A lenticular card uses a series of cylindrical lenslets placed over a stripe of content to send each stripe of content in a unique angular range. The complete image is formed by having stripes from a single image placed in the correct position below the lenslet. The stripe image can be provided in a printed sheet or flat panel display.

26 illustrates the basic operation of the lenticular display system 2700. Lenticular display system 2700 includes two cylindrical lenses or lenticulars 2702, which are shown in cross-section and placed over stripe array 2704 of content. An array of stripe 2704 of content includes content stripe 1, 2, 3, 4, A, B, C, and D. In the viewing area 4D, the viewer will see an image of the stripe 4 next to the stripe D. At the same time, the viewer of the viewing area 3C will see the image of the stripe 3 next to the stripe C. Thus, separate images can be provided simultaneously to different viewers at different locations.

There is a significant limitation with the display system 2700. The viewer in the viewing zone 3D will see the stripe-3 portion of the zone 3C image and the stripe-D portion of the zone 4D image. Far from an array of stripes of content 2704, zones 4D, 3C, 2B, and 1A are wide, respectively. Close to the array of stripes of content 2704, viewers of zones 3D, 2C, and 1B can see a combination of some of the multiple images intended for zones 4D, 3C, 2B, and 1A. . When designing a printed lenticular display, it is necessary to know the expected viewing distance so that the image stripes can be arranged to provide a consistent image in the intended viewing area, as opposed to providing a combination of some of the multiple images. For electronic displays, stripes can be dynamically assigned to produce a consistent image at the location where the viewer is currently located.

Attempts to increase the number of viewing zones by increasing the number of stripes under each lenticule quickly increase the number of distinct viewing zones and reduce the size of each. Targeting images to specific locations is becoming increasingly challenging. Due to these and other limitations, current multi-view displays are typically limited to a very small number of viewing areas. Two to four viewing zones are common and commercial units intended for three-dimensional (3D) viewing applications tend to be maximal with a few dozen small stripes per lenslet.

Flat panel electronic display pixels typically consist of spatially distinct sub-pixels (eg, red, green, and blue sub-pixels) to produce a range of colors. These techniques rely on the limited ability of the human eye to solve these levels of detail. Unfortunately, lenticules act as a magnifying glass and can make the sub-pixels very clear. For example, if the red sub-pixels are aligned in a stripe below the lenticule, the viewer at the location where it is photographed can see in red in the region of the reticule. To overcome the sub-pixel problem, it can be tilted with respect to the lower panel to cover different color sub-pixels along the long axis of the lens. Since cylindrical lenticules are not enlarged in their dimensions, color mixing works properly.

Lecticular displays using cylindrical lenses are limited to creating views in a single dimension, for example, strictly horizontally or strictly vertically. So-called "Dot" or "Fly Eye" lenticulars use two-dimensional lens arrays to direct content in both dimensions. Unfortunately, because both dimensions are enlarged, there is no equivalent trick to tilt the lenticule to allow sub-pixel mixing.

There is an alternative technique to existing lances. For example, one company, LEIA, uses diffraction optics to produce a display with 64 views (8 in each dimension). Some techniques use parallax barriers, but these techniques lose significant brightness. While steerable backlights in combination with time division multiplexed displays have also been disclosed, the number of aspects of such a system is limited by the lack of high speed liquid crystal display (LCD) panels. Up to four independent views were reported using this system.

To make large displays, it is common to tile smaller displays in a grid. Video walls and large light emitting diode (LED) signs are often designed this way. There are many advantages to this method, including that tiles are easier to ship, store, and generally handle than single large displays. In addition, the tiles can be arranged in many different configurations. Tiles can also be serviced or replaced individually without having to handle the entire display. In addition, given a certain defect density, it is easier to manufacture tiles because small tiles are much more likely to be free of defects than very large displays. It is disadvantageous to simply make a larger display than to tile the display. For example, power and video signals must be generated and transmitted for each tile. In addition, the brightness or color of each tile may be different and may need to be corrected through calibration.

Special equipment has been created to meet the needs of traditional tiled displays. For example, a video wall controller can redesign and segment a standard video stream for playback on a tiled monitor. Color calibrator is used to maintain consistent brightness and color from tile to tile. Special mechanical mounting systems hold the tiles in place and provide channels for managing many electrical cables.

Independent multi-view displays can be arranged to create the appearance of larger displays, but the multi-view displays used to make these tiled displays are easier and less expensive to construct this type of tiled display. Does not include features

Finally, most electronic multi-view displays are targeted at auto-stereo applications and do not provide an interface that directly directs arbitrary content to multiple locations simultaneously.

There is a need for a scalable and precise multi-view display system that can easily provide any (eg, different) content to a designated location to support a positioning media experience.

Various aspects of a precise multi-view display system are disclosed that can accurately and simultaneously target content to individual viewers over a wide field of view. Larger displays can be produced by tiling individual units, and various techniques are designed that make tiling easy and efficient. In addition, a simple interface is disclosed that allows a user to graphically specify a viewing area and associate content viewable in that area, as well as a calibration procedure that enables specification of content at the correct viewing location.

1 is a front perspective view of a precision MV display device according to one or more embodiments of the present disclosure,
2 is an exploded front view of a precision MV display device according to one or more embodiments of the present disclosure,
3 is a partially exploded back view of a precision MV display device in accordance with one or more embodiments of the present disclosure,
4 is a front view of an MV display system in accordance with one or more embodiments of the present disclosure,
5A-5C each illustrate a sub-pixel pattern according to one or more embodiments of the present disclosure,
6 is a front perspective view of a lens array panel including a plurality of lens assemblies according to one or more embodiments of the present disclosure,
7A-7C are exploded side perspective views of a lens assembly including three lens arrays (first, second, and third lens arrays) in accordance with one or more embodiments of the present disclosure;
8A-8D are orthogonal views of a first lens array in accordance with one or more embodiments of the present disclosure,
9A-9D are orthogonal views of a second lens array in accordance with one or more embodiments of the present disclosure,
10A-10D are orthogonal views of a third lens array, in accordance with one or more embodiments of the present disclosure;
11 is a perspective view illustrating a rear of a first lens array and a front of a second lens array according to one or more embodiments of the present disclosure;
12 is a perspective view illustrating a rear of a first lens array and a front of a second lens array according to one or more embodiments of the present disclosure;
13A and 13B are side perspective views of a second lens array and FIGS. 13C-13E are perspective views of two of the second lens arrays, according to one or more embodiments of the present disclosure;
14 is a partial cross-sectional view of a lens assembly according to one or more embodiments of the present disclosure,
15A and 15B are cross-sectional views of a first lens array in accordance with one or more embodiments of the present disclosure,
16A is a perspective view of a first lens array to which a coating is applied in accordance with one or more embodiments of the present disclosure,
16B-16D illustrate overmolding of the lens array, which serves to eliminate the coating / painting step applied to produce the first lens array 140 of FIG. 16A,
17A is a partial rear perspective view of the lens array panel, FIG. 17B is a side perspective view of a third lens array, in accordance with one or more embodiments of the present disclosure;
17C is a perspective view of an example of a unitary baffle structure,
17D is a perspective view of one example of a unitary baffle structure with non-continuous outer walls;
18 is a partial perspective view of a diffuser in accordance with one or more embodiments of the present disclosure;
19 is a block diagram of a display controller according to one or more embodiments of the present disclosure,
20A is a diagram of a graphical user interface, FIG. 20B is a flowchart of a first graphical user interface method, FIG. 20C is a flowchart of a second graphical user interface method, in accordance with one or more embodiments of the present disclosure,
21A is a block diagram of an MV display system performing a calibration procedure, FIG. 21B is a flowchart of a calibration procedure, FIG. 21C is an image that can be displayed during a calibration procedure in accordance with one or more embodiments of the present disclosure,
22A and 22B are front views, respectively, of a lens assembly displaying a calibration pattern during a calibration procedure according to one or more embodiments of the present disclosure;
23A-23F are front views of the lens assembly during the calibration procedure, respectively, in accordance with one or more embodiments of the present disclosure;
24A-24T are front views of a lens assembly during a calibration procedure, respectively, in accordance with one or more embodiments of the present disclosure;
25A-25I are enhancement images in accordance with one or more embodiments of the present disclosure,
26 is a partial view of a conventional lenticular display system.

1 is a front perspective view of a precision MV display apparatus 100 in accordance with one or more embodiments of the present disclosure. The MV display apparatus 100 includes a grid of multi-view pixels 102 and has a quadrilateral (eg, rectangular) shape. Other shapes and configurations are within the scope of the present disclosure. For the viewer, the MV display device 100 is similar to a typical light emitting diode (LED) display. In one or more embodiments, the MV display device 100 again includes an integrated camera 104 disposed over a grid of multi-view pixels 102. Camera 104 is an example of a sensing system used to monitor activity in the field of view of MV display device 100. In one or more embodiments, such a sensing system includes or consists entirely of a sensor that is not integrated into the MV display device 100.

2 is an exploded front view of a precision MV display apparatus 100 in accordance with one or more embodiments of the present disclosure. The MV display apparatus 100 includes a rear cover 106 and a front cover 108. A high resolution flat panel display (FPD) 110 lies against the back cover 106. In one or more embodiments, flat panel display 110 includes an LCD panel and a backlighting unit. Other types of flat panel displays 110 may be used without departing from the scope of the present disclosure. The flat panel display 110 may be covered by a diffuser 162 (see FIG. 18) which serves to locally mix the colors of the display sub-pixels of the flat panel display 110 as described in more detail below. The diffuser 162 is covered by the lens array panel 112.

Lens array panel 112 consists of a smaller lens assembly 132 (see FIGS. 6 and 7A-7C), each of which is stacked to create a plurality of multi-element lens systems for multi-view pixels 102. Lens arrays 140, 142, and 144 (such sixteen lens systems are included in lens assembly 132). To prevent crosstalk between the multi-view pixels 102, the lens array panel 112 is placed between the diffuser 162 and the lenses of the lens array panel 112 (baffles 150, 152) (FIG. 17A). Reference). The rail system, including the rails 134 and 136 (see FIG. 6), holds the lens assembly 132 together in a manner capable of tightly packing the multi-view pixel 102. The front cover 108 includes a plurality of apertures 108a that improve the appearance of the multi-view pixel 102. Components of the lens array panel 112 are described in more detail below with reference to FIGS. 5 through 17C.

3 is a partially exploded rear view of a precision MV display apparatus 100 in accordance with one or more embodiments of the present disclosure. In FIG. 3, the panel 106a of the rear cover 106 is removed to expose the first driver board 114 and the second driver board 116. The first driver board 114 includes a pixel processing unit 172 (see FIG. 19) specifically designed to support multi-view applications. The first driver board 114 also includes a power controller 180 (see FIG. 19) that distributes power received via a power cable connected to the power connector 118 within the MV display device 100. The first driver board 114 also includes a network controller 178 (see FIG. 19) for transmitting and receiving data via a data cable connected to the data connector 120. As shown in FIG. 3, the second driver board 116 also includes a power connector 118 coupled to the power controller 180 and a data connector 120 coupled to the network controller 178. In one or more embodiments, data connector 120 is an Ethernet® connector and network controller 178 transmits and receives data in accordance with the Ethernet® data communication standard. Providing the power connector 118 and the data connector 120 on both the left and right sides of the MV display device 100 is a convenient and neat cable when multiple display devices 100 are interconnected to form a tiled display device. Enable the ring.

In order to produce a larger display with more multi-view (MV) pixels, the MV display device 100 can be used in a tiling configuration as shown in FIG. 4. 4 is a front view of an MV display system 122 in accordance with one or more embodiments of the present disclosure. The MV display system 122 includes a plurality of MVs that are coupled together and provided with control signals that cause their MV pixels 102 to emit light so that different images are presented to the viewer in different viewing zones, as described in detail below. The display device 100 is included. The example MV display system 122 shown in FIG. 4 includes nine MV display devices 100; Other embodiments of the MV display system 122 may include different quantities of the MV display device 100.

The MV display apparatus 100 includes many features that make tiling easier and more effective. In one or more embodiments, there are no protrusions, vents, and cable connectors provided at the side edges of the rear cover 106 and the front cover 108, which allows the MV display device 100 to be in physical contact with each other. Since the mounting point is provided at the rear of the MV display device 100 (see FIG. 3), this does not interfere with tiling. The bezel, the space between the display edge and the pixels, is minimized to improve appearance. The power connector 118 and the data connector 120 are provided on the rear cover 106 at a location (eg, opposite) to which the MV display device 100 can be daisy chained, which is a tile system, For example, the amount of cables required to drive the MV display system 122 is greatly reduced. In addition, application software that controls the operation of the MV display device 100 allows the MV display system 122 to be treated as one large display, thereby facilitating calibration and use of conventional MV display devices.

There are many aspects of the MV display system 122 working together to provide the intended multi-view functionality. For example, MV display system 122 is an optical system (a type of light field display specifically optimized for multi-view applications), display controllers, calibrations, and which work together to provide intended multi-view functionality. It includes a number of subsystems, including a graphical interface. Provide the intended multi-view function. Each of these aspects is described in more detail below.

Optical system

The MV display apparatus 100 is a kind of light field display. Each pixel of a conventional display is designed to display the intensity of one color and light at a time, which is cast to the field of view of the display. In contrast, each multi-view (MV) pixel 102 of the MV display device 100 simultaneously projects different color and light intensities in various viewing zones. In this regard, the MV pixel 102 is more like a projector that simultaneously sends beamlets of individually controlled light in multiple directions.

In one or more embodiments of the present disclosure, the lens array panel 112 of the MV display device 100 includes an array of display pixels, such that the optical to be placed over a flat panel display (FPD) 110 that includes a display pixel array. An array of elements (an array of multi-element lens systems). The multi-element lens system of the lens array panel 112 is disposed above a sub-array of display pixels (eg, 100 × 100 = 10,000 display pixels) to collectively form one multi-view (MV) pixel 102. Where each beamlet corresponds to one display pixel. In this example, each MV pixel 102 can emit 10,000 beamlets based on 10,000 display pixels, and the direction, color, and brightness of each beamlet are independently controllable. Thus, the array of MV pixels 102 can be considered as an array of small projectors, each using a subsection of flat panel display 110 as the imaging device. Alternatively, the configuration may be considered as an array of magnifiers (ie, an array of multi-element lens systems) disposed on flat panel display 110. Each lens system enlarges each display pixel to fill the pupil of the multi-element lens system. The display pixel in which the viewer appears to be magnified depends on the viewing angle or the viewer's angle relative to the optical axis of the lens system disposed above the display pixel. That is, the display pixels displayed through the magnifying glass vary depending on the viewing angle. Thus, magnification allows both the selection of viewable pixels (via the viewing angle) and the magnification of the selected visible pixel to cover larger from the viewer's point of view.

The FPD-based approach (ie, the combination of lens array panel 112 and FPD 110) provides several advantages over using discrete projector arrays. For the individual projector design, the drive electronics must be created separately for each MV pixel, but in an FPD based approach, all MV pixels on the FPD 110 can use shared electronics. By an FPD-based approach in which a fixed number of beamlets (up to the primary) are each provided by a fixed number of display pixels, the number or spatial resolution of the MV pixels 102 is matched with the angular resolution of the MV display device 100. Can be offset.

Display "sub-pixel"

Many FPDs generate colors using sub-pixels of different colors (eg, red, green, and blue sub-pixels). That is, the color of each display pixel can be set using a display “sub-pixel” of different colors that collectively form the display pixel. Viewed far enough, the display sub-pixels cannot be solved individually, creating the effect of mixing the individual colors together for the corresponding display pixel. In MV applications, it is possible to see individual display sub-pixels, but the magnification of the lens system can be set high to provide distinct angular resolution. If the viewer is in the path of a beamlet of a given display sub-pixel and not in the path of another display sub-pixel that forms the display pixel, the viewer is the color of the display sub-pixel (eg, red, green, or blue). You can only see, but not the blended colors intended for the display pixels. Similar problems can occur in black and white displays with gaps between display sub-pixels.

To solve this problem, the MV display apparatus 100 uses a diffuser 162 (see FIG. 18) that effectively mixes the colors between the display sub-pixels of the flat panel display 110 to the corresponding display pixels. According to some embodiments, separate (different) diffusers may be provided for different display pixels, respectively, such that each diffuser mixes together only the display sub-pixels of that display pixel. However, this requires precisely aligning the diffuser 162 to the sub-pixels of the flat panel display 110. Thus, in another embodiment, a single diffuser layer is provided over the entire flat panel display 110 to produce sufficient local mixing for each display pixel.

There may be an engineering tradeoff in selecting an appropriate diffuser 162. Diffusers that provide wide lateral mixing will mix colors well, but smearing will limit the attainable angular resolution of the display.

The sub-pixel patterns used for the FPD 110 vary. A typical pattern is shown in FIG. 5A. In FIG. 5A, the sub-pixel pattern 124 includes a plurality of sub-pixels 126 arranged in RGB (vertical) stripes in a square pattern. For example, the red display sub-pixel occupies about one third of the space horizontally before repeating. Thus, the diffuser 162 must span a large gap. Vertically the situation is quite different. In FIG. 5A, there is little gap between the display sub-pixels 126, so very little diffusion is required. In various exemplary embodiments, diffuser 162 is an asymmetric diffuser, providing an appropriate amount of diffusion in horizontal and vertical dimensions. Optimizing each axis independently allows the system to maintain better angular resolution than with symmetric diffusers. In one or more embodiments, flat panel display 110 includes sub-pixels 126 arranged in sub-pixel pattern 124 shown in FIG. 5A. With this flat panel display 110, an asymmetric diffuser 162 can be used, which provides more diffusion along the horizontal axis and less diffusion along the vertical axis, which is further described below with reference to FIG. Will be explained in detail.

5B shows a sub-pixel pattern 126 that includes a plurality of sub-pixels 126 arranged in a square mosaic pattern. In one or more embodiments, flat panel display 110 includes sub-pixels 126 arranged in sub-pixel pattern 126 shown in FIG. 5B.

5C shows a sub-pixel pattern 128 including a plurality of sub-pixels 126 arranged in a square red, green, blue, white (RGBW) pattern. In one or more embodiments, flat panel display 110 includes sub-pixels 126 arranged in sub-pixel pattern 128 shown in FIG. 5C.

Future FPDs may include more suitable color mixing techniques (eg, field sequential colors) that may reduce the need for diffusers. Thus, the use of diffusers in FPDs using typical color filtered sub-pixel channels is desirable, and generally these diffusers will have an asymmetric scattering profile.

Lens design and internal arrangement Mechanical alignment and fixture features

In various exemplary embodiments, a multi-element lens (or multi-element lens system) is used. Constructing a lens system using multiple elements can achieve a much better balance between focus, field of view, and fill factor. Each multi-element lens can be assembled independently, including providing a baffle to prevent stray light from intersecting between MV pixels 102, and then arranged it on top of flat panel display 110. have. Such techniques can be incredibly expensive. Alternatively, using the example of a lenticular lens sheet, one can imagine stacking lens sheets to produce individual lens elements in parallel.

There are a number of problems with the naive lens sheet approach. First, it may be difficult to maintain proper spacing between the lenses along the optical axis. Second, due to differential thermal expansion, it can be difficult to keep the lens in the center of the correct display pixel as the temperature changes. For example, if the lens sheet is fixed to one edge of the flat panel display 110, thermal expansion will move the MV pixel 102 of the far non-fixed edge far more than that of the constrained edge. Third, the sheet made of optical material may provide a path through which stray light passes parallel to the flat panel display 110 and from one MV pixel 102 to another. Finally, there can be significant manufacturing problems in molding large precision lens sheets with arbitrary surfaces on both sides. As will be described later, the MV display apparatus 100 according to the present invention overcomes this problem.

It can be difficult to keep several lenses at a certain distance from each other. FPDs can be quite large and sheets of that size can show significant sagging. This can be overcome to some extent by keeping the sheet in high tension at the edges. However, this solution poses its own problems, including the need for a large mechanical frame that can cause large gaps in the stretching and tile system of the lens sheet. The present disclosure overcomes these two problems by including self-aligning functions in the area between the lenses to help maintain correct alignment. These features will be described in detail below with reference to FIGS. 6 to 14.

One way to prevent sagging is to limit the size of the sheet to small ones and then tile these pieces together. In an exemplary embodiment, the lens consists of a 4x4 lens assembly 132 that is held in place through the system of support rails 134, 136 as shown in FIGS. 6 and 7A-7C.

6 is a front perspective view of a lens array panel 112 in accordance with one or more embodiments of the present disclosure. The lens array panel 112 includes a tiled array 132 of lens assemblies comprising 16 MV pixels 102 each. In the example shown in FIG. 6, the lens array panel 112 includes seven rows each comprising four lens assemblies 132; However, two columns are removed to show the supporting machine structure. Lens array panel 112 may include different quantities of lens assemblies 132 arranged in different configurations without departing from the scope of the present disclosure. The mechanical support structure is specifically designed such that its size and shape is guided by the space available between the lens assembly 132 itself and the lens assembly 132. This makes it possible to maximize the lens aperture.

In one or more embodiments, the support structure includes a plurality of vertical rails 134 and a plurality of horizontal rails 136. For example, the vertical and horizontal rails 134, 136 may be integrally formed or soldered together. Each vertical rail 134 has a plurality of holes formed therein, and a plurality of internal threads are formed in each opening. Lens assembly 132 is coupled to vertical rail 134 using a plurality of screws 138 having external threads. After the lens assembly 132 is disposed on the vertical and horizontal rails 134, 136, the screw 138 is inserted into the opening formed in the vertical rail 134 and rotated so that the head of the screw 138 is screw 138. ) Heads contact the lens assembly 132 to securely fasten (fix) the lens assembly to the vertical rail 134.

In one or more embodiments, multiple lens assemblies 132 are tiled together to form lens array panel 112 covering flat display 110. The lens array panel 112 includes features to help align the lens assembly 132. Certain detail shapes of arrays and shapes of other sizes may be modified and are within the scope of the present disclosure.

7A, 7B, and 7C are exploded side perspective views of lens assembly 132 according to one or more embodiments of the present disclosure. Lens assembly 132 is an optical system having three elements, a 4x4 lens array, wherein the three elements include a first lens array 140, a second lens array 142, and a third lens array 144. . The first lens array 140 includes a first side 140a and a second side 140b opposite the first side 140a of the first lens array 140. The first lens array 140 also includes sixteen lenses 140c arranged in a 4x4 array. The second lens array 142 includes a first side 142a and a second side 142b opposite the first side 142a of the second lens array 142. The second lens array 142 also includes sixteen lenses 142c arranged in a 4x4 array. The third lens array 144 includes a first side 144a and a second side 144b opposite the first side 144a of the third lens array 144. The third lens array 144 also includes sixteen lenses 144c arranged in a 4x4 array. When assembled, the lens assembly 132 includes 16 MV pixels 102, each MV pixel 102 being one of the lenses 140c of the first lens array 140, the second lens array 142. Based on one of the lenses 142c, and one of the lenses 144c of the third lens array 144 stacked on top of each other, and the stack of three lenses 140c, 142c, and 144c. Formed by a sub-array of display pixels. In the present description, the individual lenses 140c, 142c, and 144c are formed by the array surface shown in the figure and the corresponding opposite surface that may or may not be shown depending on the viewing angle of the figure.

When the MV display apparatus 100 is assembled, the flat panel display 110 is positioned behind the second side 144b of the third lens array 144 at or near the imaging plane; The viewer is located in front of the first side 140a of the first lens array 140. As described below, the first lens array 140, the second lens array 142, and the third lens array 144 form a multi-element (triplet) optical system (or lens system).

8A-8D are orthogonal views of the first lens array 140 in accordance with one or more embodiments of the present disclosure. 9A-9D are orthogonal views of second lens array 142 in accordance with one or more embodiments of the present disclosure. 10A-10D are orthogonal views of third lens array 144 in accordance with one or more embodiments of the present disclosure.

Each lens assembly 132 should have constrained mechanical degrees of freedom for the flat panel display 110 as well as other lens assemblies 132. This is accomplished using several features. The rail system as described above with reference to FIG. 6 can be adapted to the FPD as well as the rolling, pitching, and yaw (rotation about the x, y, and z axes, respectively) of each lens assembly 132. Used to limit both lens-to-lens spacing. The rail system also serves to mechanically secure the lens assembly 132 (ie, rear cover 106 and front cover 108) in the enclosure. Rail system design is partially motivated to minimize physical volume since this volume may not be occupied by the optical system. The lens aperture can be maintained to facilitate the largest fill factor (amount of FPD 110 covered by all lens assemblies 132) and throughput as possible.

In order to achieve the design goal with the largest fill factor possible, the individual lenses in lens assembly 132 abut very closely. This can have the effect of leaving little space between each lens in the array, which leads to the need for a mounting system that takes up little space in the lens assembly. In addition, the lens assembly 132 is tiled in such a way that many lens assemblies 132 are “landlocked”, meaning that they are completely surrounded by other lens assemblies 132. In an exemplary embodiment, the mounting system for the lens assembly 132 includes a set of rails 134, 136 (see FIG. 6) that extend across the flat panel display 110 as a whole. The lens assembly 132 is placed on top of the rails 134, 136 and subsequently secured to them as described above. Another possibility for mounting the lens assembly 132 includes securing to the front hole array provided by the front cover 108 of the enclosure. Such fastening devices are contemplated within the scope of the present invention.

Kinematic mounting features are integrated at the interface between pairs of lens arrays 140, 142, 144. FIG. 11 is a perspective view illustrating the rear surface or the second side 140b of the first lens array 140 and the front side or the first side 142a of the second lens array 142. Although the first lens array 140 shown in FIG. 11 includes sixteen lenses 140c, the first lens array 140 may include different amounts of lenses 140c without departing from the scope of the present disclosure. have. The plurality of bosses or posts 140d extend from the surface of the second side 140b of the first lens array 140. The mating surface 140e is disposed at the distal end of each post 140d. Each of the posts 140d has a hole 140f formed therein.

The number of lenses 142c included in the second lens array 142 is equal to the number of lenses 140c included in the first lens array 140. A plurality of cylindrical or truncated conical cylindrical holes 142d extend into the surface at the first side 142a of the second lens array 142. The mating surface 142e is disposed at the bottom of each hole 142d. The posts 140d of the first lens array 140 are inserted into the corresponding holes 142d of the second lens array 142 until the mating surfaces 140e and 142e abut each other, thereby providing lens arrays 140 and 142. Movement along the z-axis (or optical axis), rolling (rotation about the x-axis), and pitching (rotation about the y-axis) degrees of freedom.

FIG. 12 is a perspective view illustrating a rear side or second side 140b of the first lens array 140 and a front side or first side 142a of the second lens array 142. One of the posts 140d includes an outer cylindrical mating surface 140g and the other post 140d includes an outer cylindrical mating surface 140h. Each hole 142d has a hole 142f formed therein. One of the holes 142d includes an inner cylindrical mating surface 142g and the other of the holes 142d has an inner truncation with two flat mating surfaces 142h (only one of which can be seen in FIG. 12). It includes a cylindrical surface.

When the post 140d of the first lens array 140 is inserted into the corresponding hole 142d of the second lens array 142, the outer cylindrical mating surface 140g abuts against the inner cylindrical mating surface 142g. This limits the x- and y-axis degrees of freedom between these two lens arrays 140, 142. In addition, the outer cylindrical mating surface 140h abuts against the mating surface 142h to rotate about the yaw or z-axis (optical axis) between the two lens arrays 140, 142. Restrict.

13A and 13B are side perspective views of the second lens array 142, and FIGS. 13C, 13D and 13E are perspective views of two second lens arrays 142 according to one or more embodiments of the present disclosure. As shown in FIG. 13A, the plurality of bosses or posts 142i extend outwardly from the surface at the second side 142b of the second lens array 142. A pair of holes 142j are formed in each post 142i. The pair of cylindrical protrusions 142k extend from the first side 142l of the second lens array 142. As shown in FIG. 13B, the first recess 142m and the second recess 142n are formed in the second side surface 142o of the second lens array 142. The first side 142l and the second side 142o are on opposite sides of the second lens array 142. In one or more embodiments, the first recess 142m has a "V" shape and the second recess 142n has a flat surface at its bottom. The second lens array 142 shown in Figs. 13C, 13D, and 13E is shown almost in conformity with each other and is opened like a book. When the second lens array 142 is mated with each other, the cylindrical protrusion 142k is disposed in the first recess 142m and the second recess 142n.

The rail system described above (see FIG. 6) serves to mount and restrain the z- and y-axis of the lens assembly 132 and to restrain rolling, pitching, and yawing. The vee-plane-cylinder features 142m and 142n constrain the x-axis of the lens assembly as well as any two adjacent lens assemblies 132 (in the column direction) to have the same rolling, pitching, and yawing. It plays a role. It should be noted that other arrangements and configurations of these features can achieve the same goal and are considered within the scope of the present disclosure.

14 is a partial cross-sectional view of a lens assembly 132 in accordance with one or more embodiments of the present disclosure. The post 140d of the first lens array 140 is inserted into the corresponding hole 142d of the second lens array 142, and the post 144d of the third lens array 144 is the second lens array 142. After being inserted into the hole 142j of the post 142i of), three screws 146 are inserted from the second side 144b of the third lens array 144 and pass through the second lens array 142. And screwed into the inner bore of the post 140d of the first lens array 140. This allows the axial compression force to be applied to the three lens arrays 140, 142 and 144, thus limiting accurate alignment. One of the screws 146 holding the lens arrays 140, 142, and 144 of the lens assembly 132 is shown in FIG. 14.

Finally, as with any optical system, the ability to adjust the focus may be desirable. In some embodiments, the distance between flat panel display 110 and lens array panel 112 may be adjusted by the placement of shims between the flat panel display 110 mounting features and each sheet. In the enclosure of the MV display device 100, the flat panel display 110 is mounted to a rigid plate so that the flat panel display 110 remains flat. This steel sheet is then mounted to the enclosure itself (eg rear cover 106). Shims may be added or removed from these mechanical connections to adjust focus or to adjust the distance between the lens assembly 132 of the lens array panel 112 and the flat panel display 110.

Stray light management techniques

inside baffler

Many optical systems consist of a series of lenses disposed axially with each other to achieve the desired optical performance. In this scenario, the lens is often placed in a black barrel. The black barrels help prevent unwanted light from entering the optical system, which can introduce ghost images, hot spots, and contrast reduction. In exemplary embodiments, an array of lenses (eg, lens assembly 132) is used, which are comprised of multiple (eg, three) lens arrays 140, 142, 144 stacked together. It may be difficult to provide a black barrel structure for each 4x4 array of 16 lenses (or 16 lens systems). One possible method for stray light in the lens assembly 132 is light that enters the surface of the lens assembly 132, propagates internally like a waveguide, and then exits the other surface of the lens assembly 132. This is undesirable because there is now a beam propagating into a space that cannot be corrected because the exact origin is unknown. To reduce this “channel crosstalk,” some embodiments use a series of grooves or recesses 140i that act as internal baffles for the lens assembly 132.

15A and 15B are cross-sectional views of the first lens array 140 in accordance with one or more embodiments of the present invention. More specifically, FIG. 15A is a cross-sectional view of the first lens array 140, and FIG. 15B is a perspective view of the cross section shown in FIG. 15A. As shown in FIG. 15B, the first lens array 140 includes a plurality of grooves or recesses 140i formed into the surface at the second side 140b of the first lens array 140. One of the grooves 140i is disposed between each pair of adjacent lenses 140c of the first lens array 140.

This inner baffle provided by recess 140i, along with the painting of a particular surface to be discussed further below, blocks light propagating in an undesirable manner within the slab of lens assembly 132. This in the material of the first lens array 140, such grooves / recesses 140i extend outwardly from the surface at the second side 140b of the first lens array 140. This has the effect of optically isolating each lens 140c in the first lens array 140 in terms of channel crosstalk. It should be noted that other shapes and configurations for these inner baffles 140i are possible and contemplated within the scope of the present invention.

Surface painting

To further address the stray light as well as the visual appearance, some of the surfaces of the first lens array 140 may be coated with a light-absorbing coating 148, for example black paint, as this is an intrinsic visual aid. In one or more embodiments, the light-absorbing coating 148 absorbs a particular portion of light incident thereon, such as red paint or coating, or a substantial portion of light incident thereon, such as black paint or coating.

16A is a perspective view of first lens array 140 with light-absorbing coating 148 applied on second side 140b, in accordance with one or more embodiments of the present disclosure. The surface coated by the light-absorbing coating 148 may include the edge 140j of the first lens array 140, the flat surface of the second side 140b where the lens 140c meets, the inner baffle / groove 140i, And boss 140d (both inner and outer bores).

Alternative methods for achieving similar objects include bonding of black materials to these surfaces and two-part injection molding contemplated within the scope of the present invention.

Painting the surface can achieve the desired effect, but the process of painting a specific area of the lens array can be difficult. Other ways of achieving a black surface in the molded lens region include "overmolding" and "in-mold decorating" described below.

Overmolding and In-Lens of the Lens Array Mold  decoration

In one embodiment, the portion (of the lens array) may be molded from a non-transparent medium, and then the optical surface may be molded from the transparent medium around the portion. This process can be carried out in two steps in the same molding process, or as a separate molding process in which parts molded in the first process are placed in a mold for the second process.

In another embodiment, when a forming medium, such as polymer plastic, is deposited in a mold to produce a component such as a plastic array, an opaque film can be placed in the mold before the mold is closed so that the film is mated and attached to the molded part. Can be. Those skilled in the art will recognize these techniques for applying decorations to molded plastic consumer goods. Typically, the film is fed from roll to roll for the time that the mold is open and fixed to one side of the mold using a vacuum system. In general, accurate matching is required to form the correct apertures for each optical device in the lens array.

16B-16D illustrate overmolding of the lens array, which serves to eliminate the coating / painting steps applied to create the first lens array 140 of FIG. 16A.

16B and 16C are side and perspective views, respectively, of the first lens array 140 composed of an opaque portion 141a and a transparent portion 141b. Opaque portion 141a molded from an opaque medium includes the kinematic mounting features described above. The transparent portion 141b is formed by molding a transparent medium on or around the opaque portion 141a.

16D is a cross-sectional view of the over-molded first lens array 140 shown in FIGS. 16B and 16C. Above and around the opaque portion 141a molded from the opaque medium is a transparent portion 141b over-molded from the transparent medium to form the optical surface of each lens in the first lens array 140. 16D is taken at the center of the series of lenses in the first lens array 140. Different relative thicknesses of the opaque portion 141a and the transparent portion 141b are within the scope of the present invention.

All paints with anti-reflective coating

During fabrication of the optical system, as described above, a series of lenses are typically placed in a black cylindrical housing. Multi-element lens assemblies use a variety of approaches to common problems. One example of this is in the normal manufacture of lens elements, the lenses being for example crushed or molded into glass or plastic. The optical element may then be applied with an optical coating. For example, anti-reflective (AR) coatings or specific band pass coatings may be applied. Finally, the edges of the lens may be painted black. It is common to place the lens in a black housing, but painting the lens edges in black will help solve the stray light problem.

In the present disclosure, a typical sequence of operations can cause undesirable effects. Therefore, it may be desirable to change the normative order of work. That is, in some exemplary embodiments, the elements of the lens assembly 132 (eg, the first lens array 140) are first defined in shape, and then all painting operations of the light-absorbing coating material are performed. Finally, an optical (eg anti-reflective or band pass) coating is applied. For AR coatings with typical properties of very low reflectance for the visible spectrum, this has the effect of producing a visually darker black when looking at the lens assembly 132 when less light is reflected and reflected back to the viewer. If you apply an AR coating first and then perform a surface painting, color artifacts may be present and surfaces painted with a given color may appear different. This is due to, for example, the optical interface made between the AR coating and the black paint. It should be noted that this is a general technique that can be applied to other coating and surface finishing solutions.

Hole array

Opaque openings can be used to manage stray light and define hole stops and pupils in the optical system. The MV display apparatus 100 may use three hole arrays 220, 222, and 224 integrated into the lens assembly 132 as shown in FIGS. 7B and 7C. These hole arrays 220, 222, 224 overcome manufacturing issues with creative shapes and arrangements. As shown in FIG. 7C, the aperture array 222 may be bonded to the surface at the second side 142b of the second lens array 142. The aperture array functions as a hole stop of 16 optical systems (ie 16 composite lenses formed by each stack 140c of lenses 140c, 142c, 144c) in lens assembly 132. As illustrated additionally in FIG. 7B, the two other hole arrays 220, 224 may comprise a first side 142a of a second lens array 142 and a first side 144a of a third lens array 144. Respectively disposed above, the stray light path through the lens assembly 132 is blocked. Other methods to achieve this goal include the placement of individual openings for each sub-optical system (eg, each lens assembly 132), painting or coating certain surfaces in black, and two-shot injection molding It is to use different materials in two manufacturing steps such as:

As shown in FIGS. 7B and 7C, the lens assembly 132 includes a first hole array 220 comprising a plurality of holes 220a arranged in a 4 × 4 array. Lens assembly 132 also includes a second aperture array 222 that includes a plurality of apertures 222a arranged in a 4x4 array. The lens assembly 132 also includes a third hole array 224 that includes a plurality of holes 224a arranged in a 4x4 array. These inner hole arrays 220, 222, 224 can be made of thin black plastic, but other material choices are contemplated within the scope of the present invention. In addition, other types of holes other than those shown in FIGS. 7B and 7C are possible and contemplated within the scope of the present invention.

Individual lens arrays 140, 142, 144 of assembly 132 include unique features for supporting, securing, and positioning hole arrays 220, 222, 224. As shown in FIG. 10B, for example, the plurality of cylindrical first bosses or posts 144d and the plurality of cylindrical second bosses or posts 144e are formed on the first side 144a of the third lens array 144. Extends outwardly. Holes 144f are formed in each second post 144e. The first post 144d is used to support and position the third hole array 224 that lies between the second lens array 142 and the third lens array 144. The third hole array 224 can be bonded to the first post 144d using, for example, adhesive glue. The second hole array 222 may be bonded to the surface at the second side 142b of the second lens array 142 using, for example, an adhesive glue. The first hole array 220 can be bonded to the surface at the first side 142a of the second lens array 142 using, for example, an adhesive glue.

The first post 144d of the third lens array 144 has several degrees of freedom of the third hole array 224; That is, it limits rolling, pitching, and yawing as well as motion along the z-axis. The second post 144e of the third lens array 144 is used to position and mount the second lens array 142 and the third lens array 144 relative to each other. As shown in FIG. 7B, a hole 224b formed in the third hole array 224 is fitted over the second post 144e. The hole 224b and the second post 144e constrain the third lens array 144 in the x and y axis directions.

baffler

Ideally, each multi-element lens (or lens assembly) 132 receives light only from the section of flat panel display 110 assigned to it. In theory, if the lens system is designed for a specific image height / field of view, it can be assumed that light emitted outside the area will not pass through the system. In practice, however, this assumption may not be true because such rays can cause scattered stray light through the system and cause contrast reduction. Most FPDs have very large emission profiles, so field fields alone are not sufficient to solve this problem. One solution is for each lens system (eg, each lens assembly 132) near the flat panel display 110 having an opaque wall such that light from the FPD region of one lens cannot be transmitted to another lens. To block. To achieve this, as shown in FIG. 17A, the baffles 150, 152 may be configured between the flat panel display 110 and the second side 144b of the third lens array 144. Baffles 150 and 152 serve to isolate each lens channel of a given array from other lens channels. Second side 144b of third lens array 144 includes fixture features 154 for positioning and securing baffle 150, 152 to third lens array 144 as shown in FIG. 17B. do.

17A is a partial rear perspective view of lens array panel 112 in accordance with one or more embodiments of the present disclosure. In other words, the side of the lens array panel 112 shown in FIG. 17A is the side shown by the flat panel display 110. The plurality of first baffles 150 and the plurality of second baffles 152 are coupled to the second side 144b of the third lens array 144.

17B is a perspective view of a second side 144b of the third lens array 144 in accordance with one or more embodiments of the present disclosure. A plurality of first fixtures 154 are provided on the surface of the second side 144b of the third lens array 144. Each fixture 154 consists of four walls 156 extending from the second side 144b, each having a quarter solid cylinder shape. Slots 158 are formed between adjacent pairs of walls 156 to accommodate the first and second baffles 150, 152. Baffles 150 and 152 are interlocked to help add rigid structures to the features.

In one or more embodiments, each of the first baffles 150 includes a plurality of first slots, each first slot extending through approximately half of the height of the first baffle 150. In addition, each of the second baffles 152 includes a second slot, which extends through half of the height of the second baffle 152. Each first baffle 150 is associated with a plurality of second baffles 152. The first and second baffles 150, 152 are interlocked at the positions of the first and second slots such that a portion of the first baffle 150 adjacent to each first slot is around a portion of one of the second baffles 152. And a portion of each second baffle 152 adjacent to the second slot is disposed around a portion of the first baffle 150.

The width of the slot 158 is approximately the same size as the width of the baffles 150, 152 such that the wall 156 holds the baffles 150, 152 firmly in place. For each fixture 154, a first baffle 150 is inserted into two collinear slots 158 of the fixture 154, and a second baffle 152 is inserted into two other collinear slots of the fixture 154. 158 is inserted. In one embodiment, the first baffle 150 is inserted as a row into the horizontal slot 158 and the second baffle 152 is inserted as a partial column in the vertical slot 158 shown in FIG. 17B. An additional second baffle 152 is held in place by the first baffle 150 at a position between the lenses 144c where the fixture 154 is not provided.

Another method of separating each optical channel is to fabricate a single-piece baffle structure 151 comprising baffles 150 and 152 as shown in FIG. 17C. The single-piece baffle structure 151 can be achieved by machining an injection molded or honeycomb structure.

The single-piece baffle structure 151 may be formed in a particular shape associated with the lens assembly 132. 17C shows an example of a single-piece baffle structure 151 prepared for lens assembly 132 having a 4x4 lens array, although other configurations are within the scope of the present invention. In single-piece baffle structures 151 with a 4x4 baffle array, the outer wall thickness can be half the inner wall, allowing these baffle structures 151 to be tiled efficiently without increasing the lens array pitch or interfering with each other. . In this particular embodiment, the orientation marker 151a is used for tiling purposes such as the orientation of the single-piece baffle structure 151, for example an arrow pointing to a particular direction (as shown in FIG. 17C), a non-continuous outer wall (eg For example, at least some of the four arbitrary sides with the smaller thickness of the outer wall may be matched with corresponding portions having a greater thickness in the adjacent baffle structure 151 to be tiled. FIG. 17D shows an example of a single-piece baffle structure with non-continuous outer walls, which is two different adjacent outer walls 151b and smaller (0) having a larger (total) thickness for easy tiling and orientation purposes. ) Two adjacent outer walls 151c having a thickness. Non-continuous exterior walls provide a full thickness wall on all sides while allowing for tiling configurations. Other direction indicators and wall configurations are also within the scope of the present invention. For example, as an orientation indicator, one boss (rounded area at the intersection of the vertical linear section) may have an increased size relative to the other boss to indicate the proper orientation of the baffle structure 151.

Enclosure  Front hole

As shown again in FIG. 2, the front cover 108 of the MV display apparatus 100 includes several technical features. First, front cover 108 is made of a much thinner material than that of the remaining enclosures (eg, rear cover 106). Since the lens elements are preferably packaged as closely as possible, there may not be enough material between the holes 108a of the front cover 108 to maintain structural integrity. The thicker the material, the larger the aperture 108a should be, so as not to limit optical performance in both the field of view and the relative brightness. At the limit of zero thickness material, the array of holes 108a of the front cover 108 will need to be at least the same diameter as the primary optical surface. As the material thickness increases from zero thickness, the diameter of the opening 108a must increase so as not to be a vignette (or blocking) light. Since the gap between the lens assembly 132 and the mounting hardware may be visible, this may not include the front cover 108, but this will negatively affect the visual appearance of the MV display device 100.

Another consideration for the opening 108a of the front cover 108 is the visual appearance. The lens of the lens assembly 132 may or may not be applied with an optical coating. The presence of an optical coating, such as an AR coating, greatly changes the appearance of the lens element itself. In order to reduce the visual impact of the busyness of the front side of the MV display device 100, it may be desirable for the opening 108a of the front cover 108 to have a dark color and reflectance visually similar to the optical element. Since MV display device 100 is essentially a visual device designed to display information to a viewer, the features scattered from optical elements or MV pixels 102 also scatter the function of MV display device 100.

Diffuser

In color filtered displays, color filters are placed over other display sub-pixels to create larger display pixels. Most FPDs work with this system. The emission divergence emitted from each display sub-pixel may be modulated to produce a color different from the primary color of the display sub-pixel. There are many different display sub-pixel configurations, but three different examples of primary color display sub-pixel structures of red, green and blue (RGB) are shown in FIGS. 5A-5C.

One approach to designing a projection system using an electronic imaging device is to assume that no diffuser is needed, and simply place the lens at an appropriate distance from the imaging device to project the image to the desired plane. In the particular case of the stripe RGB color filter FPD (see FIG. 5A) this will not provide a proper image. As the resulting image is magnified, color separation will appear, indicating the individual display sub-pixels. In the case of the visual system, ie the system of the human eye, this effect can be prominent. This is sometimes referred to as the "screen door effect".

More sophisticated approaches use diffusers or scatterers disposed between the imaging device and the lens to mix spatially distinct primary color regions or to display sub-pixels. Examples of diffusers that can be used for this purpose include frosted glass, ground glass, diffuser films visually similar to frosted glass, and the like. These diffusers often exhibit circular symmetry scattering profiles resulting from the stochastic processes used in their manufacture. This approach can lead to more uniform color than in certain areas of the projected image through inherent balance points. Since the diffuser naturally causes a loss of spatial fidelity in the image plane, a balance point can be formed in the form of reduced spatial resolution.

Various exemplary embodiments use an engineered diffuser 162 having a non-circular symmetric scattering profile as shown in FIG. 18. When such a diffuser 162 is placed over the color filtered electronic imaging device, the scattering angle can be distinguished by two orthogonal in-plane angles. This is advantageous because it allows different color diffusion along each characteristic axis of the imaging device. In the example of the striped RGB display pixel structure, the color spreading request in the vertical direction (y-axis in Fig. 5A) is much less than the color spreading request in the horizontal direction (x-axis in Fig. 5A). In the vertical direction, the purpose of the diffuser 162 is to define an inactive, non-light emitting area between any two (such as color) display sub-pixels, i.e. a black area between the display sub-pixels 126 of FIG. 5A. Minimize appearance. In the horizontal direction, the diffuser 162 has the task of scattering light from one display sub-pixel, ie the red sub-pixel, at an angle at which light from the adjacent display sub-pixels can be sufficiently mixed. If sufficient blending occurs when the FPD is imaged at a slight magnification, the red, blue and green sub-pixels will appear as white pixels rather than spatially chrominally separated sub-pixels.

The backlight configuration or emission profile of the flat panel display 110 may also serve to determine the ideal scattering angle of the diffuser 162. In an exemplary flat panel display 110 having a stripe style pixel structure, two examples of backlights that may be used are collimated and uncollimated. The collimated backlight can produce light that travels largely in a single direction imminent on the backside of the transmissive FPD. Uncollimated backlights emit light in larger cones or solid angles. These two examples require significantly different diffuser scattering profiles. Thus, the emission profile of the flat panel display 110 is an important input in the design of the diffuser scattering profile.

In general, the scattering profile of the engineered diffuser 162 is elliptical. The long axis and short axis of the diffuser 162 may be aligned with the feature axis of the sub-pixel structure of the flat panel display 110. In the stripe sub-pixel array body, the long axis of the scattering profile will be aligned with the x-axis of FIGS. 5A-5C and the short axis of the scattering profile will be aligned with the y-axis of FIGS. 5A-5C. The use of this type of scattering diffuser 162 when properly designed and aligned to the display sub-pixel structure is advantageous over that of a diffuser with a circular symmetric scattering profile. There is still some inherent loss of space fidelity, but the loss is reduced. In an exemplary flat screen display 110 having a striped sub-pixel structure, the loss of spatial fidelity in the vertical direction is caused by the diffuser 162 with elliptical scattering symmetry compared to that of a standard diffuser with a circular symmetric scattering profile. It can be quite small.

With respect to the multi-view display device 100 consisting of a stripe RGB flat panel display 110 with a lens assembly 132 disposed thereon, the diffuser 162 may play an important role. Since the striped RGB flat panel display 110 consists of display pixels having spatially separated color sub-pixels, the light from these sub-pixels will be directed in different angular directions by the lens. Thus, an observer observing this lens can see an enlarged portion of the individual display sub-pixels, thus limiting the colors that can be displayed to the viewer to the color of the primary color of the color filter. The actual application and purpose of the diffuser 162 is to scatter the light from the individual display sub-pixels to allow blending of the three RGB display sub-pixels. As discussed above, this means a reduction in spatial resolution or angular fidelity of MV pixels. In practical terms, the required amount of diffusion or blending blends the display sub-pixels together beyond the individual display pixels. The diffuser disposed above the flat panel display 110 will actually blend more than the display sub-pixels of a given display pixel. Since the display sub-pixel spacing from the red sub-pixel to the next red sub-pixel is different in the vertical and horizontal directions, it may be desirable to apply different color spreads in the vertical and horizontal directions.

Another consideration with the backlight design in the optimal design of the diffuser 162 is the physical structure of the flat panel display 110. Many display panels include multiple layers of polarizers, cover glasses, and the like. A design of the diffuser 162 is contemplated that can optimally blend the colors of the individual display sub-pixels within the flat panel display 110.

18 is a partial perspective view of an elliptical diffuser 162 in accordance with one or more embodiments of the present disclosure. Diffuser 162 is shown disposed in front of display sub-pixel 126 of flat panel display 110. In this example flat panel display 110 provides collimated backlighting. 18 shows a single axial beam 164 from a red sub-pixel 126, a single axial beam 166 from a green sub-pixel 126, and a single axial beam 168 from a blue sub-pixel 126. ). Diffuser 162 generates a cone of red light 164a from beam 164, a cone of green light 166a from beam 166, and a cone of blue light 168a from beam 168. Each cone is elliptical in cross section and shows the elliptical scattering profile of the diffuser 162.

Display controller

19 is a block diagram of a display controller 170 in accordance with one or more embodiments of the present disclosure. In some embodiments, display controller 170 may be implemented with driver boards 114 and 116 mounted in an enclosure of MV display device 100 (see FIG. 3). The display controller 170 includes a volatile memory 174, a non-volatile memory 176, a network controller 178, and a pixel processor 172 connected to the power controller 180. Display controller 170 is coupled to a first set of network and power connectors 120, 118 (see FIG. 3) to support network connection 1 and power connection 1 with other devices, such as host computer 182. 1 interface 179a. The display controller 170 is connected to another device, for example a network connection 2 and a power connection with another MV display device 100 that may be daisy-chained to the current MV display device 100 (see FIG. 4). 2) additionally includes a second interface 179B coupled to a second set of network and power connectors 120, 118 (not shown, on the second driver board 116 of FIG. 3). can do.

Display controller 170 receives data from host computer 182 via, for example, network controller 178 and generates a beamlet for generating an image towards the viewing area (s), as described below. The flat panel display 110 is driven. If the MV display device 100 is one of many MV display devices 100 daisy-chained (see FIG. 4), the data is not upstream of the current MV display device 100, but rather the host computer 182. ) May be received from another MV display apparatus 100.

Pixel processing unit

The pixel processing unit (PPU) 172 may calculate and render the beamlet pattern on the flat panel display 110 to show the appropriate image in the appropriate viewing area. That is, the PPU 172 is derived from the first set of displays on the FPD 110 and beamlets of the first bundle facing the pupil of the first viewer in the first viewing zone to form a first image in the brain of the first viewer and Beamlets of the second bundle originating from the second set of display pixels (different from the first set of display pixels) and facing the pupil of the second viewer in the second viewing area to form a second image in the brain of the second viewer. To identify.

In various embodiments, PPU 172 is associated with viewing zone coordinate data defining the location of the first and second viewing zones, content stream data used to form the first and second images, and associating different content to different viewing zones. From the host computer 182 to render the image on the flat panel display 110 to generate a viewing area to content stream mapping, calibration parameters used to calibrate the MV display device 100 and / or appropriate beamlet patterns, respectively. Receives the color palette parameter.

In various embodiments, the viewing zones are described in a viewing zone coordinate system, such as the coordinate system of a camera (eg, camera 104) looking at the environment in which the MV display device 100 is used. On the other hand, the beamlets generated by the flat panel display 110 are described in beamlet coordinate systems such as X-Y display pixel coordinates or floating point view port coordinates of the display pixels of the flat panel display 110. PPU 172 applies a mathematical transformation between the viewing zone coordinate system and the beamlet coordinate system to calculate the corresponding beamlet coordinates for the viewing zone. That is, the PPU 172 applies a mathematical transformation between the determining viewing area coordinate system and the beamlet coordinate system to display the sub-pixels to activate to produce a beamlet visible at the corresponding position (viewing region) in the viewing area coordinate system.

Each multi-view (MV) pixel 102 controlled by the PPU 172 has a unique mapping between two coordinate systems, which is associated with the associated set of calibration parameters p0, p1, ..., p15. Included. One embodiment of the mathematical mapping between the viewing zone coordinate system (X, Y, Z) and the beamlet coordinate system (U, V) using the calibration parameters (p0, p1, ..., p15) is given in equations 1-5 below. do. PPU 172 uses equations 1-5 to map between the viewing zone coordinate system (X, Y, Z) and the beamlet coordinate system (U, V).

방정식 1

Equation 1

방정식 2

Equation 2

방정식 3

Equation 3

방정식 4

Equation 4

방정식 5

Equation 5

In one or more embodiments, the PPU 172 determines that the PPU 172 receives information about the coordinate set in the viewing zone coordinate system, evaluates equations 1 through 5 to determine the corresponding coordinate set in the beamlet coordinate system and the flat panel display 110. And a processor and a memory storage instruction for outputting information about a corresponding set of coordinates in the beamlet coordinate system used to drive.

Those skilled in the art will appreciate that there are many alternative mathematical models and parameter sets that can be used to generate mappings between the viewing area coordinate system and the beamlet coordinate system. The calibration parameter for each multi-view (MV) pixel is calculated by the calibration procedure as described below.

To reduce data bandwidth and storage requirements for the content stream and frame buffer, the color bit width may be less than the intrinsic color bit width of flat panel display 110. In some embodiments, the color value is represented using 8 bits, but flat panel display 110 is driven with a 24-bit color value. PPU 172 stores a color palette that converts between the stored color bit width and the unique flat panel display 110 bit width. For example, a stored 8-bit color can be represented by a 0-255 gray scale, 3: 3: 2 RGB (i.e. 3 bits for red, 3 bits for green, 2 bits for blue) or alternate color representation Can be. You can also adjust the color palette of each panel to provide color matching across multiple panels.

In various embodiments, PPU 172 is implemented in a field programmable gate array (FPGA) or application specific semiconductor (ASIC). Those skilled in the art will recognize that there are many other alternative implementations, including a central processing unit (CPU), graphics processing unit (GPU), or hybrid processor. In addition, multiple processors may be used together to perform the tasks of the PPU 172.

PPU 172 communicates with volatile memory 174 and / or non-volatile memory 176 to perform its tasks. Volatile memory 174 may include, for example, dynamic random-access memory (DRAM) and / or static random-access memory (SRAM). Non-volatile memory 176 may include flash, electrically erasable programmable read-only memory (EEPROM), and / or disk drive. In various embodiments, PPU 172 is in communication with volatile memory 174 to provide dynamic run-in, including but not limited to viewing zone data, content stream data, viewing zone to content stream mapping, and / or frame buffer data. Save the time data. PPU 172 communicates with non-volatile memory 176 to store static data, including but not limited to calibration parameters, color palettes, firmware, identification numbers, and / or version numbers. PPU 172 may also modify the contents of non-volatile memory 176 to set stored parameters or update firmware, for example. The ability to update firmware on the fly can be easily upgraded without plugging in additional programmer cables and without running special software on the host computer 182.

PPU 172 provides buffering in the system to allow adequate performance degradation in non-ideal situations. In general, for displays such as LCDs, video data must be transmitted consistently at a constant rate (eg 30Hz, 60Hz). However, due to non-deterministic computation, rendering, and data transfer from the host computer 182, the PPU 172 may generate data at an unfixed rate. Thus, the PPU 172 includes buffering, for example, when controlling the flat panel display 110 to maintain the state of the last frame if the data is too slow or drop the frame if the data is too fast.

The PPU 172 drives the flat panel display 110 through the FPD connector 184. In various embodiments, FPD connector 184 is an embedded DisplayPort (eDP) interface. Those skilled in the art will recognize that there are many alternative display interfaces, including but not limited to DisplayPort, High Definition Multimedia Interface (HDMI), Digital Visual Interface (DVI), and Video Graphics Array (VGA). In one or more embodiments, the FPD connector 184 further includes a connection for powering, controlling, and / or modulating the backlight of the flat panel display 110.

The PPU 172 communicates with the host computer 182 and / or other display controller 170 (of the other MV display device 100) via the network controller 178. PPU 172 may include, but is not limited to, viewing zone information, content stream data, viewing zone to content stream mapping, calibration parameters, color palette parameters, identification information, addressing information, status information, and / or other configuration information. Send and / or receive data via. In various embodiments, the network is an Ethernet® network, and network controller 178 provides an Ethernet® physical layer interface. Those skilled in the art will recognize that there are many alternative data interfaces, including but not limited to Universal Serial Bus (USB), Peripheral Component Interconnect (PCI), Infiniband® and / or Thunderbolt®. In certain situations, some data interfaces may be preferred over others. For example, Ethernet® can generally span longer physical distances than USB, which can be advantageous in many installation configurations.

Multi-MV Display Device Tiling  function

Several features of display controller 170 facilitate tiling of multiple MV display devices 100 to form larger displays. For example, in various embodiments, to reduce the number of ports required by the host computer 182, reduce cable lengths, and simplify installation, features may be used to daisy-chain multiple MV display devices 100 in a daisy chain. Can be used to connect. Those skilled in the art will recognize that there are many alternative connection structures, including but not limited to buses, trees, stars, and / or meshes.

The network controller 178 includes two network interfaces 179a and 179b coupled to each data connector 120 to allow passage of the received data to the downstream MV display device 100. In various embodiments, network controller 178 includes a dual Gigabit Ethernet® transceiver. PPU 172 may receive data from first network interface 179a and transmit data to second interface 179b and vice versa. The transmitted data on the second interface 179b may be a direct copy of the received data on the first interface 179a, a filtered version of the received data, a converted version of the received data, or completely independent data.

For example, in various embodiments, viewing zone data sent by the host computer 182 is intended to be consumed by all MV display devices 100 in the MV display system 122 (see FIG. 4), while content Different portions of the stream data are intended to be consumed by the particular MV display device 100 of the MV display system 122. The first network interface 179a of the network controller 178 receives the viewing area and content stream data for all MV display devices 100, and the PPU 172 operates only on data associated with the MV display device 100. do. All viewing zone and content stream data is copied directly to the second network interface 179b of the network controller 178 and transmitted to the downstream MV display apparatus 100. In an alternative embodiment, the PPU 172 does not carry the content stream data intended for the MV display device 100 because no other MV display device 100 uses that data. This reduces the overall data traffic in the MV display system 122.

The directivity of the network interfaces 179a and 179b of the network controller 178 can be programmed immediately. This multidirectionality allows flexibility in the installation configuration. For example, one situation may require the host computer 182 to be placed in a daisy chain on the left side of the MV display device 100, while another situation may require the computer 182 to be placed on the right side of the MV display device 100. May be required to be placed in a daisy chain. This directional programming can be performed manually or with active instructions. In the former example, any data received on either network interface of network controller 178 may be operated on another interface of network controller 178 and transferred to another interface of network controller 178. In the latter example, one network interface of network controller 178 is designated an upstream interface, while the other interface is designated a downstream interface. Upstream / downstream designations may be reversed when a "set direction" command is received at the downstream interface.

Some commands may be broadcast to all display controllers 170 in a chain. For example, in various embodiments, all display controllers 170 operate on the same set of viewing zone data broadcast to all display controllers 170. However, in order for the other display controller 170 in the daisy chain to operate on different data, the display controller 170 may need to have a separate address. For example, each display controller 170 can use its own set of calibration parameters and render from a portion of its content stream. A simple way of assigning separate addresses is that each display controller 170 has a globally unique ID. For example, a serial EEPROM with a pre-programmed global unique ID can be read by the PPU 172. As another example, a unique ID number may be stored in non-volatile memory 176. The host computer 182 can query the display controller 170 by daisy chaining to their unique ID and map the content stream portion to the unique ID. However, this technique requires a separate ID memory or bookkeeping step.

In various embodiments, the temporary unique ID number is assigned at run time. For example, host computer 182 sends a " set address " command with a base address and incremental value to first display controller 170 in the daisy chain. The first display controller 170 sets its address to the given base address. Thereafter, the first display controller 170 transmits the base address to which the increment value is added to the second display controller 170 of the daisy chain along with the increment value. The second display controller 170 sets its address to the incremented base address, increments the address again, and sends a new address and increment value to the third display controller 170, etc. in the daisy chain. In this way, each display controller 170 assigns at run-time a unique address known within the daisy chain.

The host computer 182 may perform a query to determine the number of display controllers 170 in the chain at run-time. For example, display controller 170 may be designed to respond to a ping command with a unique address. The ping command is broadcast by the host computer 182 to all display controllers 170 in the chain, and all display controllers 170 respond to the ping command with their unique addresses. Thereafter, the host computer 182 may simply count or check the number of ping responses to determine the number and address of the display controller 170 in the chain. In this way, the application can be adapted to multiple MV display devices 100 in a chain without requiring a fixed number of MV display devices 100.

In addition to the network interface, the power interface of the power controller 180 may be arranged to allow daisy chaining. For example, power may be received from the first interface 179a of the power controller 180 and transmitted to the second interface 179b of the power controller 180. Alternatively, the first and second interfaces of power controller 180 may be directly connected to allow power to be transmitted in either direction to allow for more flexible installation.

Programming interface

In various embodiments, the primary method of controlling the MV display device 100 is via an application programming interface (API) running on a host computer 182 attached to the display controller 170 of the MV display device 100 via Ethernet. It works. The API is intended to be used by the programmer to control the MV display device 100. The main purpose of the API is to allow the user to (i) create and update (i.e. resize, move, etc.) a viewing zone in the viewing zone coordinate system, and (ii) create and update a content stream that can be viewed in the viewing zone. Color, text, scroll direction, image change), and (iii) assign a viewing area to the content stream.

The API allows users to do this statically and dynamically. Below are some examples of static and dynamic tasks that can help illustrate the various experiences that can be created using these three basic features.

You can use static actions to create a viewing area at a specified location and display content to the viewer based on the viewer's location. For example, the one or more MV display devices 100 are statically configured to show different advertisements on different sides of the street, display red light on a car over a certain distance away from traffic lights, and green light on a closer car. Can be. In addition, the one or more MV display apparatus 100 may be statically configured to display the text to the viewer in the native language of the country in which the viewer stands, using the world map on the floor.

Dynamic manipulation can use dynamic content and static viewing zones. Viewing zones can be created at specific locations, and external data can be used to determine which content to show in which viewing zones. For example, a person can walk behind a podium, view the content on a sign, and use the podium's dial to select the language of information displayed to the person. In addition, people sitting in the cinema's seats can use the phone to enter seat number and subtitle preferences (ie, subtitles, English, Spanish, German, etc.). In this case, the viewing area is set statically for each seat, but the content changes according to user input. Some interactive devices (eg, dials, phones, remotes, gestures, facial recognition) can be used to change what content is displayed in a static location such as a chair.

Dynamic tasks may use static content and dynamic viewing zones. The viewing area changes based on external data, but the contents are set using only internal data. For example, the API can be used to create 3D viewing zones and assign content to them, and display controller 170 turns on only the beamlets that terminate within the viewing zone (the real-time point cloud, non-time camera, or later described, or Can be determined based on other 3D sensors). This has the effect of dynamically updating the viewing area such that the viewing area is the exact size of the person standing within the viewing area. For example, the user can statically set the 3D area to be the bounding box of the viewing area. When one or more people enter the bounding box, the viewing area is updated to exactly match the people of the viewing area. That is, the 3D viewing zone can be statically set and dynamically updated. In addition, people can be tracked using canes, badges, phones, motion capture systems, vehicles or visual tags, and content is assigned without external data (ie, based on location).

In addition, the dynamic operation can be completely dynamic, the viewing area is created dynamically and the content is made dynamically based on external data. For example, you can track people using canes, badges, phones, motion capture systems, vehicles, visual tags, and so on, and content is assigned based on who the person is or whether the person has entered the system. Enter this mall and start looking at certain items). In addition, computer-assisted face recognition of the viewer's face can be used to set a viewing area around the face, identify who the viewer is, and show viewer specific content based on the viewer's identity.

In addition to the three basic functions, (a) automatic discovery, (b) manually specifying content buffer-to-display panel mapping, (c) filtering viewing areas based on the corrected area, and (d) single-view Allows for easier operation including mode.

(a) auto-discovery

The host computer 182 runs software to perform an auto-search process to find out which MV display devices 100 are connected and how they are plugged together. Without this data, the operator must manually program the address for each MV display device 100 and then inform the API of the address of the MV display device 100. Instead, the API finds all attached MV display devices 100 at start up and assigns an address to each. If the order in which the MV display apparatuses 100 are plugged is not changed, this is done in a programmatic and repeatable manner so that the addresses of the respective MV display apparatuses 100 remain the same. This is an advantage that the content can be accurately displayed because the API divides the content based on the address assigned to the MV display apparatus 100. There are many other ways to achieve persistent address assignment, such as setting a unique identifier (ID) for each MV display device 100 at the factory, but can be less efficient than the auto-discovery method, which is pre-assigned by the unique ID. Need not be.

(b) Manually specify content buffer-to-display panel mapping

When generating content for the multi-view display device 100, a single image (or frame buffer) is generated and then on each individual MV display device 100 based on the physical arrangement of the MV display device 100. One can expect to be able to allocate a portion of the image to be displayed. Since the addresses of the MV display apparatus 100 depend on the order in which they are plugged in, and the user can plug in the MV display apparatus 100 in a desired manner, adjacent addresses do not necessarily correspond to adjacent panels. In various embodiments, the MV display system 122 may allow the user to manually specify which portion of the frame buffer is mapped to which address. For example, the user may have the content delivered by the multi-view (MV) pixels (0,0) through (27,15) mapped to the first MV display device 100, while the MV pixel (28,0) To (56, 16) is mapped to the second MV display device 100 or the like. Allowing the user to allocate some of the content in this manner gives the user greater creative freedom. Alternatively, it can be assumed that the MV display device 100 is plugged in a particular way and automatically assigns a specific area of content to the MV display device 100, but this is how the user assigns to the MV display device 100. You can be cautious about connecting. Given physical constraints such as mounting, etc., it may not even be possible to plug the MV display device 100 into the desired configuration.

(c) Filtering viewing area based on the corrected area

It is sometimes difficult for a user to know exactly where the MV display device 100 has been calibrated (ie, the exact location in the viewing area coordinate system where the beamlet from each MV pixel is known to end) and is not calibrated. In general, the MV display device 100 is better within the area where the calibration has been performed (e.g., within the convex hull of all points where the calibration device 210 is placed during calibration, see FIG. 21A). To perform. As a result, it is advantageous to help the user to understand which places in the viewing area coordinate system are calibrated and which places are not calibrated. In various embodiments, this is accomplished by selectively filtering out areas of the viewing area disposed outside the calibration area and not displaying content there. Alternatively, the user may be informed that the viewing area that can be displayed is outside the corrected volume.

(d) single-view mode

When the designer uses the MV display device 100 and tries to preview the content, in order to confirm that the right content is visible in the right viewing area, the designer needs to get up from the host computer 182 and physically stand in the viewing area to see the content. There can be. To ease the design burden, MV display system 122 may include a "single view" mode. In this mode, designers can view a single content stream wherever they are physically as long as they are within the field of view of MV display device 100. Although this mode is designed to assist designers and programmers, it may be necessary for the ultimate operation of the MV display system 122 (see FIG. 4) to enable moments where everyone viewing the MV display device 100 can see the same. May be used. For example, viewers in different viewing zones can normally see different images, but in an emergency situation, the same emergency alert content can be displayed to all viewers regardless of which viewing zone each viewer is in.

Graphical user interface

In order to allow a less technical user to use the MV display device 100, a graphical user interface 186 can be used as shown in FIG. 20A. In various embodiments, graphical user interface 186 serves two main purposes. First, users can get up and running quickly with basic features. In addition, advanced users who write their own code can quickly place the viewing area for their own code. In one or more embodiments, host computer 182 executes software that causes graphical user interface 186 to be displayed on a display device.

20A is a diagram of a graphical user interface 186 in accordance with one or more embodiments of the present disclosure. In one or more embodiments, graphical user interface 186 includes a main window and viewing zone information pane 188, a content allocation pane 190, and a viewing zone coordinate system pane 192. Contains panes. The viewing zone information pane 188 allows the operator to name the various viewing zones shown in the viewing zone coordinate system pane 192. The content allocation pane 190 allows an operator to assign content to various viewing zones created using the viewing zone coordinate system pane 192. For example, the content allocation pane 190 allows an operator to name an image file or movie file that contains the content displayed in each viewing zone shown in the viewing zone coordinate system pane 192.

The graphical user interface 186 allows the operator to specify and display the viewing space representation 194 in the viewing zone coordinate system pane 192. For example, the viewing space representation 194 may be a 3D model of the room in which the MV display device 100 is used. When an operator performs a graphical operation on a display device of the host computer 182 using a pointing device (eg, a mouse) of the host computer 182, the host computer 182 may view a location on the display device. Convert to a corresponding position in the coordinate system (eg, the coordinate system of the room in which MV display system 122 is to be used). The graphical user interface 186 also allows the operator to place and manipulate the viewing area within the viewing area coordinate system pane 192. For example, an operator may use a pointing device to draw, size, and draw a first viewing zone representation 196a, a second viewing zone representation 196b, and a third viewing zone representation 196c in a viewing zone coordinate system pane 192. Change, and can be moved. In one or more embodiments, each viewing zone representation 196a-196c appears as a three-dimensional bounding box. After the user specifies three viewing zones with the viewing zone representations 196a through 196c, the host computer 182 displaying the graphical user interface 186 displays the boundary coordinates of the viewing zone representations 196a through 196c as the three coordinates of the boundary. The coordinates of the viewing area are stored after conversion to the corresponding coordinates in the viewing area coordinate system.

Providing a visual representation of the viewing zone coordinate system in the viewing zone coordinate system pane 192 may help people understand how to use the MV display device 100. The form of visual representation depends on the sensor 104 used in the MV display device (see FIG. 1) / used with the MV display device. For example, if the sensor is a 2D camera 104, the graphical user interface 186 may simply display a live feed from the camera 104. In some cases, however, humans may not easily see and understand the output of the display sensor. An example of this may be a tracking system using IR rays. The camera 104 used in the IR tracking system may block visible light, making it more difficult for a person to understand the output of the IR tracking system. Alternatively, MV display system 122 may take the data output (ie, tracked display system 122) and take the data output (ie, the location of the tracked object) and overlay it on the 3D model of the previously built environment. A particular implementation of this concept is shown in Figure 20A.The graphical user interface 186 shows the viewing zone coordinate space as a pre-built 3D model of the space and the real time point cloud 198. In the example shown, the real time point The cloud 198 graphically represents the position of the person standing in the viewing space The point cloud 198 is a visual representation of the output of the stereo camera sensor The depth is calculated from the difference between the two images for each function of the image. The corresponding image pixel for the feature is rendered in 3D space at the appropriate point, depending on its position in the image and the calculated depth of the feature. C. Because the point cloud can be somewhat difficult to visually understand, the point cloud 198 is superimposed on the 3D model of the viewing space of Fig. 20. The user sees what he sees as a person standing in front of the central room. It is easy to understand, if the point cloud 198 is displayed without the context of the model, it may be more difficult for the user to see the correlation between what the user sees in the graphical user interface 186 and the real world. It was created by scanning a space with a 3D camera such as a ® camera and then calibrating the generated model with a stereo camera whose display device 100 was calibrated and displaying both the finally aligned model and the point cloud 198. However, the computer Use other room modeling techniques such as secondary drafting (CAD), Project Tango®, and more It may be open to perform this task.

In addition to showing the general coordinate system in the form of a camera feed, point cloud, etc., the graphical user interface 186 can also show what the maximum calibrated boundary is (viewing zone filtering based on “(c) calibration area described above). " Reference). The fact that the sensor can detect in a specific area does not necessarily have to locate the viewing area. This is because the user has not calibrated the entire field of view within the field of view of the display sensor 104. To help the user understand which areas have been corrected and which areas have not been corrected, the graphical user interface 186 includes a feature that overlays the rendering of the corrected area / volume through the viewing area coordinate system visualization. In various embodiments, this may be a shaded 2D / 3D box.

In the representation of the viewing zone coordinate system, the viewing zone can be disposed within and manipulated. In 2D, this may simply be that the MV display device 100 draws and manipulates a rectangle (or potentially another 2D shape) over the camera feed to be calibrated. In 3D, this can be more complicated. For 3D cases, you need to define the volume of space in which the content is displayed. In various embodiments, any 3D volume can be used, but an axis-aligned bounding box (ie, a rectangular prism with all sides parallel to the axis of the coordinate system) can be used to speed up the computation. Moving and manipulating 3D volumes in the 3D space of a 2D computer monitor can be more difficult than the 2D case, but can be done using standard CAD methods.

20B is a flowchart of a first graphical user interface method 300 in accordance with one or more embodiments of the present disclosure. The method 300 begins at 302. For example, the user selects a user interface object displayed by the host computer 182, which causes the host computer 182 to send a message to the MV display device 100 over a network.

At 304, a display sensor (eg, 104) captures sensor data in the space where the MV display device 100 can be seen. For example, in response to a message from the host computer 182, the camera 104 of the MV display device 100 captures sensor data of a portion of the room where the viewer of the MV display device 100 is located.

At 306, sensor data is received. For example, the host computer 182 receives sensor data captured by the camera 104 via a network transmitted from the network controller 178 of the MV display device 100. In one or more embodiments, sensor data may be transmitted over a universal serial bus.

At 308, sensor data and viewing zone data are rendered on the display device. For example, the memory of the host computer 182 may cause the host computer 182 to process sensor data captured by the camera 104 when executed by a processor and display the corresponding processed data coupled to the host computer. Stores software instructions for sending to the device. The data sent to the display device is a format that causes the display device to display the graphical user interface 186 shown in FIG. 20A, which format is rendered sensor data (eg, displayed as viewing space representation 194). And viewing area data (eg, a 3D model of the viewing space) displayed, including the point cloud data 198 and the viewing area representations 196a, 196b, 196c.

After the sensor data and viewing zone data are rendered to the graphical user interface 186 on the display device at 308, the user is displayed by the viewing zone representations 196a, 196b, 196c in the context of the display sensor data, displayed on the display device. Visualized viewing zones can be visualized. After viewing the information displayed in the graphical user interface 186, the user may determine that the viewing area represented by the viewing area representation 196a needs to be adjusted by moving and resizing, for example. The user can then perform a graphical operation using a pointing device (eg, a mouse) connected to the host computer 182 to select the viewing zone representation 196a and then resize and move it on the display device. .

At 310, user input is received. For example, host computer 182 receives data corresponding to graphical operations made by a user that cause viewing area representation 196a to be resized and moved on the display device.

At 312, new coordinates of one or more viewing areas are determined. For example, the memory of the host computer 182, when executed by the processor, causes the host computer 182 to generate new coordinates of the viewing area displayed by the viewing area display 196a based on the user input received at 310. Stores software instructions for determining in the viewing area coordinate system.

At 314, an application programming interface is informed. For example, the memory of the host computer 182, when executed by the processor, causes the processor to execute a message on the host computer 182 that indicates a change in the coordinates of the viewing area represented by the viewing area representation 196a. Stores software instructions for transmission to the programming interface.

At 316, the viewing zone data is updated. For example, an application programming interface running on the host computer 182 causes data corresponding to the new coordinates of the viewing area represented by the viewing area representation 196a determined at 312 to be stored in the memory of the host computer 182. .

At 318, the updated data is sent to the display device. For example, an application programming interface running on the host computer 182 causes data corresponding to the new coordinates of the viewing area represented by the viewing area representation 196a determined at 312 to be transmitted to the MV display device 100.

At 320, method 300 ends. For example, the display controller 170 of the MV display apparatus 100 stores data corresponding to the new coordinates of the viewing area represented by the viewing area representation 196a and uses this data to display the flat panel display 100. Determine which display pixel causes the beamlet to be emitted into the viewing area represented by the viewing area representation 196a.

One feature of graphical user interface 186 is the ability to create content and assign it to a viewing area. Content designers can design images and videos for multi-view displays in other software programs and then import the images and videos. However, the user can use the graphical user interface 186 to create simple content such as scrolling and static text. Once the content is created, it can be assigned to the content group. There is one content assigned to the content group and one or multiple viewing zones. While this may be thought of as assigning content to the viewing zone, it may be more advantageous to think about assigning the viewing zone to content because in many embodiments much less content streams are supported than the viewing zone. For any reasonable sized MV display device 100 having a reasonable number of MV pixels 102, the content stream may have more data bandwidth than the viewing area when communicated from the host computer 182 to the display controller 170. Because it occupies. As mentioned above, in various embodiments, a user creates a group for all content streams. The user can move the viewing area between groups to change the content to display in any viewing area.

Each "configuration" or state may be stored which defines which viewing area is located at which location and which viewing area is assigned to which content (or content group). Graphical user interface 186 provides a list of configurations arranged in order so that all stored configurations can be quickly and easily switched. Using the configuration list, graphical user interface 186 allows the user to switch between configurations based on external triggers. For example, when a button is pressed in the environment (e.g., a visitor presses a button located near MV display device 100 at an amusement park), MV display system 122 moves to the next configuration with a different set of content. Can be. Triggers from other systems, such as lighting consoles, various sensors, timers, or media servers may also be received. Another use of the ability to store configuration information from the graphical user interface 186 is simply to store the viewing area location. Extending the previous example, if the programmer wants to dynamically change the content that is displayed when the button is pressed, the programmer can write the program using the application programming interface. As another example, a programmer may set up a viewing area in the graphical user interface 186, name the viewing area (eg, "button 1", "button 2", etc.), and then load the file into the programming interface. To assign dynamic content to the viewing zone.

20C is a flowchart of a second graphical user interface method 330 in accordance with one or more embodiments of the present disclosure. Method 330 begins at 332. For example, the user provides input to cause the host computer 182 to display the graphical user interface 186 on a display device coupled thereto.

At 334, first configuration data is generated. For example, a user may use a pointing device (eg, a mouse) coupled to host computer 182 to view viewing zone representation 196a and viewing zone in viewing zone coordinate system pane 192 of graphical user interface 186. Perform a graphics operation to generate representation 196b. When the memory of the host computer 182 is executed by a processor, it causes the host computer 182 to display the viewing zone data representing the boundary of the first viewing zone and the second viewing zone to data representing the graphical operation performed by the user. Store the software instructions for creation and storage in the viewing zone coordinate system.

The user may also use the content allocation pane 192 of the pointing device and graphical user interface 186 to assign the first content stream to the first content group and to assign the second content stream to the second content group. Do this. In addition, the user assigns the first viewing zone, represented by the viewing zone representation 196a, to the first content group and the second viewing zone, represented by the viewing zone representation 196b, for assigning the second content zone. Perform graphical operations using.

In one or more embodiments, the memory of the host computer 182, when executed by a processor, causes the host computer 182 to include viewing zone data indicative of boundaries of the first and second viewing zones, the first configuration of which includes: Data representing a content item included in the content group, data representing a content item included in the second content group, data indicating that a first viewing zone has been assigned to the first content group, and a second viewing zone in the first content group. Stores software instructions that cause data to be generated indicating whether they have been assigned.

For example, when the memory of the host computer 182 is executed by a processor, the host computer 182 may display the first configuration data in the first viewing area (eg, in the coordinates of the boundary of the first viewing area). Data associated with the identifier of the "viewing zone 1", the coordinates of the boundary of the second viewing zone, associated with the identifier of the second viewing zone (e.g., "zone 2"), and the first content stream (e.g., For example, an identifier of file name 1 is associated with an identifier of a first content group (eg, "group 1"), and an identifier of a second content stream (eg, file name 2) is associated with a second content group (eg, For example, an identifier of "Group 2"), an identifier of a first viewing zone (eg, "Zone 1") is associated with an identifier of a first content group (eg, "Group 1"), And the identifier of the second viewing zone (eg, "zone 2") is equal to the identifier of the second content group (eg, "group 2"). Stores the instructions to store a first configuration data in the table tube or other suitable data structure.

At 336, second configuration data is generated. For example, a user generates third and fourth viewing zone data, assigns a third content stream to a third content group, assigns a fourth content stream to a fourth content group, and assigns a third viewing zone to the third content zone. Assigning to the third content group, and performing a graphical operation similar to that described above to assign the fourth viewing area to the fourth content group. Host computer 182 then displays viewing zone data representing the boundaries of the third and fourth viewing zones, data representing the content of the third and fourth content groups, and data indicating that the third viewing zone has been assigned to the third content. And second configuration data comprising data indicating that the fourth viewing zone has been assigned to the fourth content group.

At 338, first and second viewing zone data are transmitted. For example, the memory of the host computer 182, when executed by the processor, causes the host computer 182 to transmit the first and second viewing zone data identified in the first configuration data to the MV display device 100. Stores software instructions.

At 340, first and second viewing streams are transmitted. For example, the memory of the host computer 182, when executed by the processor, causes the host computer 182 to transmit the first and second viewing streams identified in the first configuration data to the MV display device 100. Save the software command.

The display controller 170 of the MV display apparatus 100 uses the first and second viewing zone data transmitted at 338 and the first and second viewing stream transmitted at 340 to determine which beamlets (or corresponding) in the coordinate system of the flat panel display. Display pixels) to drive the viewer of the first viewing zone to view the first content stream and the viewer of the second viewing zone to watch the second content stream.

At 342, trigger data is received. For example, at 342, host computer 182 receives a signal from a sensor device or a message from a communication device located in a room in which MV display device 100 is located. In one or more embodiments, host computer 182 receives a message that includes data identifying particular configuration data. For example, at 342, the host computer 182 receives a message that includes or identifies the second configuration data (eg, “second configuration”).

At 344, the application programming interface is notified. For example, when the memory of the host computer 182 is executed by a processor, the application programming causes the host computer 182 to execute a message in the host computer 182 indicating a change in the configuration data identifying the second configuration data. Stores software commands sent to the interface.

At 346, third and fourth viewing zone data are transmitted. For example, an application programming interface running on host computer 182 may include a message indicating a change in configuration data, including, for example, an identifier that identifies the second configuration data or is associated with an identifier of the second configuration data. Responsive to receiving at 344, cause the host computer 182 to transmit the viewing zone data included in the second configuration data to the MV viewing apparatus 100. In one or more embodiments, the third and fourth viewing zone data causes the display controller 170 to stop driving the display sub-pixels of the flat panel display 110 and viewing zone data currently stored in the non-volatile memory 176. Is sent along with one or more instructions instructing it to delete it.

In one or more embodiments, the third and fourth viewing zone data store the third and fourth viewing zone data in non-volatile memory 176, and wherein the identifier of the content stream of the third content group is non-volatile memory ( Associated with the identifier of the third content group of the table or other suitable data structure stored in 176, and wherein the identifier of the content stream of the fourth content group is the fourth content of the table or other suitable data structure stored in non-volatile memory 176. Sent according to one or more instructions instructing display controller 170 to be associated with the group identifier.

At 348, third and fourth viewing streams are transmitted. For example, an application programming interface running on host computer 182 may be identified in the second configuration data in response to the host computer 182 receiving a message at 344 indicating a change in configuration data received at 342. The third and fourth viewing streams at 348.

At 350, method 330 ends. For example, the display controller 170 of the MV display apparatus 100 may correspond to coordinates included in the third and fourth viewing zone data transmitted at 346 in the viewing zone coordinate system in the beamlet coordinate system of the flat panel display 110. Converted to coordinates, to drive the flat panel display 110, the viewer of the third viewing zone can see the third content stream and the viewer in the fourth viewing zone can see the fourth content stream.

correction

The MV display apparatus 100 requires a calibration process. This is because the user specifies the position in the viewing area coordinate system, and the MV display apparatus 100 needs to know which beamlet should shine for each MV pixel 102. If the light is bent in each lens in the correct manner and the exact position of each lens relative to the display sensor (ie camera 104) and the correct position of the lens relative to the underlying display panel are known, then the calibration process can theoretically be eliminated. have. In practice, such measurements can be difficult to obtain and more difficult to use in real time to turn on the correct beamlet for a given viewing area coordinate.

In various embodiments, a simplified mathematical model is used to approximate which beamlet is turned on for a given viewing area coordinate. In the worst case, the approximation is in error in the order of several display pixels between the intended beamlet and the actual beamlet, which is acceptable under normal circumstances. On average, the error is much better at about 0.5 display pixels.

The calibration process determines coefficients and constants that approximate the projection / mapping of the position from the observation area coordinate system to the beamlet coordinate system in the mathematical model. To determine the coefficients and constants, the calibration device captures some ground truth mapping between the viewing area coordinate system and the beamlet coordinate system. Collected data and non-linear optimizers are used to find coefficients and constants in equations. Once the coefficients and constants are obtained, a new mapping can be efficiently generated given the viewing area coordinates.

Physical setting

Some hardware is required to collect the ground truth mapping to solve the coefficients. In various embodiments, at least three devices, the MV display device 100; Attached to the MV display device 100 for generating a viewing area coordinate space (eg, camera, stereo camera, light detection and ranging (LIDARL), non-time camera, line scan camera, etc.). Display sensor 226 (eg, camera 104); And a camera (calibration device 210) that can see the MV display device 100, can be moved around the environment, and can be found by the display sensor 226 as shown in FIG. 21A. In FIG. 21A, the dashed line represents data "shown" during the calibration procedure and the solid line represents data transmitted during the calibration procedure.

In one embodiment, the calibration device 210 takes the form of a camera attached to a checkerboard and a tablet computer (e.g., includes a processor and memory storage instructions that cause the tablet computer to perform a calibration procedure) and displays a display sensor. 226 is a 2D camera. In an alternative embodiment, the calibration device 210 is a camera and tablet computer with an infrared (IR) LED attached, and the display sensor 226 is an IR sensing stereo camera. In any case, the calibration device 210 should be able to be found in the viewing area coordinate system by a display sensor (eg, camera 104). Certain other examples of calibration device / display sensor combinations include checkerboard / stereo cameras, other printed patterns or tags / cameras (or stereo cameras), visible light LEDs / cameras (or stereo cameras), and the like. A host computer can additionally be used to control the MV display device 100, and the wireless network allows the calibration device 210 and the host computer 182 to communicate during the calibration procedure. In some embodiments, one computer may be used to remove the tablet, but this may potentially require the camera to have a cable run on the host computer 182. The display controller 170 can interface directly with the calibration device (camera) 210.

Calibration procedure

21A is a block diagram of an MV display system 122 performing a calibration procedure in accordance with one or more embodiments of the present disclosure. The MV display system 122 includes an MV display apparatus 100 communicatively coupled to a host computer 182. For example, the MV display device 100 is connected to the host computer 182 via an Ethernet® based local area network. The MV display system 122 also includes a display sensor 226 (eg, a camera 104) communicatively coupled to the calibration device 210 and the host computer 182. For example, the calibration device 210 and the host computer 182 are interconnected via an IEEE 802.11n based local area network, and the display sensor 226 (eg, the camera 104) and the host computer 182 are connected to each other. Interconnected via a universal serial bus.

During the calibration procedure, the host computer 182 transmits the display pattern data 228 to the MV display device 100. In response, the MV display apparatus 100 emits the light shaping display pattern 230 corresponding to the display pattern data 228. The calibration apparatus 210 records which beamlet is received from the MV display apparatus 100. The calibration device 210, on the other hand, includes a checkerboard pattern 232 (eg, displayable on the screen of the calibration device 210 or print and attach to the calibration device 210). If the calibration device 210 is within the field of view of the display sensor 226 (ie, the display sensor 226 can detect or detect the checkerboard pattern 232 of the calibration device 210), the display sensor ( 226 sends calibration device location data 234 to host computer 182. In one or more embodiments, calibration device location data 234 represents the coordinates of calibration device 210 in the viewing zone coordinate system based on the detected checkerboard pattern 232. The calibration device 210 transmits the beamlet coordinate data 236 stored by the host computer 182 to the host computer 182. As described below, host computer 182 calculates calibration parameters p0, p1, ..., p15 using stored calibration apparatus position data 234 and beamlet coordinate data 236. It is used by the MV display device 100 to convert the coordinates of the viewing zone coordinate system to the corresponding coordinates of the beamlet (or display pixel) coordinate system of the flat panel display 110 so that the MV display device 100 is located in different viewing zones. Different content can be presented to different viewers.

In one or more embodiments, the calibration device 210 includes a tablet computer having a memory that stores software instructions that, when executed by a processor of the tablet computer, cause the tablet computer to perform aspects of the calibration procedure. In addition, the memory of host computer 182 stores software instructions that, when executed by a processor of host computer 182, cause the host computer to perform other aspects of the calibration procedure.

The calibration procedure captures several mappings per MV pixel between the spatial 1D / 2D / 3D points of the viewing area coordinate system and the beamlets of the beamlet coordinate system that illuminate the spatial coordinate position in the world when actuated. In various embodiments, this captured mapping is spread around the entire area that will be used to view the MV display device 100. To capture these mappings, the MV display system 122 must do two things: find the calibration device 210 in the viewing area coordinate space and allow the calibration device 210 to record that the beamlet is touching at its current location. do.

In various embodiments, the calibration device 210 is found by placing the checkerboard pattern 232 in the feed of the display sensor 226. This represents the current position of the calibration device 210 and provides spatial coordinates in the viewing zone coordinate system included in the calibration device location data 234. As mentioned above, display sensor 226 (eg, camera 104) may be a 1D, 2D, or 3D sensor. Each of these has an hint as to how the MV display apparatus 100 operates. The dimensions of the display sensor 226 determine the dimensions of the coordinate space in which the end user can define the viewing area. Thus, if the MV display device 100 is calibrated with the 2D display sensor 226, the viewing area can be defined only as an area of the 2D surface, and all locations where the calibration device 210 is placed must be within that 2D surface. . The disadvantage of using the display sensor 226 which is 2D or 1D is that the MV display device 100 can only work well in the corresponding plane or line because the mathematical model assumes that the viewer is standing on that plane or line. will be. If the MV display device 100 is small compared to the viewer's distance from the MV display device 100, the difference between beamlets hitting the viewer on and off the plane can be small and neglected. However, as the MV display device 100 grows (eg, multiple MV display devices 100 are tiled together), the difference between beamlets for a person standing on the calibrated surface and a person away from it It may not be small and some MV pixels appear to be turned on for the viewer. To solve this problem, in various embodiments, the display sensor 226 may comprise a 2D camera and it is possible to measure the distance between the calibration device 210 and the display sensor 226. The distance is then used as the third coordinate to add an additional dimension to effectively convert the 2D display sensor 226 into a 3D sensor. Therefore, the user can designate the distance between the area of the 2D image and the camera.

21B is a flowchart of a calibration procedure 360 in accordance with one or more embodiments of the present disclosure. Calibration procedure 360 begins at 362. For example, the user provides input to cause the host computer 182, the calibration device 210, and the display sensor 226 to execute certain software instructions stored in their respective memory.

At 364, the calibration device 210 is located within the field of view of the MV display device 100. The calibration device 210 may be located at any point within the viewing zone coordinate system defined by the display sensor 226.

At 366, display sensor 226 determines the location of calibration device 210. In one or more embodiments, the memory of the display sensor 226 causes the display sensor 226 to capture an image of the checkerboard pattern 232 displayed by the calibration device 210 and corresponding image data when executed by the processor. And determine the coordinates of the calibration device 210 in the viewing zone coordinate system based on the image data, and store the instructions to send the calibration device location data 234 including the determined coordinates to the host computer 182. do. In some embodiments, the display sensor 226 sends sensor data to the host computer 182, which processes the sensor data to determine the coordinates of the calibration device 210 in the viewing zone coordinate system. .

At 368, MV pixel 102 of MV display device 100 is positioned by calibration device 210. In one or more embodiments, the host computer 182 generates display pattern data 228 that turns off all of the MV pixels 102 after the MV display device 100 turns on all of the MV pixels (see FIGS. 22A and 22B). The camera of the calibration device 210 captures an image of the MV display device 100 when both MV pixels 102 are turned on and off. The memory of the calibration device 210 when executed by the processor causes the calibration device 210 to process image data corresponding to the image, compare the image data, and correspond to each MV pixel based on the comparison of the image data. Stores a software instruction that allows you to determine the position in the image.

At 370, each MV pixel 102 is identified. In one or more embodiments, host computer 182 may display display pattern data (MV display device 100) to turn on and off each of MV pixels 102 in accordance with a unique code that is assigned or associated with each MV pixel 102. 228) (see FIGS. 23A-23F). The camera of the calibration device 210 captures an image of the MV display device 100, while the MV display device 100 turns each MV pixel 102 on and off according to a unique code. The memory of the calibration device 210 stores software instructions that, when executed by a processor, cause the calibration device 210 to process image data corresponding to the image using a unique code to identify each MV pixel 102. do.

At 372, a display pixel ID (or beamlet ID) corresponding to the location of the calibration device 210 is determined. In one or more embodiments, host computer 182 generates display pattern data 228 that causes MV display device 100 to turn each beamlet on and off in accordance with a unique code assigned to each beamlet. This makes the calibration device 210 view the “on” and “off” of the MV pixel 102 (see FIGS. 24A-24T) when the beamlet corresponding to the position of the calibration device 210 is turned on and off. The calibration device 210 captures an image of the MV display device 100, while the MV display device 100 turns each beamlet on and off according to a unique code assigned to it. When the memory of the calibration device 210 is executed by a processor, the calibration device 210 processes image data corresponding to the image using a unique code assigned to each beamlet to correspond to the location of the calibration device 210. Store a software instruction to determine a display pixel ID (or beamlet ID).

In this embodiment of step 372 in one embodiment, the goal is to find any of ˜10,000 beamlets under each MV pixel, for example, wherever MV pixel 102 is placed wherever calibration device 210 is placed. MV display device 100 needs to be turned on to appear on. In an ideal case, the MV pixel 102 would appear to be "off" when only one of the beamlets is on, but appears to be "on" when one (correct) beamlet is on. The MV display apparatus 100 displays a pattern encoding the ID for each beamlet on the flat panel display 110. Thus, for a given MV pixel and position in the viewing zone coordinate system, the calibration device 210 will see the pattern as shown in FIGS. 24A-24T.

At 374, the purification process may be performed as described below with reference to FIGS. 25A-25I.

At 376, calibration parameters are determined as described below.

21C shows an image 200 that can be displayed during a calibration procedure in accordance with one or more embodiments of the present disclosure. The image 200 corresponds to a room in which the MV display apparatus 100 is to be used. Image 200 shows a plurality of locations 202 of the display sensor 226 (ie, stereo camera 104) that captured the checkerboard pattern 232 of the calibration device 210 within the point 3D point cloud where it was rendered. It includes a marker. Various aspects of the calibration procedure 360 are described in more detail below with reference to FIGS. 22A-25I.

Once the location of the calibration device 210 is found, the MV display system 122 must determine whether the beamlet of each MV pixel impinges on the calibration device 210. To accomplish this, the host computer 182 may cause the MV display device 100 to display a series of patterns. Each pattern is used to provide specific information to the calibration device 210. The patterns are listed below in the order of one exemplary embodiment, but other orders may be used.

Correction Step 1: The MV pixel position is found.

22A and 22B are front views of the MV display apparatus 100 including one lens assembly 132 according to an embodiment of the present invention. In the example shown, the MV display apparatus 100 includes sixteen (4x4) MV pixels 102. None of the MV pixels 102 are illuminated in FIG. 22A, but all of the MV pixels 102 are shown illuminated in FIG. 22B. The MV display apparatus 100 flashes on and off all MV pixels 102 as shown in FIGS. 21B and 21A, respectively. This allows the calibration device 210 to capture the illumination pattern to determine which areas of the image contain the MV pixels 102 and allow for greater error checking.

Correction step 2: MV pixel ID is found.

23A through 23F are front views of the MV display apparatus 100 including the single lens assembly 132 as in FIGS. 22A and 22B described above. 23A-23F show a series of images encoding the MV pixel ID of each MV pixel, with the images shown arranged in little-endian order.

Display pattern data 228 transmitted by host computer 182 causes MV display device 100 to display a series of images (patterns) on calibration device 210 using MV pixels 102. 23A to 23F encode specific ID numbers for each individual MV pixel 102 in the MV display apparatus 100. In various embodiments, the ID is encoded in binary format to reduce the number of images needed to encode all MV pixel IDs, although other encoding schemes may be used. For example, you can use color to encode MV pixel IDs or use gray codes for that purpose.

In various embodiments, each of FIGS. 23A-23F represents one bit of a number in binary encoding representing one MV pixel ID. If a particular MV pixel 102 is turned off in an image (or pattern), 0 is assigned to the image for that MV pixel, and if a particular MV pixel 102 is turned on, 1 is assigned to the image (or pattern) for that MV pixel. Is assigned. Then a series of bits are converted into corresponding ID numbers.

For example, in Figures 23A-23F, the circular MV pixel 102 has a binary encoding of 000111, with an ID number of seven. 000111 shows that the circular MV pixel 102 is turned on (ie assigned a value of 1) in the three first images of FIGS. 23A-23C ("111" on the right) and FIGS. 23D-23F (left) Based on being turned off (assigned a value of 0) in the three latter images of "000". As another example, the MV pixel to the left of the circular MV pixel has a binary encoding of 000110 (ID No. 6), and the MV pixel to the right of the circular MV pixel has a binary encoding of 001000 (ID No. 8).

Calibration device 210 captures an image corresponding to FIGS. 23A-23F at 370 of calibration procedure 360. This allows the calibration device 210 to know which region of the image belongs to which MV pixel 102. Since the calibration device 210 knows which region of the image belongs to which MV pixels 102 respectively, the mapping can be captured for all MV pixels 102 of the MV display device 100 at a time. This may be done per MV display device 100, but there is an exemplary embodiment that does this across the entire MV display system 122 including multiple MV display devices 100. Every MV pixel 102 of the MV display system 122 is assigned its own ID.

Calibration Step 3: The Display Pixel ID is Found.

24A-24T are front views of an MV display 100 including a lens assembly 132 having sixteen (4 × 4) MV pixels 102, as in FIGS. 22A-23F. More specifically, FIGS. 24A-24J are front views of the MV display apparatus 100 during X gray coding of beamlet IDs (or display pixel IDs) and FIGS. 24K-24T are Y gray coding of beamlet IDs (or display pixel IDs). While is a front view of the MV display device 100. The illustrated images are arranged in little-endian order. The calibration device 210 captures the image of FIGS. 24A-24T at 372 of the calibration procedure 360.

When the memory of the calibration device 210 is executed by the processor of the calibration device 210, the calibration device 210 causes the display pixels of the beamlets (or beamlet IDs) to each of the MV pixels 102 of the display device 100. Stores a software command to process image data corresponding to the image of the MV display device 100 shown in FIGS. 24A-24T to determine the ID.

In this step, one exemplary embodiment uses gray code encoding (again, other encodings may be used) to flash a particular sequence where each beamlet is a unique ID. The ID number is simply the x-beamlet coordinates followed by the y-beamlet coordinates. For a given MV pixel 102, there is one "best" beamlet that best illuminates the location of the calibration device 210. At this stage, the MV pixel 102 is turned off or on at the calibration device 210 (i.e., critical), meaning that the "best" beamlet is off or on and that data is used to decode the ID of the beamlet. It is assumed that the case appears below or above the luminance value. Thus, in Figures 24A-24T, circular MV pixel 102 has been read "ON" in x images 0, 4, and 7 and y images 2, 3, 4, 6, and 8. This provides gray code encoding of 0010010001 (right to left) and 0101011100 (right to left) for y. To convert these encodings into binary encoding, the memory of the calibration device 210 is executed by the processor of the calibration device 210 so that the calibration device 210 converts the standard gray code into a binary equation (ie, equations 6-15). Store a software instruction to determine that the binary representation of the gray code encoding is 0011100001 for x and 0110010111 for y. This equates to the x coordinate of 225 and the y coordinate of 407.

Binary Method [9] = Gray Code [9] Equation 6

회색 코드[8] 방정식 7Binary Method [8] = Binary Method [9]

Gray code [8] Equation 7

회색 코드[7] 방정식 8Binary Method [7] = Binary Method [8]

Gray code [7] Equation 8

회색 코드[6] 방정식 9Binary Method [6] = Binary Method [7]

Gray code [6] Equation 9

회색 코드[5] 방정식 10Binary Method [5] = Binary Method [6]

Gray code [5] Equation 10

회색 코드[4] 방정식 11Binary Method [4] = Binary Method [5]

Gray code [4] Equation 11

회색 코드[3] 방정식 12Binary Method [3] = Binary Method [4]

Gray code [3] Equation 12

회색 코드[2] 방정식 13Binary Method [2] = Binary Method [3]

Gray code [2] Equation 13

회색 코드[1] 방정식 14Binary Method [1] = Binary Method [2]

Gray code [1] Equation 14

회색 코드[0] 방정식 15Binary Method [0] = Binary Method [1]

Gray code [0] Equation 15

Calibration Step 4: Improve Calibration

In practice, the calibration device 210 may be between two (or even four) beamlets. This is more likely when the lens is poorly focused on the MV display device 100, in which case the calibration device 210 ideally sees (or identifies) only one beamlet as the “best” beamlet in 372 of FIG. 21B. Can see multiple beamlets. To mitigate this problem, optionally, an "improvement" step is performed at 374 of the calibration procedure 360.

As described above, after the MV pixel location, MV pixel ID, and display pixel ID (or beamlet ID) have been found at 368, 370, and 372, respectively, the calibration device 210 determines which beamlets are present in the calibration device 210. Have enough information to evaluate whether it best corresponds to the location. To verify the accuracy of the estimate, in an improvement step at 374, the calibration apparatus 210 sends beamlet coordinate data 236 to the host computer 182 (see FIG. 21A), where the beamlet coordinates are the data 236. Contains information regarding the initial estimation of the best beamlet for each MV pixel 102. Thereafter, the host computer 182 displays the display pattern data 228 in which the MV display device 100 turns on nine display pixels one by one around the estimated "best" display pixel (including the estimated "best" display pixel itself). Send it.

25A-25I are enhancement images in accordance with one or more embodiments of the present disclosure. Each image included in FIGS. 25A-25I shows one beamlet 216 on the MV pixel 102 captured by the calibration device 210. As shown, each MV pixel 102 emits a plurality of (eg, 14x14 = 196) beamlets from a plurality of display pixels 215 contained therein.

The calibration device 210 determines for each MV pixel 102 any of the nine enhancement images shown in FIGS. 25A-25I, which, as described above, determine the MY pixel location and the MV pixel ID. That is, the calibration device 210 determines which of the nine beamlets 216 shown in FIGS. 25A-25I is the best. Once the best beamlet 216 is determined, as shown in FIG. 21A, the calibration apparatus 210 returns the beamlet coordinate data 236 for each MV pixel 102 back to the host computer 182 for further processing. send. As noted above, in one or more embodiments, each beamlet 216 corresponds to one of the display pixels 215, and each display pixel 215 is a plurality of (eg, RGB) display sub-pixels. 126 (see Figs. 5A to 5C). In the illustrated embodiment, each MV pixel 102 includes 196 (= 14 × 14) display pixels 215 each comprised of a plurality of display sub-pixels 126. Thus, each MV pixel 102 may emit 196 beamlets each having a different color / luminance in 196 different directions from 196 display pixels 215.

There are also a number of alternative ways to improve. For example, the illustrated embodiment selects eight display pixels around the estimated best display pixel, but instead of nine display pixel regions 3x3, 25 display pixel regions (5x5) centered around the estimated highest display pixel. ) May be used. Encoding methods can also be used to reduce the number of images needed for the improvement process. One such encoding involves sequentially showing each row and column instead of each display pixel. In the case of nine display pixel regions (3x3), using such an encoding method reduces the number of images required from nine to six. This method finds which row image is the brightest and which column image is the brightest at the position of the MV pixel. Based on this information, it can be uniquely determined which display pixel is the brightest for the MV pixel (ie, the display pixel located in the brightest column and row).

After the calibration procedure, the MV display system 122 knows which beamlet 216 corresponds to the location of the calibration device 210 and which coordinates in the viewing zone coordinate system correspond to the location of the calibration device 210. In various embodiments, the calibration procedure (364-374 in FIG. 21B) is carried out to the calibration device 210 at a number of different locations (eg, a minimum of about 11 and a maximum of about 50) spread around the calibration space. Run many times, the coefficients and constants of the mathematical model can be estimated at 376. In various embodiments, this is performed on the host computer 182 with all of the viewing area coordinate / beamlet mappings stored therein. The collected mapping and the objective function as given by equations 1-5, for example, a number of unknown coefficients and constants (e.g., calibration parameters (p ₀ , p ₁ ,... ., p ₁₅ )) is entered into the non-linear solver. The nonlinear solver attempts to iteratively converge to the "best fit" of the provided data using the data. In view of the present disclosure, those skilled in the art will understand how to apply a non-linear solver to complete this task. Once the coefficients and constants are found, the mathematical model (which now has the coefficients and constants determined for each MV pixel 102) can take the viewing area coordinates as input and return the ID of the corresponding beamlet. In various embodiments, this model transmits to the display controller 170 of the MV display device 100 for use in converting coordinates in the viewing zone coordinate system to corresponding coordinates in the beamlet coordinate system of the flat panel display 110. do.

Modified

The calibration procedure described is time consuming and noise prone. For example, in one implementation, the calibration device 210 may always need to be placed in the 2D plane to calibrate with a 2D camera (this is strict because the system may be adapted to allow any 2D surface). May not be required). To alleviate some of these problems, you can make minor changes to the process to improve the results.

For example, an inverse pattern can be used. When the encoding pattern is captured (as described above, MV pixel ID and beamlet (display pixel), while determining the ID), the inverse of the pattern can also be captured. That is, if the MV pixel is "on" in the pattern, it is "off" in the opposite image and vice versa. This allows the MV display system 122 to subtract the inverse image of the pattern from the image of the pattern to double the signal to noise ratio. This is because when two images are subtracted, any baseline luminance (i.e., light reflected from the surface of the MV display device 100) of the image is subtracted, leaving only the signal from the MV pixel 102 left.

As another example, aperture adjustment may be used. In order for the calibration procedure to work correctly, the calibration device 210 may need to be able to tell the difference between when the MV pixel 102 is "on" and "off". Since “off” may not be a complete absence of light (eg, the MV pixel may appear to be “on” due to light leakage from the backlight), the calibration device 210 may turn off the “off” off pixel. It can be adjusted to read and the "on" MV pixel to be the appropriate amount of light to read on. To accomplish this, the MV display device 100 shows a pattern in which half of the MV pixels are on and the other half are off. The user then adjusts the camera's aperture ring until the off MV pixel appears off in the camera feed.

As another example, a calibration robot can be used. Since one embodiment of calibration uses the 2D camera 104 attached to the MV display device 100, the user needs to move the calibration device 210 relative to the camera 104 of the MV display device 100. It may be efficient to calibrate the MV display device 100 with the camera 104 without. The MV display apparatus 100 may be precalibrated. Orthodontic robots can be used to solve this problem. The robot is configured such that the MV display device 100 and / or the calibration device 210 are disposed on the robot. The robot then moves the MV display device 100 and the calibration device 210 in an automated manner to capture the mapping based on the list of desired locations supplied for placing the calibration device 210. Once the robot has completed the capturing mapping, the coefficients and constants can be calculated in a mathematical model and stored coefficients and constants for use in subsequent processing.

One way the robot can be built is to move the calibration device 210 camera within the viewing space with the MV display device 100 stationary. This can lead to very large robots that require a lot of room. Instead, the robot can be constructed such that the calibration device 210 camera remains within a certain line, and the MV display device 100 moves to simulate the calibration device 210 camera moving around the MV display device 100. Pan and tilt. The calibration device 210 camera still moves back and forth in line to ensure that the points captured around the MV display device 100 are on a plane rather than a hemisphere. In this way, the number of actuators required for robot function is reduced. The software that drives the robot may use a formula that converts the physical location (ie, x, y, z offset from the MV display device 100) supplied to it to movement, tilt and distance coordinates. This allows the calibration robot to calibrate the MV display device 100 to any set of points.

The robot can be placed in a controlled lighting environment such that the lighting can be changed according to different parts of the calibration process. This can ensure that the checkerboard is well illuminated in the calibration device 210 to reduce the noise of the meter (thus making the display sensor 226 easier to see). Illumination may be turned off for a portion of the calibration process in which the calibration device 210 captures the pattern, reducing the reflected light on the MV display device 100.

In the case of the individual MV display apparatus 100 to which the camera 104 is attached, the MV display apparatus 100 may be fully calibrated before being installed. This is generally the case with the relatively small MV display device 100 and the camera 104 being unable to move in relation to any MV pixel. However, when a plurality of MV display devices 100 are used, since the exact position of each MV display device 100 relative to the display sensor 104 may not be known in advance (for example, the MV display device 100 It may be difficult to fully precalibrate the MV display device 100) prior to tiling together). In various embodiments, a robot may be used to partially calibrate the MV display device 100 prior to completing calibration in the field. The calibration robot may determine the unique property of the MV display apparatus 100 and determine the unique property of the site. For example, in various embodiments, the radial distortion coefficients and lens center constants (i.e., displaying pixels over which the lens (lens system) is over) are calibrated with a calibration robot, which is the location or display of the MV display device. This is because it does not change regardless of how it is oriented with respect to sensor 104. Then, the fractional linear projection equation describing the position of the lens (lens system) relative to the display camera 104 is corrected in the field. Since some of the coefficients and constants are precalibrated, the solver has less freedom in determining the remaining coefficients. This allows you to capture fewer points than performing a full calibration in the field. After obtaining the fractional linear projection equation coefficients, we can combine them with the pre-calibrated coefficients to obtain the complete set of coefficients for use in the mathematical model.

The disclosure of US patent application Ser. No. 15 / 809,147, filed November 10, 2017, is incorporated herein in its entirety.

The various embodiments described above can be combined to provide further embodiments.

These and other changes to the embodiments can be made in light of the above detailed description. In general, in the following claims, the terms used should not be construed as limiting the claims to the specific embodiments disclosed herein and in the claims, and the full scope of equivalents to which such claims are entitled. Should be construed as including all possible embodiments. Accordingly, the claims are not limited by this disclosure.

Claims (15)

A multi-view display device,
A display comprising an array of display pixels;
A lens array panel comprising an array of lenses, each of the lenses and the display pixels on which the lenses are disposed form a multi-view (MV) pixel, wherein the MV pixels emit beamlets in different directions to produce an MV pixel. A lens array panel, wherein the array is configured to form different images, each visible in a different viewing area positioned relative to the array of MV pixels; And
And an enclosure comprising the display and the lens array panel. The method of claim 1,
The enclosure includes a rear cover disposed adjacent the display and a front cover disposed adjacent the lens array panel,
And the front cover defines an aperture corresponding to each of the MV pixels. The method of claim 1,
Each display pixel is formed of a plurality of display sub-pixels,
The multi-view display device comprises a diffuser disposed between the flat panel display and the lens array panel,
And the diffuser is asymmetrical to provide more diffusion along a first axis and less diffusion along a second axis different from the first axis according to the display sub-pixel configuration of the display. The method of claim 1,
A controller having a quadrilateral shape and provided on both opposite edges of the quadrangle shape, the network and power connector comprising a first network interface coupled to a first network connector and a second network interface coupled to a second network connector Including,
One of the first and second network interfaces serves as an upstream interface for inputting signals, the other of the first and second network interfaces serves as a downstream interface for outputting signals,
The controller further comprises a first power interface coupled to a first power connector and a second power interface coupled to a second power connector,
One of the first and second power interfaces serves as an upstream interface for inputting power and the other of the first and second power interfaces serves as a downstream interface for outputting power. . The method of claim 1,
The lens array panel comprises a frame formed of a rail; And
And a plurality of lens assemblies supported by the frame and tiled adjacent to each other to collectively form a lens array of the lens array panel. The method of claim 5,
Each of the lens assemblies comprises at least two lens arrays stacked together,
Each of the lens assemblies includes a first lens array comprising a first mechanical coupler, a second mechanical coupler connectable with the first mechanical coupler, and a second lens array including a third mechanical coupler, and a third mechanical coupler. A third lens array comprising a possible fourth mechanical coupler,
Each lens assembly includes an internal baffle configured to block stray light from crossing among the MV pixels,
And the inner baffle is formed with a recess defined in the lens array. The method of claim 6,
At least one surface of the lens assembly is coated with a light-absorbing material,
The light-absorption coating is applied before the anti-reflective coating or a band pass coating is applied to one or more surfaces of the lens assembly,
The array of lenses is formed by overmolding a transparent medium on the surface of a hole structure molded from an opaque medium or by in-mold bonding of an opaque film to an array of lenses molded from the transparent medium. Device. The method of claim 1,
And a sheet of baffle disposed between the lens array panel and the display and configured to block stray light from crossing between the MV pixels. A method of controlling the operation of two or more multi-view (MV) display devices coupled to each other, wherein each MV display device is configured to project a plurality of images to a plurality of viewing areas located relative to the MV device.
Assigning addresses to the at least two MV display devices, respectively;
Receiving a specification of one or more viewing zones;
Receiving a specification of the underlying content on which one or more images are formed in the one or more viewing zones; And
Controlling two or more MV display devices to project one or more images to one or more viewing zones, respectively. The method of claim 9,
Receiving a user specification of a mapping between the content and two or more MV display devices. The method of claim 9,
In receiving a specification of the one or more viewing zones, distinguishing a viewing zone in a calibration space from a viewing zone in a non-calibration space in a viewing zone coordinate system. The method of claim 9,
In single-view mode, allowing the same content to be viewed in all viewing zones within the field of view of two or more MV display devices. The method of claim 9,
Superimposing a representation of the viewing area and the 3D model of the viewing space of the MV display device; And
Superimposing a representation of the viewing area and point cloud data representing the viewing space in real time. The method of claim 11,
Displaying, by an operator, a graphical user interface configured to create and / or adjust a plurality of viewing zones;
Displaying a graphical user interface configured to allow an operator to create and / or adjust content; And
Assigning one or more viewing zones to one or more content groups. The method of claim 9,
Storing an MV display configuration, each MV display configuration defining a series of viewing zones and which viewing zones are assigned to which content groups, and
Selecting one of the MV display configurations based on a trigger, the trigger comprising one or more of a user-input trigger, a sensor-based trigger, a pre-programmed trigger, and a timing trigger.

Images