TWI390367B - Interface and circuit arrangement in particular for holographic encoding units or holographic display devices - Google Patents

Description

Especially for the interface of holographic coding unit or holographic display device And circuit configuration

The present invention relates to an interface and circuit arrangement in which digital image data is transmitted to a hologram encoding unit or a hologram display device including a hologram encoding unit. These image data are generated from video sequences. A coding unit then generates a complex holographic value and/or a coded pixel value for the holographic display device based on the transmission data. The coding unit means is, for example, a dedicated calculation unit.

In a conventional two-dimensional monitor, the parsed image data can be transmitted to the monitor through the interface, and the image can be represented by individual pixel values. Compared with the conventional two-dimensional monitor, the holographic display device is characterized in that the modulated modulated light that can generate interference travels in the form of an optical wavefront in front of the observer's eyes, and the optical wavefront is controllable in terms of amplitude and phase values. The light wave front thus reconstructs the three-dimensional scene. Controlling the optical modulator device with the full image value of the video holographic image causes the transmitted wavefield (the individual pixels have been modulated accordingly) to reconstruct the desired three dimensional scene by creating interference.

The holographic display device preferably used in the present invention is substantially based on the principle that the scene resolved into object points is encoded into a complete hologram on at least one of the light modulator devices. The scene is seen as a reconstruction from the visible region, which is located at a periodic interval of video holographic reconstruction. A sub-image is defined for each object point of the scene to be reconstructed. A complete hologram can be formed by the superposition of sub-images. In general, the principle is to primarily reconstruct the one that will be emitted by the object into one or more visible areas. Wave front.

In particular, such devices are based on the principle that reconstruction of individual object points is only required as a sub-image of a subset of complete holograms that are encoded on the optical modulator device. The hologram display device includes at least one screen device. The screen device is a holographic image of the scene that is encoded by itself or an optical component (such as a lens or lens) onto which the hologram or wavefront of the image being encoded on the light modulator is projected.

The definition of the screen device and the corresponding principles of scene reconstruction in the visible region are described in other documents filed by the applicant. In the documents WO 2004/044659 and WO 2006/027228, the screen device is the light modulator itself. In document WO 2006/119760 "Projection apparatus or method for holographic reconstruction of a scene", the screen device is the optical element to which the hologram is projected, which is projected onto the light modulator. In the document DE 10 2006 004 300 "Projection device for holographic reconstruction of a scene", the screen device is the optical element to which the wavefront is projected, the wavefront being encoded on the optical modulator.

The "visible area" is the restricted area by which the observer sees the entire reconstructed scene. In this visible region, the wavefield interferes to form a wavefront so that the observer sees the reconstructed scene. The visible area is located in or near the observer's eyes. The visible area is movable in the X, Y, and Z directions and can be tracked to the actual observer position by the aid of a known location detection and tracking system. For each observer, it is possible to use two visible ranges (one for each eye). In general, the visible area of it His embodiment is also possible. It is more likely that the video hologram will be encoded such that individual objects or the entire scene appear to be behind the light modulator for the viewer.

Document WO/2006/066906 filed by the applicant describes a method of calculating a video hologram. It typically involves dividing the scene into segment planes (which are parallel to the plane of the light modulator), converting all of the segment planes into the visible region, and adding them there. The resulting result is then converted back to the holographic plane of the configurable optical modulator, thus determining the complex holographic value of the video hologram.

The document DE 10 2006 042 324, filed by the applicant, describes a method for the instant generation of a video holographic image. The principle used in this method is that the reconstruction of a single object point is only required as a sub-image of a subset of the complete hologram, which is encoded on the optical modulator. It is characterized in that for each object point, the contribution of the sub-image can be extracted from at least one of the search tables, and the sub-images can be superimposed to form a complete hologram to reconstruct the entire scene.

A variety of further methods and apparatus are known in the holographic field, however they differ from the principles described above. For example, document US Pat. No. 4,109,996 A describes a method of producing a composite holographic image, while document US 2005/0286101 A1 describes a hologram projector.

Document US 6 621 605 B1 describes a computer-aided method and apparatus for three-dimensional image recovery, and an interface for transferring data. The 3D image is segmented into a plurality of 2D segment images. The two-dimensional segment image data is transmitted to the display device along with the color information, and the display device sequentially presents the two-dimensional image so that the observer seems to see the three-dimensional scene. The so-called "response" in this article should be interpreted as such. The disadvantage is that the 2D data also includes portions of the image that are not visible to the observer. More data than needed is therefore transmitted. When the visibility is optimized, the amount of data will still be very large, or larger than necessary.

Document US 2006/0187297 A1 describes a holographic 3D television. The 3D TV system captures the 3D action scene, converts it according to the 3D image method, transmits it, and finally converts it into a full image signal using an efficient calculation method to establish the omnipresent image. The so-called transmitted data includes the conversion of the 3D scene to the net mode. One disadvantage here is that more data is transferred than needed.

The object of the present invention is to minimize the amount of data transmitted to the coding unit, especially when displaying video sequences, the amount of data transmitted should be minimal.

Another object of the invention is to transmit data that is adapted for certain viewer positions, wherein the amount of data should still be minimal. These data should be structured to enable the coding unit to generate complex holographic values and/or to perform pixel value encoding of the hologram display device with minimal computational load. The delay between signal generation and holographic rendering should also be reduced.

The broad idea of the present invention defines the interface and circuit configuration required for a holographic coding unit to enable generation of complex holographic data, and/or for pixel value encoding of omnidirectional display devices.

According to the invention, the interface is characterized in that it converts a depth map representing depth information and a color map representing color information of the pixelized image data by means of a transmission device and using a communication protocol. This circuit configuration is based on the same idea. The individual transmissions that are spatially separated on the transmission device are transmitted. May use the same The device is transmitted and individual transmissions are performed by time division multiplexing. In either case, it is also possible to use different communication protocols.

The depth map includes depth information for the pixelated image. Similarly, the color map includes color information related to the pixelated image. An image herein is, for example, a single image of a video sequence. The data is, for example, extracted from the memory of the image card. More specifically, the color map is from a so-called frame buffer and the depth map is from the Z-buffer of the image card. Alternatively, the data may be extracted from a proprietary memory device or bus system, or a further system whose hardware and software allow access to the data.

Different principles can be used when transmitting depth and color information. A summary of the related content is as follows: In a first specific design of the invention, a standardized depth map and a standardized color map are transmitted. These spectral results are for example from a specially defined field of view or a specially defined observer position. Such a particular field of view is, for example, defined to give the viewer a central location and a given distance before the display screen. The position of the observer is usually defined by the position of the pupil of his eye.

In the preferred embodiment, the depth map corresponding to the pupil position of an observer's eye is transmitted for each viewer. This means that two depth maps are transmitted for each observer (one for each eye). Alternatively, it is possible to transmit only the depth map corresponding to the pupil position of the eye, especially if the second field of view is available from a depth map.

These embodiments are analogously applied to color maps.

In an embodiment relating to a color map, a color map corresponding to the pupil position of an observer's eye is transmitted for each viewer. Two color maps are transmitted for each observer. It is also possible to transmit only the color map corresponding to the pupil position of a single eye, and accordingly the color map is used for the second pupil because the chromatic aberration typically occurring when the holographic value is generated can be ignored. If the resulting color difference is negligible or acceptable, it is more likely to use a color map for another observer.

Preferred forms of data sources, data receivers, transmission devices, and communication protocols are listed below. Sources of the interface include, for example, the memory of an image card, or one or more image system links of a computer, workstation, CD player, or DVD player. The information is particularly preferably provided by one or more data servers and is preferably transmitted by a plurality of parallel transmission devices. Another preferred source of information is provided by a PCI Express ("Peripheral Component Interconnect Express") bus that allows connection to the image system memory device to be provided, as well as the depth and color map to be read. . Further inventions that allow memory segments to be accessed, and depth and color maps to be read, and future busbar systems form a possible source of information.

Protocols include, for example, cable, satellite, internet, wireless local area network, Ethernet, IEEE 802.11, Digital Video Interface (DVI), or other appropriate agreement currently available or available for future use. These agreements usually comply with industry standards or corresponding recommendations. For example, the Digital Video Interface (DVI) is a standard for displaying the Job Group DDWG based on digits. According to a particularly preferred embodiment of the interface of the present invention, the source of the data includes two The DVI outputs points so that the depth and color maps are transmitted separately but simultaneously as described above.

An external hologram encoding unit or a hologram display device including one encoding unit is particularly preferably provided as a data receiver. Particularly preferred coding units will be described in the individual specific designs below. The coding unit also contains a dedicated computing unit, such as a CPU, GPU or specially designed processor. The method used by the coding unit illustrates the computational load required by the scene rendered by the depth map to reduce the coding unit. When only the minimum amount of data is involved, the depth and color maps are used to provide all the necessary information to the coding unit. The field of view, visibility, and the like have been defined by the depth map so that these jobs need not be determined by the coding unit.

The interface and circuit configuration of the present invention allows image material from a video holographic image to be generated for simple and economical transmission. The computational load of the coding unit is therefore minimized. The present invention further allows for a regional and holistic network connection of systems in a digital video full field.

A first preferred embodiment for an observer is described below. For this observer, for example, a depth map corresponding to its pupil position is transmitted from the interface to the coding unit. This means that two depth maps are transmitted. The same applies to the color map. The color map corresponding to the pupil position of the observer's eye is transmitted as an observer. This means that two color maps are transmitted, the above data is extracted from the data server and transmitted to the coding unit using the DVI protocol. The depth information and color information are each transmitted individually by a DVI cable. However, the same communication protocol is used here.

The coding unit representing the data receiver is a method for generating a video holographic image based on a holographic display device as described in the document DE 10 2006 042 324, wherein the display device comprises at least a scene segmented into object points encoded into a complete hologram. a light modulator device, wherein the scene can be viewed as a reconstruction of a visible region from a periodic interval of video holographic reconstruction, wherein the visible region defines a sub-image with the individual object points of the scene to be reconstructed, and wherein the image is complete The hologram is formed by the superposition of sub-images, and wherein the sub-image contribution for the complete hologram can be extracted from the at least one search table by the coding unit for each object point of the depth map. As illustrated by the method, the computational load of the coding unit is reduced because the information content of the depth of the object point is directly read from the depth map transmitted by the interface of the present invention. Using this information, the contribution of the corresponding sub-hologram can be extracted from at least one of the search tables. A complete hologram can be generated instantly.

The second embodiment illustrates the following (again for an observer to simplify things. For the observer, the depth map corresponding to the pupil position of the eye is transmitted from the interface to the coding unit, which is similar to the above example. Two depth maps are transmitted, but only one color map corresponding to the pupil position of the eye is transmitted. The data is extracted from the image card memory, in detail, the color spectrum is from the frame buffer, and the depth map is from The Z-buffer of the video card. The depth and color maps are transmitted, for example, via Ethernet. The coding unit is used (which determines the depth information from the first eye pupil position for the second eye pupil position). The transmission of the second depth map may be omitted when based on the hologram value. Based on the production as described in document WO/2006/666966 Another particularly preferred coding unit for the video holographic method is provided as a data receiver. This coding unit performs the following steps of computer assistance for the scene represented by the depth map.

- the diffracted image is calculated from each object data set of each tomographic scene segment in a respective two-dimensional distribution of the wavefields of the observer plane, the observer plane being at a finite distance of the segment plane and parallel to the A segment plane in which the wavefields of all segments are calculated for at least one generally visible region.

- the calculated distribution of all segments is added to define the aggregated wavefield for the visible region in the dataset, which is the corresponding observer plane;

- generating holographic data for a holographic image produced by a general computer, the reference set being converted to a holographic plane, the holographic plane being at a finite distance of the reference plane and parallel to the reference plane, wherein the optical modulator device Located in the hologram plane.

The so-called "conversion" should be considered to include any mathematical calculation or technique that is the same or similar to the conversion. Mathematical transformations are only approximate physical procedures, which provide a more accurate description by Maxwellian's volatility formula. For example, a Fresnel transformation, or a transformation of a special transformation group called Fourier transformation, describes a second-order approximation. Image conversion is usually represented by algebraic and non-differential, and can therefore be processed efficiently and with high performance using known calculations. Furthermore, it can be accurately used using an optical system Patterned.

The embodiments described above can of course be selectively combined for multiple viewers. This applies in particular to the transmission of depth and color maps, ie whether one or two depth maps and one or two color maps are selectively transmitted for an observer.

Figure 1 shows a third embodiment of the circuit arrangement between the image card and the dedicated computing unit in which the encoding unit is implemented. A coding unit (HEU), for example, implements one of the methods described above. The computing unit of the coding unit (HEU) is a city image processor (GPU) that can be integrated into the image system of the image card or can be spatially separated. It is shown separately in the schema to simplify things. The data source is represented by a 3D imaging pipeline (3D-RGP) frame buffer (FB) and Z-buffer (ZB). The color map can be read from the frame buffer (FB) while the depth map is read from the Z buffer (ZB). The frame buffer and the Z buffer are typically individual memories. If a common memory is used, the frame buffer and the Z buffer are distinguished by specifications. Transmission lanes (L1, L2, ...) (ie at least one transmission lane (L1) for the depth map and at least one transmission lane (L2) for the color map) are implemented as data sources and data receivers in accordance with the present invention The data link is allowed to allow depth and color maps to be transmitted separately. If only one data link is provided, the depth map and color map will be transmitted by time division multiplexing.

The technology disclosed in this case can be implemented by a person familiar with the technology, and its unprecedented practice is also patentable, and the application for patent is filed according to law. However, the above embodiments are not sufficient to cover the scope of patents to be protected in this case. Therefore, the scope of the patent application is attached.

3D-RGP‧‧‧3D imaging line device

ZB‧‧Z buffer

FB‧‧‧ frame buffer

HEU‧‧‧ coding unit

FIG. 1 is a circuit configuration of an image card and a dedicated arithmetic unit in a coding unit according to the present invention.

3D-RGP‧‧‧3D imaging line device

ZB‧‧Z buffer

FB‧‧‧ frame buffer

HEU‧‧‧ coding unit

Claims (26)

An interface for a transmission of digital image data for at least one hologram encoding unit, the hologram encoding unit generating complex values and/or coded pixel values from image data having depth information to control a hologram display device At least one optical modulator device, wherein the interface transmits a depth map and a color map of the image data by using a transmission device and using a communication protocol, wherein the depth map includes the depth information, and the depth map includes the depth information, and the depth map includes the depth information. The color map contains the color information of the pixelized image of an image sequence. An interface for the transmission of digital image data as described in claim 1 wherein a normalized depth map and a normalized color map are transmitted. An interface for the transmission of digital image data as described in claim 1, wherein the depth map corresponding to the position of the one eye pupil is transmitted for at least one observer, or corresponds to any eye pupil The two depth maps, and/or wherein the color map corresponding to one of the pupils of one of the eye pupils is selected for at least one observer, or a two color map corresponding to any of the pupils of the eye. An interface for the transmission of digital image data as described in one of claims 1-3, which transmits the depth map from a Z-buffer and a frame buffer from an image card. The color map of the device or other dedicated memory segments from an imaging system. An interface for the transmission of digital image data as described in any one of claims 1-3, wherein the transmission is for the holographic coding unit, wherein a scene represented by the depth map is used A computer-assisted coding unit performs the following steps: - for each observer plane from each object profile of each tomographic scene segment A form of a two-dimensional distribution of the wave field calculates a diffraction image, the observer plane being located at a finite distance of the segment plane and parallel to the segment plane, wherein all segments are calculated for at least one generally visible region The wave field; - the calculated distribution of all segment layers is added to define a concentrated wave field of the visible region in a data set corresponding to the observer plane position; - a hologram generated for a general computer of the scene Generating a holographic data set, converting the reference data set to a holographic plane, the holographic plane being located at a finite distance of the reference plane and parallel to the reference plane, wherein the optical modulator device is located in the hologram In the plane. An interface for the transmission of digital image data as described in one of claims 1 to 3, wherein the transmission is for a hologram encoding unit for a hologram display device, wherein the hologram display The device includes at least one light modulator device, on the at least one light modulator device, a scene that is decomposed into object points is encoded as a complete hologram, and wherein the scene can be viewed as a visible region Reconstructing, the visible region is located in the reconstructed one-period interval of the video hologram, wherein the visible region defines a sub-image with the object points of the scene to be reconstructed, and wherein the complete hologram is sub-holographic The superposition is formed, and wherein for each object point of the depth map, the contribution of the sub-hologram to the entire reconstruction of the scene can be extracted from the at least one search table by the coding unit. An interface for the transmission of digital image data as described in claim 6 of the patent application, the transmission being directed to a holographic coding unit, wherein a field of view of the scene represented by the transmitted depth map is by an observer The position and the direction of the viewpoint are defined, and wherein the computer-assisted coding unit performs the following steps for each object point of the depth map: - Step (1): finding the sub-image for each object point The position; - step (2): extracting the contribution of the corresponding sub-hologram from at least one search table; - step (3): repeating the two steps for all object points, wherein the sub-images are stacked Add to form a complete hologram of the reconstruction of the entire scene. The interface for the transmission of digital image data as described in one of claims 1-3, wherein the depth map and the color map are transmitted by one or more links of an image system. The interface for the transmission of digital data images as described in one of claims 1-3, which transmits the depth map and color map from one or more data servers. An interface for the transmission of digital image data as described in any one of claims 1-3, wherein the communication protocol comprises an internet protocol, an Ethernet network, a wireless range network, an IEEE 802.11, a DVI. Or similar. An interface for the transmission of digital image data according to any one of claims 1-3, wherein the transmission is for a holographic coding unit that generates a plurality of image data having depth information. Full image value and/or coded pixel value for controlling at least one light modulator device of a hologram display device, wherein the hologram display device includes at least one light modulator device on the light modulator device a scene that is decomposed into object points is encoded into a complete hologram, and wherein the scene can be viewed as a reconstruction of a visible region located in the reconstructed one-cycle interval of the video hologram, wherein The visible region defines a sub-image of the object along with the object points to be reconstructed, the scene being represented by the depth map, and wherein the complete hologram is formed by a superposition of the sub-images. An interface for the transmission of digital image data as described in claim 11 of the patent application, the transmission being directed to a holographic display device having a screen device, wherein the holographic device is encoded by the hologram of the scene The light modulator device itself, or wherein the screen device is encoded on the light modulator device A hologram or an optical component to which the wavefront is projected. An interface for the transmission of digital image data as described in claim 12, wherein the optical component of the holographic display device is a lens or mirror. A circuit configuration for a transmission of digital image data for at least one hologram encoding unit, the at least one hologram encoding unit generating a complex value and/or a coded pixel value from image data having depth information to control at least one At least one optical modulator device of a holographic display device, wherein the circuit configuration separately transmits a depth map and a color map of the image data by using a transmission device and using a communication protocol, wherein the depth map includes the depth Information and the color map includes color information of a pixelized image of an image sequence. A circuit configuration for the transmission of digital image data as described in claim 14 wherein a normalized depth map and a normalized color map are transmitted. The circuit configuration for the transmission of digital image data as described in claim 14, wherein the depth map corresponding to a position of an eye pupil is transmitted for at least one observer, or corresponds to any eye. The two depth maps of the pupil, and/or wherein the color map corresponding to one of the pupils of one of the eye pupils is selected for at least one observer, or a two color map corresponding to any of the pupils of the eye. A circuit configuration for the transmission of digital image data according to any one of the preceding claims, wherein the depth map from a Z-buffer and a frame from an image card are transmitted. The color spectrum of the buffer, or other dedicated memory segments from an imaging system. A circuit configuration for the transmission of digital image data according to any one of the preceding claims, wherein the transmission is for the hologram encoding a unit in which the following steps are performed by a computer-assisted coding unit using a scene represented by the depth map: - a wave from each object data group of each tomographic scene segment for an observer plane A form of a two-dimensional distribution of the field calculates a diffracted image, the observer plane being located at a finite distance of the segment plane and parallel to the segment plane, wherein all segments are calculated for at least one generally visible region The wave field; - the calculated distribution of all segment layers is added to define a concentrated wave field of the visible region in a data set corresponding to the observer plane position; - a hologram generated for a general computer of the scene Generating a hologram data set, converting the reference data combination into a holographic plane, the holographic plane being located at a finite distance of the reference plane and parallel to the reference plane, wherein the optical modulator device is located in the hologram plane in. A circuit configuration for the transmission of digital image data as described in one of claims 14 to 16, wherein the transmission is for a hologram encoding unit for a hologram display device, wherein the hologram The display device includes at least one light modulator device, on the at least one light modulator device, a scene that is decomposed into object points is encoded as a complete hologram, and the scene can be viewed as a visible region Reconstructing, the visible region is located in the reconstructed one-period interval of the video hologram, wherein the visible region defines a sub-image with the object points of the scene to be reconstructed, and wherein the complete hologram is sub-holographic A superposition is formed, and wherein for each object point of the depth map, the contribution of the sub-hologram to the entire reconstruction of the scene can be extracted from the at least one search table by the coding unit. The circuit configuration for the transmission of digital image data as described in claim 19, wherein the transmission is for a holographic coding unit, wherein the depth of the transmission is defined by the position and the direction of the field of view of an observer. A field of view of the scene represented by the map, and wherein for each object point of the depth map, the computer-assisted coding unit performs the following steps: - step (1): finding the position of the sub-hologram for each object point; - step (2): extracting the contribution of the corresponding sub-hologram from at least one search table; - step (3): for all The object points repeat the two steps, wherein the sub-images are superimposed to form a complete hologram of the reconstruction of the entire scene. A circuit arrangement for the transmission of digital image data as described in one of the preceding claims, wherein the depth map and the color map are transmitted by one or more links of an image system. A circuit configuration for the transmission of digital image data as described in any one of claims 14 to 16, wherein the depth map and color map are transmitted from one or more data servers. A circuit configuration for the transmission of digital image data according to any one of the preceding claims, wherein the communication protocol comprises internet, Ethernet, WLAN, IEEE 802.11, DVI or Similar. The circuit configuration for the transmission of digital image data as described in one of the above-mentioned claims, wherein the holographic coding unit uses the depth information from the omni-directional coding unit. The image data generates a plurality of full image values, and/or a coded pixel value to control at least one light modulator device of a hologram display device, wherein the hologram display device includes at least one light modulator device, at least A scene of a light modulator device that is decomposed into object points is encoded into a complete hologram, and wherein the scene can be viewed as a reconstruction of a visible region located in the reconstruction of the video hologram In a periodic interval, wherein the visible region defines a sub-image with the object points to be reconstructed, the scene is represented by the depth map, and wherein the complete hologram is formed by a superposition of the sub-images. The transmission for digital image data as described in claim 24 of the patent application Circuit arrangement for a hologram display device having a screen device, wherein the screen device is itself of the light modulator device in which the hologram of the scene is encoded, or wherein the screen device is the light A full image or wavefront of the scene encoded on the modulator device is projected onto one of the optical elements. The circuit arrangement for the transmission of digital image data as described in claim 25, wherein the optical component of the holographic display device is a lens or a mirror.