DE102012002442B3 - Method for processing monoscopic, stereoscopic or multiview image signals for e.g. factory planning application, involves performing computation of geometric distortion and determination of color independent-calibration parameter - Google Patents

Abstract

The method involves carrying out production of output signals by decoding input signals for separation of views and combining an application of calibration parameters with a coding of the views in a required output-format. An input pixel actually influencing the output signals is generally passed through calibrating sub steps. Computation of a geometric distortion and determination of pixel-specific, position-dependent and color independent-calibration parameter are performed for an output pixel position even when the views influence the output signals at a position. An independent claim is also included for a device for controlling stereoscopic or multiview-capable multisegment-projection systems.

Description

Technical area

The invention relates to a method for processing image signals for multi-segment projection systems, which are constructed from monoscopic or stereoscopic projectors, as well as a device for controlling such multi-segment projection systems with one or more video signals, the image content of one or more views.

State of the art

Stereoscopic vision systems are used today in many areas. There are a variety of different methods for providing different image contents to the human eye viewers for the left and right eyes. In stereoscopic imaging, a distinction can be made between systems which process the individual images of a stereoscopic image pair separately (for example two separate projectors projecting onto the same surface) and systems which collect stereoscopic image pairs in summary and have only one imaging unit which is suitable for both frames are used (such as shutter methods, anaglyph glasses).

• – Verzerrungen, die von der Projektor-Optik verursacht werden ([1])
• – Verzerrungen durch nicht exakt senkrechte Projektion (Trapezentzerrung, [2])
• – Verzerrungen durch nichtebene Projektionsflächen
• – Helligkeits- und Farbunterschieden innerhalb eines Segments ([3])
• – Helligkeits- und Farbunterschieden zwischen den Segmenten ([4], [5])
• – Helligkeitsunterschieden in den Überlappungsbereichen (Edge-Blending, [4])

For many purposes such as product development, virtual prototyping, factory planning or product presentations, a very large image area is required with high image resolution. Individual display systems do not meet such requirements, so use segmented display systems that combine separate display systems such that a homogeneous image impression over the total area arises. A widespread feature of this technique are multi-segment projection systems in which project several projectors on individual portions of a common projection screen, the images of adjacent projectors can overlap partially. In these systems, the need for calibration to compensate for:

• - Distortions caused by the projector optics ([1])
• - Distortions due to not exactly vertical projection (trapezoidal strain, [2])
• - Distortions by non-planar projection surfaces
• - Brightness and color differences within a segment ([3])
• - Brightness and color differences between the segments ([4], [5])
• - Brightness differences in the overlapping areas (edge blending, [4])

For calibration of such systems different approaches are pursued. The spectrum ranges from fully manual procedures, where the correction parameters have to be set manually, through interactive and semi-automatic procedures to fully automated camera-based systems. Special calibration patterns are displayed on the system and the actual status is recorded by means of a camera. From this, the calibration system calculates the calibration parameters required for transferring the actual state into the desired state. These can be applied to image content and cause a geometric distortion (warping [1], [2], [5], [7], [4], [8], [9]) of the image data as well as an adjustment of the color values ([ 4], [5]). If the image data processed in this way are reproduced with the projectors, the result is a homogeneous image impression on the projection surface.

• a) Benutzung der doppelten Anzahl (monoskopischer) von Projektoren, so dass jeder Punkt der Bildfläche von jeweils mindestens zwei Projektoren ausgeleuchtet wird, in Kombination mit einem geeigneten Filtersystem zur Trennung der Bildpaare (zum Beispiel Polarisationsfilter) oder
• b) Einsatz von stereoskopischen Projektoren.

Stereoscopic multi-segment projection systems can be realized in two ways:

• a) using twice the number (monoscopic) of projectors, so that each point of the image area is illuminated by at least two projectors, in combination with a suitable filter system for the separation of image pairs (for example polarization filter) or
• b) Use of stereoscopic projectors.

The variant a) has the advantage that it can be integrated directly into the previous image processing chain, because each segment only - just as in the monoscopic case - must take into account a section of a single image. However, such systems are in principle complex and expensive, in addition to doubling the cost of the projector also results in a greater effort for mechanical attachment of the projectors. Any necessary off-axis projections and the use of lens shift lead to stronger distortions. Other disadvantages are the increased space requirements and power consumption. It is therefore desirable to use stereoscopic projectors in multi-segment projection systems.

• – nebeneinander oder untereinander mit jeweils halbierter Auflösung pro Ansicht
• – zeilenweise oder spaltenweise verschachtelt
• – Schachbrett-Muster (insbesondere bei DLP-Projektoren verbreitet)
• – Sequentielle Zuführung mit doppelter Frequenz (typisch 120 Hz)

There is currently no universally accepted standard for delivering stereoscopic image content to such projectors, but a number of different stereo formats that describe how the two frames must be encoded within the overall image:

• - side by side or below each other, each with halved resolution per view
• - nested line by line or column by column
• - Checkerboard pattern (especially popular with DLP projectors)
• - Sequential feeding with double frequency (typically 120 Hz)

Because of this, there is a need to be able to interconvert stereomata. There are individual methods and devices that can perform such conversions ([10], [6]).

The method and the device according to [6] allows the conversion of the image content in real time and can work with any number of input and output signals, the image data can also be further processed to z. As to scale the images or adjust the stereoscopic orientation, so to change the positioning of the left and right view to each other. However, the method or the device is not sufficient for driving stereoscopic multi-segment projection systems, since it does not apply a calibration method, but solves only the sub-problem of controlling the projectors in the respective stereoscopic format, or when using several monoscopic projectors on the projecting the same image area, including the separate feeding of the views and, if appropriate, the compensation of moderate positional differences by adjusting the stereoscopic alignment and keystone correction. All image distortions occurring beyond this (eg due to the projection optics or not perfectly planar projection surfaces) are thereby neglected.

Problem description / disadvantages of the previous methods

When using stereoscopic projectors within multi-segment projection systems, the problem arises that the calibration parameters must be used individually for each image view. It is generally not easily possible to directly equalize an input signal prepared for such a projector, since the stereo coding would be destroyed by the geometric distortion.

The obvious approach to solving this problem is decoding the input signals into the individual image views, applying the calibration parameters to the individual views, and re-encoding the views into a format suitable for the projector. The method from [6] works according to this principle: the image contents are separated into two separate channels for the left and right view and the image processing is carried out separately on these channels, the combination of the method from [6] with a calibration method then only provides an extension the already existing image processing steps.

The main disadvantage of this approach lies in the high computational effort and the resulting costs for computer systems. In particular, there are modern graphics cards with six outputs. This allows, for example, a multi-segment projection system of 3 × 2 stereoscopic projectors operate with only a single computer, if it is possible to perform the necessary data processing efficiently using this graphics card.

• a) die Kalibrierungsverarbeitung für alle Pixel aller Ansichten durchgeführt wird, unabhängig davon, welche Ansichten auf Grund des benötigten Ausgabeformates tatsächlich in den jeweiligen Ausgabepixeln sichtbar sind, und
• b) das Kalibrierverfahren immer unabhängig für alle Ansichten durchgeführt wird, und dabei für alle Ansichten bei einer gegebenen Ausgabeposition identische Kalibrierschritte mehrfach durchgeführt werden.

The naive approach leads to an increased processing effort, since

• a) the calibration processing is performed for all pixels of all views, regardless of which views are actually visible in the respective output pixels due to the required output format, and
• b) the calibration procedure is always performed independently for all views, and identical calibration steps are carried out repeatedly for all views at a given output position.

goal

The aim of the present invention is to control stereoscopic multi-segment projection systems consisting of monoscopic or stereoscopic projectors, wherein the input signals represent any monoscopic, stereoscopic or multiview image contents. In addition to monoscopic content, stereoscopic and multiview image data in different stereo or multiview formats should be supported in the image reception as well as the image output and an automatic conversion of the formats should be carried out.

The object of the invention is to apply the calibration parameters in real time to these image contents. In order to be able to process a high overall resolution, the efficiency of the image processing is of crucial importance. Therefore, it is not necessary to perform the stereo conversion and application of the calibration data as separate steps (by first separating the stereoscopic images, applying the calibration sequentially to the frames of the image pair, and finally combining them back to the output image in the desired format), but it is the objective the invention of combining the generation of the stereoscopic image output directly with the application of the calibration such that only the pixels actually visible in the output image must be taken into account in the application of the calibration and sub-steps in the application of the calibration which are identical for both individual images, also be executed only once.

In addition to stereoscopic vision systems, there are already the first multiview systems that use more than two views of a scene. On the one hand, this enables the realization of multi-user stereo systems in which each user is presented with their individual stereoscopic view. On the other hand, this is used for autostereoscopic vision systems. Such systems will become more widespread in the future. In view of this, it is a further object of the present invention to also process more than the two views required for conventional stereoscopy.

Description of the invention

• – einem oder mehreren Rechnersystemen mit je – einer oder mehreren Grafikkarten mit je einem oder mehreren Ausgängen – einem oder mehreren Videograbbern mit je einem oder mehreren Eingängen

In order to achieve the aforementioned objects, the invention proposes a method for processing monoscopic, stereoscopic and multiview image contents for the control of multi-segment projection systems. The invention further includes a device consisting of

• - One or more computer systems, each with - one or more graphics cards, each with one or more outputs - one or more video grabbers, each with one or more inputs

The total number of video card outputs in the system is denoted by A below, the total number of video inputs by E. To operate a multi-segment projection system with P separate projectors, the system must be designed so that A> = P. The number of inputs E must be> = 1.

List of formula symbols:

• A number of outputs
• B _a Image width (number of pixels) of output a
• C = (C _x , C _y ) normalized coordinates in the input coordinate system, (0, 0) corresponds to left lower corner and (1, 1) right upper corner
• D _e input image format of input e
• E number of inputs
• F = (F _r , F _g , F _b ) RGB color sample from an input image buffer
• F '= (F' _r , F ' _g , F' _b ) corrected RGB color sample after input format decoding
• G _a Output image format of output a
• H _a Image height (number of pixels) of output a
• K = {K ₀ , ... K _k-1 } List of intermediate views of a specific output pixel
• L = {L ₀ , ..., L _l-1 } List of active intermediate views
• M _e image width (number of pixels) of input e
• M ' _e Width of an input pixel in the normalized coordinate system
• N _e Image height (number of pixels) of input e
• N ' _e height of an input pixel in the normalized coordinate system
• O = (O _x , O _y ) pixel coordinates in the output coordinate system, (0, 0) corresponds to left lower corner and (B _a-1 , H _a-1 ) right upper corner
• P number of projectors
• Q size per input ring buffer
• R _e Number of used entries in the input ring buffer of input e
• S = {S ₀ , ..., S _s-1 } List of inputs in a synchronization group
• T = (T _x , T _y ) Coordinates in the device coordinate system of the projector
• U = (U _x , U _y ) normalized coordinates in the intermediate coordinate system (0, 0) corresponds left lower corner and (1, 1) right upper corner
• V _i i th view of an input or output image buffer
• 
  List of intermediate views associated with output a
• _Zj = ( _{Zj, r} , Zj _{, g} , Zj _{, b} ) RGB color sample from the intermediate view's intermediate view associated with the output view j
• Z combined color sample of all views described as a vector of color components, not necessarily three-dimensional
• Z 'output color sample after correction by the calibration method
• a Index of an output, 0 <= a <= A - 1
• e index of an input, 0 <= e <= E - 1
• i Intermediate view
• j Index of an input or output view
• k Number of intermediate views of a specific output pixel
• l Number of active intermediate views
• p _i pixels in the intermediate image buffer of view i
• p _a pixel in the output image buffer of view a
• s Number of inputs in a synchronization group S
• v _e Number of input views of input e
• w _a Number of output views of output a
• α pixel-specific calibration parameters (depending on the device coordinates, but independent of the pixel color)
• β _a General calibration parameters for projector at output a
• δ ₀ , δ ₁ Parameter for input format decoding anaglyph
• μ ₀ , μ ₁ , μ _m Parameters for output format coding anaglyph
• ε _e Parameter for multiview input format decoding
• ε _a Parameter for the multiview output format coding

Used mathematical functions

• x mod y: x modulo y: Rest der Ganzahldivision x/y
• floor(x): x gerundet in Richtung der nächstkleineren Ganzzahl
• dot(x, y) = x_x·y_x + x_y·y_y + x_z·y_z: Skalarprodukt der dreidimensionalen Vektoren
• clamp(x, a, b) = a, falls x < a x, falls a <= x <= b b, falls x > b
• nearest(x, y) = (floor(x·y – 0.5) + 0.5)/y
• factor(x, y) = (x·y – 0.5) – floor(x·y – 0.5)
• pixel(x, e): Farbwert (RGB-Vektor) des Pixels des Eingabe-Bildpuffers von Eingang e, dessen Pixelzentrum den Koordinaten x = (x_x, x_y) am nächsten liegt, wobei das Koordinatensystem derart normiert ist, dass (0, 0) der linken unteren Bildecke und (1, 1) der rechten oberen Bildecke entsprechen. Die tatsächlichen Pixelkoordinaten x' zum Zugriff auf den Bildpuffer mit der gegebenen Auflösung M_e·N_e entsprechen dabei x' = (clamp(floor(x_x·M_e), 0, M_e – 1), clamp(floor(x_y·N_e), 0, N_e – 1).

bilinear(f, x₁, x₂, x₃, x₄, e) = (1 – f_y)·((1 – f_x)·pixel(x₁, e) + f_x·pixel(x₂, e)) + f_y·((1 – f_x)·pixel(x₃, e) + f_x·pixel(x₄, e)) In describing the method, reference is made to mathematical functions defined herein:

• x mod y: x modulo y: remainder of the integer division x / y
• floor (x): x rounded in the direction of the next smallest integer
• dot (x, y) = x _x · y _x + x _y · y _y + x _z · y _z : scalar product of the three-dimensional vectors
• clamp (x, a, b) = a, if x <ax, if a <= x <= bb, if x> b
• nearest (x, y) = (floor (x · y - 0.5) + 0.5) / y
• factor (x, y) = (x · y - 0.5) - floor (x · y - 0.5)
• pixel (x, e): color value (RGB vector) of the pixel of the input image buffer of input e whose pixel center is closest to the coordinates x = (x _x , x _y ), the coordinate system being normalized such that (0 , 0) correspond to the lower left corner and (1, 1) to the upper right corner. The actual pixel coordinates x 'for accessing the image buffer with the given resolution M _e · N _{e in} this case correspond to x' = (clamp (floor (x _x * M _e ), 0, M _e - 1), clamp (floor (x _y · N _e ), 0, N _e - 1).

bilinear (f, x ₁ , x ₂ , x ₃ , x ₄ , e) = (1-f _y ) * ((1-f _x ) * pixel (x ₁ , e) + f _x * pixel (x ₂ , e)) + f _y * ((1-f _x ) * pixel (x ₃ , e) + f _x * pixel (x ₄ , e))

• – Platzierung und Ausrichtung der Projektoren. Jedem Projektor muss nur der Teilausschnitt zugeführt werden, der seinem Anteil auf der gemeinsamen Projektionsfläche entspricht
• – Ausgleich geometrischer Verzerrungen der Projektors
• – Ausgleich von Helligkeitsunterschieden in den Überlappungsbereichen mehrerer Projektoren
• – Angleichen von Farben und Helligkeiten zwischen den Projektoren
• – Angleichen von Farb- und Helligkeitsunterschieden innerhalb des projizierten Bildes eines einzelnen Projektors.

The method described herein makes reference to a calibration method for rendering image content regarding at least one of the following aspects:

• - Placement and orientation of the projectors. Each projector must be fed only the partial section, which corresponds to its share on the common projection screen
• - Compensation of geometric distortions of the projector
• - Compensation of brightness differences in the overlapping areas of several projectors
• - Matching colors and brightness between the projectors
• - Matching color and brightness differences within the projected image of a single projector.

• warp_a(x, β_a): geometrische Anpassung: Berechnet 2D-Koordinaten im Intermediate-Koordinatensystem, welches dem Ausgabepixel mit den Gerätekoordinaten x = (x_x, x_y) vom Projektor am Ausgang a entspricht. Damit lässt sich sowohl der grundsätzliche Bildausschnitt als auch ein Image-Warping zur Entzerrung der Projektorbilder realisieren.
• param_a(x, β_a): Berechnung positionsabhängiger aber farbunabhängiger Parameter: Berechnet einen nicht näher spezifizierten Parametervektor mit Kalibrierungsparametern in Abhängigkeit des Ausgabepixels mit den Gerätekoordinaten x = (x_x, x_y) vom Projektor am Ausgang a. Dieser Parametervektor ist nur abhängig von der Pixelposition, nicht aber der später zu setzenden Pixelfarbe oder der gewählten stereoskopischen Ansicht. Als Beispiel eines solchen Parameters kann der Blend-Faktor für Soft-Edge-Blending genannt werden.
• color_a(Z, α, β_a): Anpassung der Farben: Berechnet den Farbvektor, der die korrigierte Projektor-Eingangsfarbe für den Projektor am Ausgang a und die gewünschte Farbe Z beschreibt, dabei kann auf die Parameter α = param_a(x) zurückgegriffen werden, die getrennt von dieser Funktion berechnet wurden.

So that this can be combined with the method presented here, it is assumed that the necessary image processing can be described according to the calibration method with the three mathematical functions defined below:

• warp _a (x, β _a ): geometric fit: Computes 2D coordinates in the intermediate coordinate system, which corresponds to the output pixel with the device coordinates x = (x _x , x _y ) from the projector at output a. Thus, both the basic image detail and an image warping for equalization of the projector images can be realized.
• param _a (x, β _a ): Calculation of position-dependent but color-independent parameters: Calculates an unspecified parameter vector with calibration parameters depending on the output pixel with the device coordinates x = (x _x , x _y ) from the projector at output a. This parameter vector is only dependent on the pixel position, but not the pixel color to be set later or the selected stereoscopic view. As an example of such a parameter, the blend factor for soft edge blending may be mentioned.
• color _a (Z, α, β _a ): Adjusting the colors: Computes the color vector that describes the corrected projector input color for the projector at the output a and the desired color Z, while allowing the parameters α = param _a (x) be recalculated separately from this function.

All three functions have the parameter β _a , which represents all specific parameters of the selected calibration method for the projector at the output a.

The process for processing the image content is carried out in 3 phases:

Phase 1: Installation of the system

a): Configuration of the system

• - Each input e of the E inputs are assigned manually: - an input format D _e , and in the case of a multiview input format also a parameter ε _e . The input format specifies the number of input views v _e, each of _e v input views any number of intermediate views is assigned. The possible input formats are described in more detail in the section "Conversion of input formats". - Parameter values δ _{0, e} and δ _{1, e} , if anaglyphes input format was selected. The position and orientation of the segment in the normalized intermediate coordinate system in the form of four 2D point coordinates describing a rectangle. A synchronization group characterized by an integer. All inputs to which the same number is assigned form a synchronization group.
• - For each output a of the A outputs: - Determination of the calibration parameters β _a for the projector operated at this output a. - Manual assignment of an output format G _a , which is suitable for this projector. In the multiview case, the parameter ε _a must be set additionally. The output format determines the number of output views w _a , with each of the w _a output views being assigned exactly one intermediate view. The list is created
  
  The possible output formats are described in more detail in the section "Conversion of output formats". - Definition of the pixel resolution: B _a width, H _a height - Definition of the parameter values μ ₀ , μ ₁ and μ _m if an anaglyphic output format has been selected.
• - Configuration of further parameters - Determine the pixel resolution of the intermediate frame buffer. - Determination of the ring buffer size Q

Phase 1a) is carried out after the system has been set up, as well as whenever the playback situation or configuration of the system (number and type of projectors) has changed. It should be noted that such configuration data can be stored in computer systems and retrieved again, so that a fast switching between previously defined feed situations can be realized.

The determination of the actual calibration parameters β _a remains unaffected. The procedure and times of calibration depend directly on the calibration procedure used, but have no further influence on the configuration of this method.

b) Initialization of the intermediate image buffer

• - Determination of the actually required intermediate views by listing all assigned intermediate views on all A outputs, with duplicates removed, resulting in a list L = {L ₀ , ..., L _l-1 }, which are the l actually relevant (because there are at least one projector output) intermediate views.
• Generation of an intermediate image buffer containing 1 separate color buffers. The resolution of each buffer is identical and is set in phase 1a).

Phase 1b) must be executed when starting the procedure and run again after every configuration change (phase 1a)).

c) initialization of the input image buffer

• - A ring buffer consisting of image buffers and an additional input image buffer are created for each input. The pixel resolution of all buffers of an input is identical in each case and corresponds to the resolution of the signal present at the input from M _e columns and N _e rows.
• - For each input e, the number of used buffers in the ring buffer R _{e is} initialized with 0.

Phase 1c) is executed at the start of the process and iterated during a configuration change or a change in the resolution of an input signal.

d) Initialization of the output image buffers

• - For each output an output frame buffer with B _a columns and H _a rows is created. If G _a = stereo has been selected, then a stereoscopic output buffer is created consisting of a view for the left and a view for the right eye, each in full resolution. If G _a = mv has been selected, a multivew output buffer consisting of ε _a views, each in full resolution, is created.

Phase 1d) is executed at the start of the process and run through again during a configuration change.

Phase 2: Process for the preparation of the image content

In Phase 2, the image data of the input signals are processed and the intermediate image buffers updated accordingly. shows a program flow chart for performing this phase.

a) Updating the input signals

• - As soon as new image information is present at an input e ', it is checked whether R _e' <Q holds. If this is the case, image information is copied to the next free slot of the ring buffer and R _{e '} is incremented by 1, otherwise the image content is discarded (frame dropping).
• Let S = {S ₀ , ..., S _s-1 } be the synchronization group to which input e 'belongs. If the synchronization group SR _e > 0 applies to all s inputs e, then the image content of this synchronization group must be further processed as described in phase 2b), otherwise the synchronization group is not complete and no update of the input image buffer is required and phase 2a) begins again.

b) Updating the input image buffers

• - iteration over all inputs e of the synchronization group S. Copy of the oldest unprocessed image content from input buffer e associated ring buffer in the input e associated input image buffer and decrement of R _e by 1. Thus, the input image buffer contain the new image content of the synchronization group at the synchronization time and the ring buffers have room for at least one more frame.
• Since at least one input image buffer has been updated in phase 2b), the entire intermediate buffer must be updated. Therefore, for each input e of all E inputs (not just the synchronization group), the method described in phase 2c) is performed.

c) processing the input contents

• - iteration over all input views assigned to the input j.
• - iteration over all intermediate views associated with the input from the configuration in phase 1a). Let i be the respective intermediate view. If view i is not present in the list of associated views L, the image information of that view will not be considered, otherwise:
• - iteration over all the pixels p _i in the intermediate coordinate system which are enclosed by the input e assigned rectangular (rasterization), and performing the processing described in the following as phase 2d).

d) Processing of input pixels

• Calculation of the pixel coordinates C in the input coordinate system using an image that maps the intermediate coordinates of the quadrilateral to the pixel coordinates of the vertices of the input image. This leads (optionally) to scaling, rotation and mirroring of the image information.
• - Determination of the color value
  
  by sampling the input image buffer corresponding to step 1 of the input data conversion method depending on the input format D _e , the current view j, and the calculated input pixel coordinates C.
• - Modification of the color value F to color value
  
  depending on the input format D _e and the current view j according to step 2 of the input data conversion method.
• Updating the current pixel p _i in the intermediate image i associated color buffer of the intermediate image buffer with the resulting color value F '.

Conversion of the input formats

• decode1(C, j, e): Sampling des Farbwertes in Abhängigkeit von Pixelposition und Ansicht: In diesem Verarbeitungsschritt wir der Farbwert an den Eingabe-Pixelkoordinaten C der Ansicht j des Bildes im Eingabebildpuffer e ermittelt. Nach Möglichkeit wird eine bilineare Interpolation der Pixelfarbwerte in der 2×2 Pixel großen lokalen Nachbarschaft durchgeführt. Dabei muss berücksichtigt werden, dass benachbarte Pixel einer Ansicht nicht direkt benachbarten Pixeln im Eingabe-Framepuffer entsprechen müssen, es kann daher eine Anpassung der Eingabe-Pixelkoordinaten in Abhängigkeit der Eingabe-Ansicht erfolgen.
• M'_e = 1/M_e: Pixelbreite in normierten Koordinaten
• N'_e = 1/N_e: Pixelhöhe in normierten Koordinaten
• j: Ansicht (Zählung beginnt bei 0)
• C = (C_x, C_y): Die unmodifizierten normierten Eingabe-Pixelkoordinaten
• decode2(F, j, e): Modifikation des Farbwerts in Abhängigkeit der Ansicht: Manche Stereoformate benutzen Farbkanäle zur Kodierung unterschiedlicher Ansichten. In diesen Fällen ist es notwendig, die Farbinformationen in Abhängigkeit der Eingabe-Ansicht zu modifizieren.

In phase 2d), reference was made to the conversion method for input data. This conversion always takes place in two steps, which were referred to here as the mathematical functions decode1 and decode2. These functions are specified below for each supported input format:

• decode1 (C, j, e): Sampling of the color value as a function of pixel position and view: In this processing step, the color value at the input pixel coordinate C of the view j of the image in the input image buffer e is determined. If possible, bilinear interpolation of the pixel color values in the 2x2 pixel local neighborhood is performed. It has to be taken into account that neighboring pixels of a view do not have to correspond directly to neighboring pixels in the input frame buffer, therefore an adaptation of the input pixel coordinates can take place depending on the input view.
• M ' _e = 1 / M _e : pixel width in normalized coordinates
• N ' _e = 1 / N _e : pixel height in normalized coordinates
• j: view (counting starts at 0)
• C = (C _x , C _y ): The unmodified normalized input pixel coordinates
• decode2 (F, j, e): Modification of the color value depending on the view: Some stereo formats use color channels to encode different views. In these cases, it is necessary to modify the color information depending on the input view.

Input format mono: monoscopic

An input view is defined. The input frame buffer contains the input view 0 in full screen. In this picture coding is sketched by way of example. C '= (nearest (C _x , M _e ), nearest (C _y , N _e )) f = (factor (C _x , M _e ), factor (C _y , N _e )) x ₁ = C' x ₂ = C '+ (M' _e , 0) x ₃ = C '+ (0, N' _e ) x ₄ = C '+ (M' _e , N ' _e ) decode 1 _mono (C, j, e) = bilinear (f, x ₁ , x ₂ , x ₃ , x ₄ , e) decode2 _mono (F, j, e) = F

Input format ilh: Interlaced (line by line)

Two input views are defined. The input image buffer contains line by line views 0 (even line numbers) and view 1 (odd line numbers). In this picture coding is sketched by way of example. C ' _x = nearest (C _x , M _e ) f _x = factor (C _x , M _e ) h _y = (C _y * N _e + j-0.5) / 2 C' _y = (floor (h _y ) 2 - j + 0.5) / N _e f _y = (C _y - C ' _y ) / (2 * N' _e ) C '= (C' _x , C ' _y ) x ₁ = C' x ₂ = C ' + (M ' _e , 0) x ₃ = C' + (0.2 · N ' _e ) x ₄ = C' + (M ' _e , 2 · N' _e ) decode1 _ilh (C, j, e) = bilinear ((f _x , f _y ), x ₁ , x ₂ , x ₃ , x ₄ , e) decode _{2 ilh} (F, j, e) = F

Input format ilv: Interlaced (column by column)

Two input views are defined. The input image buffer contains column-by-column alternately view 0 (even column numbers) and view 1 (odd column numbers). In this picture coding is sketched by way of example. h _x = (C _x * M _e + j - 0.5) / 2 C ' _x = (floor (h _x ) * 2 - j + 0.5) / M _e f _x = (C _x -C' _x ) / (2 M ' _e ) C' _y = nearest (C _y , N _e ) f _y = factor (C _y , N _e ) C '= (C' _x , C ' _y ) x ₁ = C' x ₂ = C ' + (2 · M ' _e , 0) x ₃ = C' + (0, N ' _e ) x ₄ = C' + (2 · M ' _e , N' _e ) decode _{1 ilv} (C, j, e) = bilinear ((f _x , f _y ), x ₁ , x ₂ , x ₃ , x ₄ , e) decode _{2 ilv} (F, j, e) = F

Input format cb: Checkerboard (checkerboard pattern)

Two input views are defined, which are spread across the input image buffer in a checkerboard pattern. If the sum of column and row number is even, the pixel will be View 0, otherwise, View 1. In this picture coding is sketched by way of example. h _x = floor ((M _e * C _x -N _e * C _y ) + 1-j) / 2) * 2 + j; h _y = floor ((M _e * C _x + N _e * C _y ) + j) / 2) * 2 + 1-j; C _'x = 0.5 * (h _x + h _y) / M _e C' _y = 0.5 * (h _x - h _y) / N _e decode1 _cb (C, j, e) = pixel ((C _'x, C ' _y ), e) decode2 _cb (F, j, e) = F

Input format sbs: Side-By-Side

Two input views are defined, which are coded as two frames (each with halved resolution) side by side in the input frame buffer. In this picture coding is sketched by way of example. h _x = 0.5 · (C _x + j) C ' _x = nearest (h _x , M _e ) C' _y = nearest (C _y , N _e ) f = (factor (h _x , M _e ), factor (C _y , N _e )) C '= (C' _x , C ' _y ) x ₁ = C' x ₂ = C '+ (M' _e , 0) x ₃ = C '+ (0, N' _e ) x ₄ = C '+ (M' _e , N ' _e ) decode 1 _sbs (C, j, e) = bilinear (f, x ₁ , x ₂ , x ₃ , x ₄ , e) decode _{2 sbs} (F, j, e ) = F

Input format tb: Top-Bottom

Two input views are defined, which are coded as two frames (each with halved resolution) among each other in the input frame buffer. In this picture coding is sketched by way of example. C ' _x = nearest (C _x , M _e ) h _y = 0.5 · (C _y + j) C' _y = nearest (h _y , N _e ) f = (factor (C _x , M _e ), factor (h _y , N _e )) C '= (C' _x , C ' _y ) x ₁ = C' x ₂ = C '+ (M' _e , 0) x ₃ = C '+ (0, N' _e ) x ₄ = C '+ (M' _e , N ' _e ) decode 1 _tb (C, j, e) = bilinear (f, x ₁ , x ₂ , x ₃ , x ₄ , e) decode 2 _tb (F, j, e ) = F

Input format ag: anaglyph

• – δ₀: dreidimensionaler Vektor mit einem Koeffizient pro Farbkanal, beschreibt den Anteil von Ansicht 0
• – δ₁: dreidimensionaler Vektor mit einem Koeffizient pro Farbkanal, beschreibt den Anteil von Ansicht 1

Two input views are defined, which are coded in different color channels. The resulting individual views correspond to a monochrome image content. The coding of the color channels is controlled by the following parameters:

• - δ ₀ : three-dimensional vector with one coefficient per color channel, describes the proportion of view 0
• - δ ₁ : three-dimensional vector with one coefficient per color channel, describes the proportion of view 1

In this picture coding is sketched by way of example. C '= (nearest (C _x , M _e ), nearest (C _y , N _e )) f = (factor (C _x , M _e ), factor (C _y , N _e )) x ₁ = C' x ₂ = C '+ (M' _e , 0) x ₃ = C '+ (0, N' _e ) x ₄ = C '+ (M' _e , N ' _e ) decode1 _ag (C, j, e) = bilinear (f, x ₁ , x ₂ , x ₃ , x ₄ , e) h = dot (F, (1-j) * δ ₀ + j * δ ₁ ) decode 2 _ag (F, j, e) = (h, h, h)

Input format sbsmv: Side-By-Side Multiview

An arbitrary number of ε _e input views are defined, which are coded as frames with the image width M _e / ε _e next to one another in the input image buffer. In this picture coding is sketched by way of example. h _x = (1 / ε _e ) · (C _x + j) C ' _x = nearest (h _x , M _e ) C' _y = nearest (C _y , N _e ) f = (factor (h _x , M _e ), factor (C _y , N _e )) C '= (C' _x , C ' _y ) x ₁ = C' x ₂ = C '+ (M' _e , 0) x ₃ = C '+ (0, N ' _e ) x ₄ = C' + (M ' _e , N' _e ) decode 1 _sbsmv (C, j, e) = bilinear (f, x ₁ , x ₂ , x ₃ , x ₄ , e) decode _{2 sbsmv} ( F, j, e) = F

Input format tbmv: Top-Bottom Multiview

It is defined any number of ε _e input views that are encoded as frames with each image height N _e / ε _e with each other in the input frame buffer. In this picture coding is sketched by way of example. C ' _x = nearest (C _x , M _e ) h _y = (1 / ε _e ) · (C _y + j) C' _y = nearest (h _y , N _e ) f = (factor (C _x , M _e ), factor (h _y , N _e )) C '= (C' _x , C ' _y ) x ₁ = C' x ₂ = C '+ (M' _e , 0) x ₃ = C '+ (0, N ' _e ) x ₄ = C' + (M ' _e , N' _e ) decode 1 _tbmv (C, j, e) = bilinear (f, x ₁ , x ₂ , x ₃ , x ₄ , e) decode _{2 tbmv} ( F, j, e) = F

Completion of phase 2:

After all input data of the synchronization group have been processed, the intermediate image buffer contains the new image information, separated into individual intermediate views, as full images. If the intermediate image buffer has been modified during Phase 2, Phase 3 must now be performed to update the output buffers as well. Otherwise, Phase 2 will start again from the beginning.

Phase 3) Generation of the output signals

In Phase 3, the output image buffers are updated so that eventually the changed input image data is also displayed by the projectors. shows a program flow chart for performing this phase.

a) determination of the intermediate views per pixel,

• - iterate over each output a of the A outputs and iterate over each pixel p _{a of} the output a and perform the procedure described in phase 2b).

b) determining the color values of all output views per pixel,

• - O = (O _x , O _y ) corresponds to the pixel coordinates of p _a ,.
• Calculation of the device pixel coordinates T (T _x , T _y ) according to step 1 of the output data conversion method.
  
• - Determining the intermediate image coordinates U, which are assigned to the device pixel coordinates T. This conversion is done using the stored calibration parameters of the associated projector, which in some form describe a mapping of the device coordinates to the intermediate coordinates, which should not be further considered in the description of this invention. U = warp _a (T, β _a )
• - Determining the device pixel coordinates associated parameter vector α, such. For example, a blend factor for soft edge blending. α = param _a (T, β _a )
• Determination of the list K = {K ₀ ,..., K _k-1 } of the k required output views depending on output format for output coordinates O according to step 2 of the output data conversion method.
  
• Determination of the associated intermediate views i for all views j of the list K by means of the assignment list W _a from output views to intermediate views from the configuration
• For each of the detected intermediate views i: Bilinear filtering of the associated intermediate image buffer at the intermediate image coordinate U. The result for each output view j is a color sample Z _j
• After all color samples of all views j from list K have been calculated, phase 3c) is performed

c) Generation of the output pixel color value

• - Combination of the color samples Z _j all output views according to step 3 of the conversion method for output data for color sampling Z.
  
• - Conversion of the color value to the output color value for the projector based on the calibration parameters. Z '= color _a (Z, α, β _a )
• Updating the output pixel p _a with color value Z '

Conversion of the output formats

• encode1(O, a): Ermittlung der Gerätekoordinaten in Abhängigkeit der Ausgabepixelposition: Diese Geräte-Pixelkoordinaten dienen als Eingabekoordinaten der Image-Warping-Stufe der Geometriekalibrierung.
• encode2(O, a): Ermittlung der notwendigen Ansichten in Abhängigkeit der Ausgabepixelposition: Jedes Ausgabe-Pixel kann Bilddaten von keiner, einer oder mehreren Ausgabe-Ansichten enthalten. In diesem Schritt muss ermittelt werden, welche Ausgabe-Ansichten tatsächlich auf dieses Ausgabe-Pixel wirken. Es entsteht eine Liste K, die alle k Ausgabe-Ansichten enthält, wobei jede Ansicht durch eine Ganzzahl definiert wird, deren Zählung bei Null beginnt. Die Liste der Ansichten ist abhängig von den Ausgabe-Pixelkoordinaten O.
• encode3(O, a, Z₀, ..., Z_k-1): Kombination der Farbwerte mehrerer Ansichten In Phase 3b) wird für jede Ausgabe-Ansicht j der Liste K ein Farbsample Z_j der der Ausgabe-Ansicht i zugeordneten Intermediate-Ansicht ermittelt. Diese Farbsamples werden mit dieser Funktion zu einem finalen Farbwert Z kombiniert. Dies ist notwendig für Ausgabeformate, die unterschiedliche Ansichten in unterschiedlichen Farbkanälen kodieren, sowie für stereoskopische oder Multiview-Ausgabebildpuffer.

The conversion of the intermediate image data into the respective output format always takes place in three steps, which are abstractly described by the mathematical functions encode1, encode2 and encode3.

• encode1 (O, a): Obtain Device Coordinates Dependent on Output Pixel Position: These device pixel coordinates serve as input coordinates of the image warping level of the geometry calibration.
• encode2 (O, a): Determining the Necessary Views Dependent on the Output Pixel Position: Each output pixel may contain image data from none, one or more output views. In this step, it must be determined which output views actually affect that output pixel. A list K is created containing all k output views, each view being defined by an integer whose count starts from zero. The list of views depends on the output pixel coordinates O.
• encode3 (O, a, Z ₀ , ..., Z _k-1 ): Combining the Color Values of Multiple Views In step 3b), for each output view j of the list K, a color sample Z _j of the intermediates associated with the output view i View determined. These color samples are combined with this function to form a final color Z value. This is necessary for output formats that encode different views in different color channels, as well as for stereoscopic or multiview output image buffers.

Output format mono: monoscopic

An output view is defined. The output frame buffer contains the output view 0 in full screen. In this picture coding is sketched by way of example. encode1 _mono (O, a) = O encode2 _mono (O, a) = {0} encode3 _mono (O, a, Z _j ) = Z _j

Output format stereo: Stereoscopic image buffer

Many workstation graphics cards support the so-called quadbuffer mode. The framebuffer contains separate image buffers for recording the images for the left and right eye (in combination with the usual double buffering, there are four buffers, hence the name). It makes sense to support such an output format directly. Since both views are always handled together in this process, we consider the color samples of a pixel for both views as a vector with twice the number of components Z = (Z _r1, Z _g1, Z _b1, Z _r2, Z _g2, Z _b2). When updating an output pixel, both views are always updated. encode1 _stereo (O, a) = O encode2 _stereo (O, a) = {0, 1} encode3 _stereo (O, a, Z ₀ , Z ₁ ) = (Z _{0, r} , Z _{0, g} , Z _{0, b} , Z _{1, r} , Z _{1, g} , Z _{1, b} )

Output format mv: Multiview image buffer

In view of future developments, it makes sense to have an output format with any number ε _a of views at which all views are available in the full output resolution. For this purpose, a general multiview output format is defined in which the output buffer ε contains _a separate image buffer. Since in this method all views are always processed together, we consider the color samples of a pixel for all ε _a views as a vector with the ε _a -fold number of components Z = (Z _r1 , Z _g1 , Z _b1 , Z _r2 , Z _g2 , Z _b2 , ...). When updating an output pixel, all views will always be updated. encode1 _mv (O, a) = O encode2 _mv (O, a) = {0, 1}

Output format ilh: Interlaced (line by line)

Two output views are defined. The output frame buffer contains line by line views 0 (even line numbers) and view 1 (odd line numbers). In this picture coding is sketched by way of example. encode1 _ilh (O, a) = O encode2 _ilh (O, a) = {O _y mod 2} encode3 _ilh (O, a, Z _j ) = Z _j

Output format ilv: interlaced

Two output views are defined. The output image buffer contains column-by-column alternately view 0 (even column numbers) and view 1 (odd column numbers). In this picture coding is sketched by way of example. encode1 _ilv (O, a) = O encode2 _ilv (O, a) = {O _x mod 2} encode3 _ilv (O, a, Z _j ) = Z _j

Output format cp: Checkerboard (checkerboard pattern)

Two output views are defined, which are spread across the output image buffer in a checkerboard pattern. If the sum of column and row number is even, the pixel will be View 0, otherwise, View 1. In this picture coding is sketched by way of example. encode1 _cp (O, a) = O encode2 _cp (O, a) = {(O _x + O _y ) mod 2} encode3 _cp (O, a, Z _j ) = Z _j

Output format sbs: Side-By-Side

Two output views are defined, which are coded as two frames (each with halved resolution) side by side in the output frame buffer. In this picture coding is sketched by way of example. encode1 _sbs (O, a) = ((2 * O _x ) mod B _a , O _y ) encode 2 _sbs (O, a) = {floor (2 * O _x / B _a )} encode _{3 sbs} (O, a, Z _j ) = Z _j

Output format tb: Top-Bottom

Two output views are defined, which are coded as two frames (each with halved resolution) among each other in the output frame buffer. In this picture coding is sketched by way of example. encode1 _tb (O, a) = (Ox, (2 · O _y ) mod H _a ) encode2 _tb (O, a) = {floor (2 · O _y / H _a )} encode3 _tb (O, a, Z _j ) = Z _j

Output format ag: anaglyph

• – μ₀: dreidimensionaler Vektor mit einem Koeffizient pro Farbkanal, beschreibt den Anteil von Ansicht 0
• – μ₁: dreidimensionaler Vektor mit einem Koeffizient pro Farbkanal, beschreibt den Anteil von Ansicht 1

Two output views are defined, which are coded in different color channels. The coding of the color channels is controlled by the following parameters:

• - μ ₀ : three-dimensional vector with one coefficient per color channel, describes the proportion of view 0
• - μ ₁ : three-dimensional vector with one coefficient per color channel, describes the proportion of view 1

In this picture coding is sketched by way of example. encode1 _ag (O, a) = O encode2 _ag (O, a) = {0, 1} encode3 _ag (O, a, Z ₀ , Z ₁ ) = μ ₀ × Z ₀ + μ ₁ × Z ₁

Output format agm: anaglyph_monochrom

• – μ_m: dreidimensionaler Vektor mit einem Koeffizient pro Farbkanal, beschreibt die Konvertierung in monochromatische Darstellung encode1_agm(O, a) = O encode2_agm(O, a) = {0, 1} h₀ = dot(Z₀, μ_m) h₁ = dot(Z₁, μ_m) Z'₀ = (h₀, h₀, h₀) Z'₁ = (h₁, h₁, h₁) encode3_agm(O, a, Z₀, Z₁) = μ₀·Z'₀ + μ₁·Z'₁

Analogous to the format "anaglyph", however, the color samples are converted by means of an additional parameter into a monochromatic representation.

• - μ _m : three-dimensional vector with one coefficient per color channel, describes the conversion into monochromatic representation encode1 _agm (O, a) = O encode2 _agm (O, a) = {0, 1} h ₀ = dot (Z ₀ , μ _m ) h ₁ = dot (Z ₁ , μ _m ) Z ' ₀ = (h ₀ , h ₀ , h ₀ ) Z ' ₁ = (h ₁ , h ₁ , h ₁ ) encode3 _agm (O, a, Z ₀ , Z ₁ ) = μ ₀ · Z' ₀ + μ ₁ · Z ' ₁

Output format sbsmv: Side-By-Side Multiview

An arbitrary number of ε _a output views are defined, which are coded as frames with in each case the image width B _a / _{e e} next to one another in the input image buffer. In this picture coding is sketched by way of example. encode1 _sbsmv (O, a) = ((ε _a * O _x ) mod B _a , O _y ) encode 2 _sbsmv (O, a) = {floor (ε _a * O _x / B _a )} encode _{3 sbsmv} (O, a , Z _j ) = Z _j

Output format tbmv: Top-Bottom

It is defined any number of ε _a output views that are encoded as frames with each image height H _a / ε _e with each other in the input frame buffer. In this picture coding is sketched by way of example. encode1 _tbmv (O, a) = (Ox, (ε _a · O _y ) mod H _a ) encode2 _tbmv (O, a) = {floor (ε _a * O _y / H _a )} encode3 _tbmv (O, a, Z _j ) = Z _j

Completion of Phase 3:

Thus, the process for an image content to be displayed (consisting of the image signals of all relevant inputs) is completed, the projectors receive as input signal the distorted according to the calibration parameters and color-matched partial images in the required for each projector monoscopic, stereoscopic or multiview format. After completing Phase 3, the process will be resumed with Phase 2 to continuously process and display the new input signals in real time.

An advantage of this method is, in particular, that the formats of the input and output views are independently selectable independently of each other and the application of the actual calibration method is combined with the conversion of the data in the output format, so that the processing cost is minimized. Furthermore, it is advantageous that the method can be executed directly on the graphics card with modern computer technology in the form of vertex and fragment shaders.

For the control of large systems consisting of very many projectors, the use of a single computer is no longer sufficient. In this case, the method can also be used in a computer cluster. It should merely be noted that, in the case of overlapping projector images, the input image data relating to this area may have to be distributed to several computers. The method presented here can then be executed individually on each computer of the cluster, wherein the image data originating from another cluster node are each regarded as a further input signal.

Exemplary embodiment: Stereoscopic powerwall with 4 stereoscopic projectors and 3 inputs

For the exemplary embodiment, it is assumed that a system consisting of 4 structurally identical stereoscopic FullHD projectors, which are arranged as a 2 × 2 array so that the projected images on the planar projection surface each overlap by approximately 20%. The projectors used are single-chip DLP projectors, which encode stereo contents encoded as checkerboard data (with effectively halved resolution per view) and display the two views alternately with a frequency of 120 Hz using the shutter method.

The image source is a graphics workstation that provides stereoscopic image content as full-screen images in 1920 × 1200 resolution via its own graphics card output. Furthermore, the output of a notebook should be displayed in a small rectangle in the upper right corner of the picture (picture-in-picture). The notebook has only one output, but it should still represent stereoscopic image content. The resolution of the notebook is 1024 × 768 pixels.

In an example structure of such a system is outlined.

Phase 1)

a)

There are inputs 0, 1, 2 and outputs 0 to 3.

• – Eingang 0: – D₀ = mono (monoskopisches Eingabe-Format) – Es wird nur eine Ansicht wie folgt zugeordnet: – Eingabe-Ansicht 0 -> Intermediate-Ansicht 0 – Als Rechteck wird das achsparallele Rechteck (0, 0), (1, 0), (1, 1), (0, 1) gewählt, damit der Eingang auf Vollbild-Darstellung skaliert wird – Zuweisung von Synchronisationsgruppe 0
• – Eingang 1: – D₁ = mono (monoskopisches Eingabe-Format) – Es wird nur eine Ansicht wie folgt zugeordnet: – Eingabe-Ansicht 0 -> Intermediate-Ansicht 1 – Als Rechteck wird das achsparallele Rechteck (0, 0), (1, 0), (1, 1), (0, 1) gewählt, damit der Eingang auf Vollbild-Darstellung skaliert wird – Zuweisung von Synchronisationsgruppe 0
• – Eingang 2: – D₂ = ilh (Eingabe-Format: Interlaced (zeilenweise) – Das interlaced-Format definiert 2 Ansichten, die wie folgt zugewiesen werden: – Eingabe-Ansicht 0 -> Intermediate-Ansicht 0 – Eingabe-Ansicht 1 -> Intermediate-Ansicht 1 – Als Rechteck wird (0.5, 0.75), (1.0, 0.75), (1.0, 1.0), (0.5, 1.0) gewählt, damit dieser Eingang in der rechten oberen Ecke erscheint. – Zuweisung von Synchronisationsgruppe 1

The inputs are configured as follows

• - Input 0: - D ₀ = mono (monoscopic input format) - Only one view is assigned as follows: - Input view 0 -> Intermediate view 0 - As a rectangle, the axis-parallel rectangle (0, 0), ( 1, 0), (1, 1), (0, 1) is selected so that the input is scaled to full screen representation - assignment of synchronization group 0
• - Input 1: - D ₁ = mono (monoscopic input format) - Only one view is assigned as follows: - Input view 0 -> Intermediate view 1 - As a rectangle, the axis-parallel rectangle (0, 0), ( 1, 0), (1, 1), (0, 1) is selected so that the input is scaled to full screen representation - assignment of synchronization group 0
• - Input 2: - D ₂ = ilh (Input format: Interlaced - The interlaced format defines 2 views, which are assigned as follows: - Input view 0 -> Intermediate view 0 - Input view 1 - > Intermediate view 1 - Select a rectangle (0.5, 0.75), (1.0, 0.75), (1.0, 1.0), (0.5, 1.0) so that this input appears in the upper right corner - Assigning synchronization group 1

• – Ausgang a: – Ermittlung der Kalibrierparameter β_a. In diesem Beispiel wird ein kamerabasiertes Kalibrierverfahren verwendet, welches an Hand bekannter projizierter Bildmuster die Position und geometrische Verzerrung ermittelt. Weiterhin berechnet das Kalibrierverfahren einen Blendfaktor für jedes Pixel des Projektors, der für den Helligkeitsabgleich in den Überlappungsbereichen eigesetzt wird. Derartige Verfahren sind im Abschnitt „Stand der Technik” erläutert. – G_a = cb (Ausgabe-Format: Checkerboard) – Das Checkerboard-Ausgabeformat definiert 2 Ansichten, denen wie folgt Intermediate-Ansichten zugewiesen werden: W_a = {0, 1} – Ausgabe-Ansicht 0 -> Intermediate-Ansicht 0 – Ausgabe-Ansicht 1 -> Intermediate-Ansicht 1 – B_a = 1920, H_a = 1080 (FullHD)

In this example, the identical projectors are all to be operated in the same settings. Therefore the configuration is identical for each output a of the A outputs:

• - Output a: - Determination of the calibration parameters β _a . In this example, a camera-based calibration method is used which determines the position and geometric distortion based on known projected image patterns. Furthermore, the calibration method calculates a glare factor for each pixel of the projector that is used for the brightness adjustment in the overlap areas. Such methods are explained in the section "Prior Art". - G _a = cb (Output Format: Checkerboard) - The checkerboard output format defines 2 views to which intermediate views are assigned as follows: W _a = {0, 1} - Output View 0 -> Intermediate View 0 - Output View 1 -> Intermediate View 1 - B _a = 1920, H _a = 1080 (FullHD)

By way of example, 3840 × 2160 pixels are selected as the resolution for the intermediate buffer, because this corresponds to the overall resolution of the projector array (without consideration of the overlap). As a ring buffer size, 3 is used as an example.

b)

The enumeration of all intermediate views of all outputs with the removal of the duplicates yields L = {0, 1}. Thus, 2 intermediate views are used so that 2 intermediate frame buffers, each with a resolution of 3840 × 2160 pixels, are created

c)

• – M₀ = M₁ = 1920, N₀ = N₁ = 1200
• – M₂ = 1024, N₂ = 768

For each input e, a ring buffer with Q = 3 image buffers and an additional input image buffer are created. The resolutions of the image buffers are constant per input and result from the recorded image signals as follows:

• - M ₀ = M ₁ = 1920, N ₀ = N ₁ = 1200
• - M ₂ = 1024, N ₂ = 768

Furthermore, e R _{e is} initialized to 0 for all inputs.

d)

An output image buffer of 1920 × 1080 pixels is applied for each of the 4 outputs.

Phase 2)

a)

First, at the entrance e '= 1, it enters a new image. Since R ₁ = 0 <Q, the input image is copied to the ring buffer of input 1. The synchronization group to which e 'belongs is S = {0, 1}. Since the condition R ₀ > 0 does not apply, the process is continued again at phase 2a and waits for further image data.

It now enters a new image at the input e '= 0. Since R ₀ = 0 <Q, the input image is copied to the ring buffer of input 0. The synchronization group to which e 'belongs is S = {0, 1}. Since R _e > 0 for all e from S, the process continues in phase 2b).

b)

For each input e from S, the oldest unprocessed image content of the circular buffer is copied into the input frame buffer and R _e is decremented by 1 so that again R ₀ = R ₁ = 0. The oldest image contents in this case are the first (and only) image data copied to the ring buffer. It is now carried out for each input e of the inputs Phase 2c).

c) e = 0

Since v ₀ = 1, there is an input view j = 0. From the configuration (phase 1a) view 0 can be assigned the intermediate view i = 0. This is contained in list L, so the rasterization is performed: for each pixel p _i , the intermediate image buffer for view i, which lies in the rectangle of input 0 in the normalized intermediate coordinate system, phase 2d) is executed.

d) e = 0, pixel p _i

For the pixel, the corresponding input coordinates are calculated. This is done via a linear mapping which assigns normalized input pixel coordinates to the normalized intermediate coordinates, such that the given vertices of the rectangle in the intermediate coordinate system correspond to the vertices of the input image (0, 0), (1, 0), (1 , 1), (1, 0). This can be achieved mathematically with the affine mappings scaling, mirroring and translation. In the example, the unit transformation results for input 0.

Here, by way of example, the procedure for the pixel p _i with the normalized intermediate coordinates (0.5, 0.5), ie the center of the image, is selected. The result is C = (0.5, 0.5). D ₀ = mono, so the decode functions of the monoscopic image format must be used, decode1 _mono describes ordinary bilinear filtering in the 2 × 2 neighborhood of C. Thus F results in the bilinear filtered image content of the input frame buffer 0 to the Coordinates C, decode2 _mono describes a unit map such that F '= F. Finally, the color value of the pixel p _{i of} the intermediate image buffer of the view i = 0 is set to F '.

Phase 2d) is performed for each additional one of the 3840 × 2160 pixels of the rectangle in the intermediate frame buffer, resulting in a (distorted) scaled, bilinear filtered copy of the input image of input 0 in the intermediate image buffer of view 0. If no more intermediate views are assigned, phase 2c) is also connected for input 0, followed by the execution of phase 2c) and d) for input e = 1.

c) e = 1

The execution differs from the already described execution for e = 0 only in that now in the intermediate view i = 1 is rasterized. The intermediate image buffers now contain the complete stereoscopic signal of the workstation, but not yet the picture-in-picture portion of the notebook. This is followed by the execution of phase 2c) and d) for input e = 2.

c) e = 2

Since v ₂ = 2, there are two input views, 0 and 1. Therefore, the view j = 0 is processed first. From the configuration (phase 1a) input view 0 can be assigned the intermediate view i = 0. This is included in the list L, so the rasterization is done: for each pixel p _i , of the intermediate image buffer for view i, which is in the rectangle of input 0 in the normalized intermediate coordinate system, phase 2d) is executed.

d) e = 2, j = 0, i = 0, pixel p _i

The rectangle (0.5, 0.75), (1.0, 0.75), (1.0, 1.0), (0.5, 1.0) is applied to the normalized input coordinates (0, 0), (1, 0), (1, 1), (1 , 0). This figure can be realized by a factor of 2 by a horizontal scaling and by a factor of 4 by a vertical scaling and a subsequent shift by (-1, -3).

Here, the procedure for the pixel p _i with the normalized intermediate coordinates (0.75, 0.875) (center of the rectangle) is chosen by way of example. The result is C = (0.5, 0.5). D ₂ = ilh, so the decode functions of the line by line interlace image format must be applied: C ' _x = nearest (0.5, 1024) = 511.5 / 1024 f _x = factor (0.5, 1024) = 0.5 h _y = (0.5 · 768 + 0 - 0.5) / 2 = 191.75 C' _y = (floor (191.75) · 2 - 0 + 0.5) / 768 = 382.5 / 768 f _y = (0.5 - 382.5 / 768) / (2 · (1/768)) = 0.75

For better traceability, the pixel coordinates x 'corresponding to the normalized coordinates x are indicated here:
x ' ₁ = (511, 382), x' ₂ = (512, 382), x ' ₃ = (511, 384), x' ₄ = (512, 384).

Due to the input view j = 0, only pixels in even-numbered rows were selected. The function decode1 _ilh thus provides the bilinear interpolation of the color values of these pixels with the interpolation factors f = (0.5, 0.75), and decode2 _ilh does not modify the color value.

Phase 2d) is performed for each additional pixel of the rectangle in the intermediate frame buffer. The decoding function only samples the even-numbered lines of input 2 input buffer. Since no more intermediate views are assigned to the input view, phase 2c) continues with input view j = 1 for input 2.

c) e = 2 (continued)

The input view j = 1 is assigned the intermediate view i = 1, which is also contained in L. Thus, the rectangle is rasterized into the view i, and phase 2d) is executed again per pixel.

d) e = 2, j = 1, i = 1, pixel p _i

Due to the changed parameter j, the results differ from the execution of phase 2d) in the previous view. For the example pixel p _i at the coordinates C = (0.5, 0.5) we now have: C ' _x = nearest (0.5, 1024) = 511.5 / 1024 f _x = factor (0.5, 1024) = 0.5 h _y = (0.5 · 768 + 1 - 0.5) / 2 = 192.25 C' _y = (floor (192.25) · 2 - 1 + 0.5) / 768 = 383.5 / 768 f _y = (0.5 - 383.5 / 768) / (2 · (1/768)) = 0.25 x ' ₁ = (511, 383), x' ₂ = ( 512, 383), x ' ₃ = (511, 385), x' ₄ = (512, 385).

Due to the input view j = 1, only pixels in odd-numbered lines were selected. The function decode1 _ilh thus provides the bilinear interpolation of the color values of these pixels with the interpolation factors f = (0.5, 0.25), and decode2 _ilh does not modify the color value.

Phase 2d) is performed for each additional pixel of the rectangle in the intermediate frame buffer. The decoding function only samples the odd lines of Input 2 input buffer. Since no more intermediate views have been assigned to the input view and all v ₂ = 2 input views have been processed, the execution for input e = 2 is completed.

The intermediate image buffers of each view now contain the input image of inputs 0 and 1 scaled to buffer size, and the input image of input 2 as picture-in-picture in the upper right corner, whereas in intermediate view 0 only pixels with even line number and In Intermediate view 1, only pixels with odd row number of input 2 have been processed, i. H. the stereo signal was decoded. For the illustration of the method it does not matter that the input frame buffer of input 2 still contains the initial values at this time, since no input picture was present at this input. It is obvious that the procedure just described leads to the desired result if it is ensured that the input buffer contains the correct image information. Since input 2 forms its own synchronization group, this results in processes in phase 2a) and 2b) that with the availability of new image data at input 2 and the associated input image buffer is updated and in phase 2c) and 2d), the actual image content be further processed.

Since there are no more inputs, phase 3) is now performed.

Phase 3)

a)

Phase 3a) describes the iteration over all A outputs. Therefore, let a = 0 initially. For each pixel p _{a of} the output image buffer of output a, the method described in phase 3b) is carried out.

b) a = 0, p _a

By way of example, the method is to be carried out here for a pixel close to the lower left margin with the coordinates O = (10, 0). Since G _a = cb was chosen in the configuration, the encode _cb functions are now used for output encoding. This format defines the device pixel coordinates as identical to the output pixel coordinates, ie: T = encode1 _cb (0, 0) = (10, 0).

Now these device coordinates are mapped into the intermediate coordinate system. This figure provides the calibration procedure. By way of example, the calibration method U = warp _a (T, β _a ) = (0.01, 0.42), ie the lower left corner of Projector 0 is located slightly below the center at the left edge of the image area. Furthermore, it is assumed that the calibration method soft-edge blending is realized via a position-dependent glare factor. The parameter vector α contains this glare factor. Since a smooth transition is to be realized in soft edge blending, and the current output pixel is located in the overlap area at the edge of the projector, here α = param _a (T, β _a ) = 0.01 is assumed.

First, all intermediate views are determined that must be encoded in this output pixel. K = encode2 _cb (O, 0) = {0}. Using list W ₀ , the intermediate view can now be assigned to each output view j.

In checkerboard coding, one view is coded per pixel, for the output pixel at O = (10, 0), j = 0 and, according to W _0, also i = 0. The intermediate image buffer for view i = 0 at the coordinates U = (0.01, 0.42) sampled with bilinear filter. The resulting RGB color vector is stored as Z ₀ . Since all views have been processed, the execution of phase 3c follows)

c) a = 0, p _a for O = (10, 0) and K = {0}

The color value Z results from encode3cb (O, 0, Z ₀ ) = Z ₀ , ie the color value is adopted unchanged. Finally, the color value is adapted by the calibration method, which in this step performs the adjustment of brightness, blending and the color space-specific conversions on the basis of the previously determined calibration parameters: Z '= color _a (Z, α, β _a ). Z 'is thus the color value that has to be sent to the projector a so that the desired color value Z or, in overlapping areas, the proportion of the projector a at the desired color value Z is produced on the projection surface. Therefore, the output pixel p _{a is set} to the color value Z '.

Pixel p _a is updated and in step 3b) the next pixel is transitioned. At the coordinates O = (11, 0) encode2 _cb (O, 0) = {1}, so here the intermediate image buffer for view i = 1, which is assigned to the output view j = 1, is sampled.

After iterating over all pixels of the output image buffer of the output a = 0, the desired checkerboard coding of the intermediate views has been created.

The process is therefore continued in phase 3a) with the next output a = 1. In this case, this process differs from output 0 only in that the calibration functions supply other parameters and thus the image detail suitable for the respective projector is generated. Further explanations for the outputs 1 to 3 are therefore omitted here.

Upon completion of Phase 3, all projectors will display the stereoscopic image content. The process will begin again with Phase 2a), so that new image data will be processed continuously. illustrates the data flow during the procedure.

bibliography

• [1] R. Y. Yang, D. Gotz, J. Hensley, H. Towles, and M.S. Brown, "PixelFlex: a reconfigurable multi-projector display system," in Proceedings of the Conference on Visualization, 01, San Diego, California, 2001.
• [2] H. Chen, S. Rahul, G. Wallace, and K. Li, "Scalable alignment of large-format multi-projector displays using camera homography trees." In Proceedings of the conference on Visualization '02 (VIS '02 ), Washington, DC, USA, 2002.
• [3] A. Majumder and R. Stevens, "Color nonuniformity in projection-based displays: analysis and solutions," in IEEE Transactions on Visualization and Computer Graphics, 2004.
• [4] R. Raskar, M.S. Brown, R. Yang, W.-C. Chen, G. Welch, H. Towles, B. Seales and H. Fuchs, "Multi-projector displays using camera-based registration," in Proceedings of the conference on Visualization '99: celebrating ten years (VIS '99), lot Alamitos, CA, USA, 1999.
• [5] CO Jaynes and SB Webb, "Multi-Projector Intensity Blending System". patent US2007188719 , 2007.
• [6] AW Divelbiss and WV Tserkovnyuk, "General Purpose Stereoscopic 3D Format Conversion System and Mehod". US patent US20040218269 , 2004.
• [7] MJ Johnson, R. Chandrarasekhar and C.-J. Boss, "Super-Resolution Display". patent US6456339 / WO0007376 , 2001.
• [8] M. Schiewe and S. Klose, "Calibration Method and Calibration System". patent DE10 2006 002 602 / US2009067749 / WO2007082690 / EP1972134 , 2006.
• [9] RJ Surati and KTF Jr., "Method and Apparatus for Calibrating a Tiled Display." patent EP1116386 / WO0018139 / DE69917470 / US6525772 / US6310650 ,
• [10] TA Barrett and B. Crosby, "Method and apparatus for providing the display of video content". patent WO2011084169 , 2011.

Claims (10)

A method of rendering monoscopic, stereoscopic or multiview image signals that processes image data from one or more inputs for output at one or more outputs such that image modifications for color and geometry calibration in the form of calibration parameters for each output are applied to the input data in real time be used so that the calculated one or more output signals for driving stereoscopic or multiview-capable multi-segment projection systems, using a separate partial steps calibration method is used and wherein each input signal each consists of one or more views in different formats to a The resulting output signals also contain one or more views, which in turn are linked in a particular, independent of the format of the inputs, format to an overall image and the generation the output signals in this case by decoding the input signals to separate the views and the combination of the application of the calibration parameters with the coding of the views in the required output format such that only the actually affecting the output signal input pixels must pass through all calibration steps and the calculation of the geometric distortion and determination of pixel specific position-dependent but color-independent calibration parameter is performed only once for an output pixel position, even if multiple views affect the output signal at this position. A method according to claim 1, characterized in that the calibration method comprises the separate sub-steps for "geometrical adaptation", "calculation of position-dependent but color-independent parameters" and "adjustment of colors", and a method for encoding of multiple views into an overall image in separate sub-steps for "determining the device coordinates depending on the output pixel position", "determining the necessary views depending on the output pixel position" as well as "combining the color values of several views", so that the coding steps "determination of the device coordinates", "determination of the views" and the calibration steps "Geometric Fit" and "calculate position dependent but color independent parameters" for a given output pixel position will only be executed once, even if the data of several views is processed for that output pixel position n, and even for the actual output signals at this point influencing views at all the calibration and format encoding is performed. A method according to claim 2, characterized in that the decoding of the input image data into several views is broken down into separate sub-steps for "sampling the color value as a function of pixel position and view" and "modifying the color value as a function of the view", such that a multiplicity of different Image formats for the input data as well as the output data as well as different methods for applying calibration parameters to image contents are supported by defining the individual sub-steps mentioned in claims 2 and 3 for the respective image format or calibration method, so that each input is independent of the other inputs Decoding and each output independently of the other outputs a calibration and coding can be assigned. A device for driving stereoscopic or multiview capable multisegment projection equipment that applies color and geometry calibration techniques in real time to one or more input signals using calibration parameters provided for each output, and outputs to one or more outputs using a multi-part calibration method and wherein each input signal consists of one or more views which have been combined in different formats into one overall image, and the resulting output signals also contain one or more views, which in turn combine into an overall image in a specific format independent of the format of the inputs and the generation of the output signals by decoding the input signals to separate the views and the combination of the application of the calibration parameters with the coding of the views in the required output format d erart takes place so that only the input pixels actually influencing the output signal have to pass through all the calibration steps and the calculation of the geometric distortion and determination of pixel-specific position-dependent but color-independent calibration parameters is performed only once for an output pixel position, even if several views influence the output signal at this position. An apparatus according to claim 4, characterized in that one or more computers with one or more graphics cards each having one or more outputs is used and the decoding of the input signals for separating the views and the coding of the output signal and the color adjustment of the calibration by means of vertex and fragmentation shades is implemented. An apparatus according to claim 5, characterized in that one or more video grabbers are used, each with one or more inputs for receiving video signals. A device according to one of claims 5 or 6, characterized in that a plurality of computer systems are connected in a network and the input image data or parts of the input image data are exchanged between the computer systems in the network. A device according to one of claims 5 to 7, characterized in that a framebuffer consisting of several image buffers, one per view, are used as the output image buffer, and in the fragment shader the color value for each of the views is calculated and written into the respective image buffer. An apparatus according to claim 8, characterized in that the output image buffer is a stereoscopic ("quad buffered") frame buffer, each with a color buffer for the left and the right eye is used. A device according to any one of claims 4 to 9, characterized in that a plurality of sets of configuration data and calibration parameters can be stored and switched at any time between them, so that the configuration of the multi-segment projection system can be adapted dynamically to changes.

Images