JP2005004201A - Method and system for projecting image onto display surface - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method for projecting one or more images onto a curved display surface. <P>SOLUTION: First, a predetermined structured light pattern is projected onto the display surface. A stereo pair of images is acquired of the projected images on the display surface. Then, a quadric transfer function between the predetermined images and the stereo pair of images, via the display surface, is determined. Thus, an arbitrary output image can be warped according to the quadric transfer function so that when it is projected onto the display surface it appears correct. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates generally to image projection, and more particularly to projection of an image onto a curved surface.

  Projector systems have been used to render large images on a display surface. If a plurality of projectors are used, a larger and seamless display can be generated. Such a system is particularly useful in building an immersive visualization environment capable of presenting high resolution images for entertainment, education, training, and scientific simulation. Known multi-projector technologies include "Surround-screen Projection-based Virtual Reality: The Design and Implementation of the CAVE" SIGGRAPH 93 Conference Proceedings, Vol. 27, pp. 135-142, 1993, Staadt et al. "The blue-c: Integrating real humans into a networked immersive environment" ACM Collaborative Virtual Environments, 2000.

And positioned using an electro-optical techniques, finding a blending parameters (Li et al., "Optical Blending for Multi-Projector Display Wall System " ^{Proceedings of the 12 th Lasers and Electro} -Optics Society, see 1999), or the camera in the loop (Surati “Scalable Self-Calibrating Display Technology for Seamless Large-Scale Displays” Massachusetts Institute of Technology Doctoral Dissertation, 1999, Chen et al. “Automatic Alignment of High-Resolution Multi-Projector Displays Using An Un-Calibrated Camera” IEEE Visualization, 2000, and Yang et al. `` PixelFlex: A Reconfigurable Multi-Projector Display System '' IEEE Visualization, 2001, Brown et al. `` A Practical and Flexible Large Format Display system '' Tenth Pacific Conference on Computer Graphics and Applications, pp. 178- 183, 2002, Humphreys et al., "A Distributed Graphics System for Large Tiled Displays" IEEE Visualization, 1999, and Humphreys et al., "WireGL: A Scalable Graphics. System for Clusters "Proceedings of SIGGRAPH, 2001), several techniques are known for generating seamless images on flat surfaces.

  When multiple projectors are used, it is important to accurately estimate the geometric relationship of overlapping images in order to achieve seamless display. Geometric relationships affect the rendering process and soft edge blending. A camera-based method that utilizes homography represented by a 3 × 3 matrix allows an inadvertently installed projector while eliminating tedious manual alignment.

  A surface relationship according to a quadratic equation can be defined using a secondary image transfer function. See Shashua et al., “The quadric reference surface: Theory and applications” Tech. Rep. AIM-1448, 1994.

  Multi-projector alignment of curved surfaces can be aided by projecting a “navigator” pattern and then manually adjusting the position of the projector. For large displays, such as those used in the New York Planetarium, it takes several hours every day for technicians to align seven overlapping projectors.

  One problem is that when a 3D image is displayed on a curved surface, the image is perspectively correct only at one point in space. This 3D position is known as a virtual viewpoint, or “sweet spot”. As the viewer moves away from the sweet spot, the image appears distorted. If the display screen is very large and there are many viewpoints, it is difficult to eliminate this distortion. However, for real-world applications, the viewer wants to be in exactly the same place where the projector should ideally be placed. Furthermore, placing the projector in a sweet spot means using a projector with a very wide field of view, which is expensive and tends to be excessive in radial or “fish-eye” distortion. .

  In another method, the camera is placed at the sweet spot by a non-parametric process. The camera acquires an image of the structured light pattern projected by the projector. Next, samples are taken in a trial and error manner to construct an inverse warping function between the camera input image and the projected output image by interpolation. This function is then applied and resampled until the warping function correctly displays the output image. Jarvis "Real Time 60Hz Distortion Correction on a Silicon Graphics IG" Real Time Graphics 5, pp.6-7, February 1997, and Raskar et al. "Seamless Projection Overlaps Using Image Warping and Intensity Blending" Fourth International Conference on Virtual Systems and See Multimedia, 1998.

  It would be desirable to provide a parametric method of aligning multiple projectors that extends a homography-based approach for flat surfaces to a quadratic surface.

  In computer vision, some research has been done on the use of quadratic equations for image transfer functions. See Shashua et al. And Cross et al., “Quadric Surface Reconstruction from Dual-Space Geometry” Proceedings of 6th International Conference on Computer Vision, pp. 25-31, 1998. However, a linear method intended for a camera causes a large error when used in conjunction with a projector instead of a camera, as described below.

  In multi-projector systems, several techniques are known for seamlessly aligning images on a flat surface using flat homography relationships. However, little research has been done on techniques for parameterized warping and automatic registration of images displayed on higher order surfaces.

  This is a serious nuisance because quadratic surfaces appear in many shapes and forms on projector-based displays. Large flight simulators have traditionally been cylindrical or dome shaped (see Scott et al. “Report of the IPS Technical Committee: Full-Dome Video Systems” The Planetarian, Vol. 28, p.25-33, 1999) Planetarium and OmniMax theater use hemispherical screens (Alplane's “Planetarium special effects: A classification of projection apparatus”, The Planetarian, Vol. 23, pp.12-14, 1994), and many virtual reality systems A cylindrical screen is used.

  Therefore, it would be desirable to provide a method for calibrating an image projected on a curved display surface, a second order transfer function, and parametric luminance blending.

  Curved display screens are increasingly being used in high resolution immersive visualization environments. The present invention provides a method and system for displaying a seamless image on a quadratic surface, such as a spherical or cylindrical surface, using one or more overlapping projectors. A new secondary image transfer function is defined that achieves subpixel alignment while interacting and displaying 2D or 3D images.

  FIGS. 3 and 4 illustrate the present invention using one or more projectors 301 and 401 to display a seamless image on a convex or concave curved surface, such as concave dome 302 or convex dome 402. Shows the basic setup of the system. When one projector is used, the projected image covers most of the display surface (see dotted lines), and when multiple projectors are used, the projected images partially overlap as shown by the dashed lines. FIG. 7 shows how a larger rectangular image 710 can be created with three overlapping images 701.

Secondary transfer function The mapping between two arbitrary perspectives of an opaque quadric surface Q in 3D can be expressed using a secondary transfer function Ψ. The quadric surface can be a sphere, hemisphere, spheroid, dome, column, cone, paraboloid, hyperboloid, hyperbolic paraboloid, or ellipsoid. A quadratic transfer function according to the present invention refers to an image transfer function from a first view, eg a projector output image, to a second view, eg a camera input image, via a quadric surface. The flat homography transfer function can be obtained from the correspondence of four or more pixels, but the secondary transfer function requires nine or more correspondences. The quadratic transfer function can be defined in a closed form using a 3D quadratic surface Q and additional parameters related to the perspective projection of the two views.

The quadric surface Q is represented by a 4 × 1 vector, and the 3D homogeneous point X on the surface is represented by a 4 × 4 symmetric matrix so that the quadratic constraint X ^T QX = 0 is satisfied. The quadric surface Q has 9 degrees of freedom corresponding to the independent elements of the matrix. This matrix is symmetric and is defined up to the entire scale.

  Corresponding pixels: the homogeneous coordinates of x in the first view and x 'in the second view are

  Related by.

  As shown in FIG. 6, B is between a projected output pixel x601 and a corresponding camera input pixel x′602 via a point 603 on a plane 604 tangent to a quadric surface 605 where the pixel 601 is visible. 3 × 3 homography matrix.

  Given the pixel correspondence (x, x ′), conventionally using this equation there are 21 unknowns: unknown 3D quadratic surface Q, 3 × 3 homography matrix B, and epipole e in homogeneous coordinates e Is calculated. Epipole e is the projection center of the first view in the second view. The symbol ≡ represents equality to the homogeneous coordinate scale. The matrix Q is:

  It is disassembled as follows.

Thus, Q ₃₃ is the top 3 × 3 symmetric submatrix of Q and q is 3 vectors. Q (4,4) is not zero when the quadric surface does not pass through the origin, ie, the projection center of the first view. Therefore, 1.0 can be safely assigned to most display surfaces. The final 2D pixel coordinates of the homogeneous pixel x ′ are (x ′ (1) / x ′ (3), x ′ (2) / x ′ (3)).

Simplification The form described above is used by Shashua et al., 1994 and “Q-Warping: Direct Computation of Quadratic Reference Surface” IEEE Conf. On Computer Vision and Pattern Recognition, CVPR, June, 1999 by Wexler and Shashua. Yes. This form contains 21 variables, which is 4 more than needed. In the present invention, by defining A = B−eq ^T and E = qq ^T −Q ₃₃ , part of the ambiguity is removed, and a form used in the present invention is obtained.

Here, x ^T Ex = 0 defines the conic curve of the quadric surface in the first view. The contour cone curve can be visualized geometrically as an image of a silhouette or point on the surface where the view ray touches the surface locally, for example an image of an elliptical silhouette of a sphere viewed from outside the sphere.

  The value A is a homography through the polar plane between the first view and the second view. Note that this equation contains only one ambiguous degree of freedom due to the relative scaling of E and e, apart from the overall scale. This ambiguity can be removed by introducing additional normalization constraints, such as E (3,3) = 1.

  Furthermore, the sign in front of the square root is fixed within the contour cone curve in the image. This sign is easily determined by testing the above equation using the coordinates of the corresponding pixel pair. Note that the parameters of the second order transfer function can be calculated directly from the correspondence of 9 or more pixels in the projected coordinate system. For this reason, without calculating any Euclidean parameter, it would be desirable to follow the same method as estimating flat homography in the case of a flat display. However, as described below, in practice, it is difficult to estimate the epipole relationship in many cases. Therefore, in the present invention, the pseudo Euclidean method is used as described below.

Pre-processing All registration information is pre-calculated for a stereoscopic image pair. Here, it is assumed that the stereoscopic camera is viewing the entire 3D display surface. One of the camera images is arbitrarily selected as having an origin c _o . The stereoscopic image is used to obtain only 3D points on the display surface. Thus, any suitable 3D acquisition system can be used.

  As shown in FIG. 1, the basic steps of the pre-processing method 100 of the present invention are as follows. Details of these steps are described below. For each projector i, a predetermined image 101, for example a structured pattern in the form of a checkerboard, is projected onto a quadric surface (110).

  In step 120, the image features of the predetermined image 101 obtained by the stereoscopic camera 102 are detected, and the correspondence 103 on the quadric surface corresponding to the 3D points, that is, the features of the predetermined image is reconstructed. Next, the quadric surface Q104 is approximated to the detected correspondence (130).

For each projector i, a projector pose 105 for the camera is determined 140 using the correspondence between the projector pixels and the 3D coordinates of the points on the surface illuminated by the pixels. A secondary transfer function ψ _i and an inverse secondary transfer function ψ ^I _i between the camera c ₀ and the projector i are obtained (150). Next, a luminance blending weight Φ _i in an area where the projected images overlap is obtained (160). At this point, a projector image that appears as an undistorted output image 171 can be projected onto a quadric surface by warping, blending, and projection (200).

Runtime Processing FIG. 2 shows the basic steps of the rendering method 200 of the present invention. First, a projection image is generated (210). The projector image is represented as a triangular texture mapping set. This image can be acquired by a stereoscopic camera or a video camera, or can be generated by computer graphic rendering a 3D scene from a virtual viewpoint. The projector image is warped into the projector frame according to the secondary transfer function Ψ _0i (220). Next, the pixel intensity is attenuated by blending weight Φ _i (230) and then projected (240).

  However, these steps include several problems that need to be solved. Second-order transfer function estimation is a linear operation but requires non-linear optimization to reduce pixel reprojection errors. Furthermore, 3D points projected on a quadric surface are usually substantially flat and lead to degradation conditions, so it is difficult to estimate the projector pose, ie, external parameters. These and other problems, as well as actual solutions are described below.

Calibration
In the present invention, the parameters of the second order transfer function Ψ _oi = {A _i , E _i , e _i } are obtained so that the projected output image is geometrically aligned on the curved surface. Prior art methods known for cameras find secondary transfer parameters directly from pixel correspondence. This involves estimating a 4 × 4 quadratic matrix Q in 3D using a triangular decomposition of the corresponding pixels and a linear method. If the internal parameters of the two views are not known, all calculations are performed in the projection space after calculating the epipole geometry, ie the epipole and the base matrix.

  However, if a projector is involved rather than a camera, the linear method results in a very large reprojection error in estimating the 3D quadric surface Q. This error is on the order of about 30 pixels for a conventional XGA projector.

  There are several reasons for this. It is relatively simple to calibrate the camera directly from the acquired images. However, the calibration of the projector can only be performed indirectly by analyzing the image of the projection image obtained by the camera. This can lead to errors. Further, in projectors, the internal optical system is usually offset from the principle point at which the image is projected upward. Therefore, the internal parameters of the projector are difficult to estimate and are different from those of the camera. Furthermore, if the 3D point on the quadric surface illuminated by a single projector does not have very deep variations in display settings, such as a spherical or column segment, the fundamental matrix will inherently contain noise. Many.

  Thus, the present invention uses a pseudo-Euclidean approach in which the camera and projector internal and external parameters are roughly known. These parameters are used for strict Euclidean transformation estimation. Therefore, unlike a flat case, a three-dimensional quantity is involved in calculating an accurate image transfer function in the case of a curved screen.

Quadratic surface In the present invention, a precise stereo camera pair C ₀ and C ′ ₀ is used as a base for calculating the relationship between all geometric shapes. Select one of the cameras arbitrarily to define the origin and coordinate system. A checkerboard pattern 101 is used to calibrate the base stereo camera pair. In the calibration of the present invention, the camera does not necessarily have to be placed at the sweet spot, which is an important difference to some of the prior art non-parametric approaches.

The stereoscopic camera pair 102 observes the structured pattern projected by each projector. Using the triangulation, N 3D point sets {X _j } on the display surface corresponding to the pattern features are detected. A quadric surface Q passing through each X _j is calculated by solving a linear set X ^T jQX _j = 0 for each 3D point. This equation can be written in the form x _i V = 0, where x _i is a 1 × 10 matrix that is a function of X only, and V contains a separate independent unknown variable of quadratic surface Q It is a homogeneous vector. Using N ≧ 9, construct an N × 10 matrix X and solve the linear determinant XV = 0.

  In the case of a point at a general position, the elements of V and Q are a one-dimensional null space of the matrix X.

Projector View In addition to the quadric surface Q, the present invention needs to know the internal and external parameters of each projector relative to the origin of the camera. In the present invention, the pose and internal parameters are calculated using the correspondence between the projector pixel and the coordinates of the 3D point illuminated by the projector pixel.

  However, finding the projector pose from known 3D points on the quadratic surface is prone to errors because the 3D points are usually much closer to the plane and the solution becomes unstable. See Faugeras, Three-dimensional computer vision: a geometric viewpoint, MIT Press, 1993, and Forsyth et al. "Computer Vision, A Modern Approach" FUTURESLAB, 2002, ActiveMural, Argonne National Labs, 2002.

  Dealing with a nearly flat point is a difficult problem. If the depth of the points is dispersed, a linear method can be used to easily estimate the projector's internal as well as external parameters. On the other hand, if the point is known to be flat, the external parameters can be estimated when some of the internal parameters are known.

  In order to deal with substantially flat surfaces, the present invention uses an iterative procedure. If you know the projector's internal parameters, first find an initial estimate of the external parameters based on homography, then Lu et al. `` Fast and globally convergent pose estimation from video images '' IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol 22: 6, pp.610-622, 2000, an iterative procedure can be used. Powell's method can be used for non-linear improvement of reprojection error. However, it is equally difficult to estimate the internal parameters of the projector. As mentioned above, if the projector cannot be easily moved, calibration of the projector typically requires a large surface that is illuminated in more than one position.

  The strategy of the present invention is to use the projector's internal parameters that are roughly known. In the present invention, internal parameters of only one projector are found and these internal parameters are used for all projectors. Later, the same and other projectors will likely have different zoom settings and other mechanical or optical deviations. Furthermore, the external parameters calculated by Lu et al. Are only approximations.

Therefore, the present invention uses the camera's perspective projection parameters along with the projector's approximate projection matrix to find a secondary transmission from the camera to the projector using a linear method. Next, the solution is improved using nonlinear optimization.

The quadratic transfer parameter Ψ _oi = {A _i , E _i , e _i } is obtained from the quadric surface Q, the camera projection matrix [I | 0], and the projector projection matrix [P _i | e _i ].

  Easily calculated.

  As mentioned above, the prior art linear method of finding parameters is too inaccurate for this purpose. The misalignment on the display screen may be about 15 to 30 pixels per projector, resulting in an unpleasant visual artifact. Seamless display requires sub-pixel accuracy. Thus, the present invention applies nonlinear minimization to improve the results obtained through linear methods. This can be done using an objective function or a “cost” function.

  For the objective function, take the total secondary transfer error for all pixels.

  Where

Is the pixel after transmission of each known projection pixel x _j , ie

  It is a pattern feature point expressed as

  Note that the code found using the linear method remains the same for all pixels and remains the same during non-linear optimization, eg, the known Nelder-Mead simplex.

Partial Euclidean Reconstruction Ignore all Euclidean methods and proceed directly to projection space and nonlinear optimization. If you have accurate projector internal parameters, you may avoid the nonlinear optimization step. However, as mentioned earlier, ignoring the Euclidean view parameters and solving the secondary transfer function purely from pixel correspondence results in poor reprojection errors. Furthermore, since the estimated 3D quadric surface Q does not converge, it cannot be used for estimation of subsequent nonlinear optimization.

  Accurate internal projector parameters only reduce reprojection errors and do not eliminate errors. This includes many types of errors in 3D Euclidean calculations, including estimating 3D points on the display surface by triangulation, estimating 3D quadric surfaces using linear methods, and finding projector poses. This is to propagate. Non-linear optimization attempts to minimize the physical quantity to which the most attention is paid, i.e., pixel reprojection errors in the image transmitted from the camera of a known corresponding pixel set to the projector.

  Since correspondence between overlapping projector pixels is indirectly defined by this image transfer function, minimizing pixel reprojection errors ensures a geometric alignment between displayed projector pixels. Is done.

Rendering Rendering involves a two-pass approach. In the case of 2D image data, an appropriate input image is extracted. In the case of a 3D scene, first, a 3D model is rendered according to a sweet spot. In the second pass, the resulting image is then warped into the projector image space using a secondary image transfer function.

Virtual View When a 3D scene is displayed on a curved screen, the image is only perspective correct from a specific point in space. Such points are generally known as sweet spots or virtual viewpoints. In the case of a concave hemisphere, the sweet spot is on a line through the center of the sphere perpendicular to the cut plane that cuts the sphere in half. In the case of a convex hemisphere, the sweet spot can be mirrored through the surface. In the case of a pillar surface, the sweet spot is arranged in the same manner as long as the projection image is not covered on the entire surface, and when the projection image is covered on the entire surface, the sweet spot may be located anywhere. The sweet spot is independent of the focal length of the camera, but depends on the shape and size of the curved display and the surrounding viewing angle. The projected image can appear in front of or behind the display surface depending on the location of the sweet spot, the focal length of the visual camera, or the viewer. As the viewer leaves the sweet spot, the image appears distorted. In addition, it is necessary to specify the viewing frustum, ie the viewing direction or principal axis, and the field of view.

  Depending on the display, it is possible to automatically determine the sweet spot. For example, in the case of a concave hemispherical dome, the center of the dome can be regarded as a good sweet spot. The sweet spot can be directly obtained from the equation of the quadric surface Q, that is, Q (1,1) q. In the case of a cylindrical screen, a point on the axis of the cylinder in the middle along the extent of the cylinder is a good choice. Sometimes the sweet spot is determined by actual considerations. For example, a spot that is approximately at eye level is considered ideal because the image is almost always aligned with the horizontal and vertical axes of the real world.

  In the case of the present invention, since the Euclidean reconstruction approximates the display geometry, the virtual viewpoint can be fixed or moved interactively. Because the 3D coordinates of the location are known in the camera coordinate system, finding a sweet spot for the camera is relatively simple.

If it is difficult to determine the parameters of the virtual viewpoint for sweet spot rendering from surface points, one technique finds a plane that most closely approximates the set of points found on the illuminated portion of the display surface. A directed bounding box (OBB) is approximated to a 3D point set {X _i } on the display surface. A point that passes through the center of this box and is at an appropriate distance from the front of the screen along a vector perpendicular to the best approximation plane can be selected as the sweet spot. Since all 3D points on the quadric surface are present in the 3D point concave outer shell, OBB can be obtained as follows.

  Y is

  N × 3 matrix (where,

Is the center of the N3D point), the eigenvector corresponding to the smallest eigenvalue of the 3 × 3 matrix Y ^T Y gives the normal to the best approximation plane, ie the axis with the least variance. On the other hand, the largest side of the OBB, that is, the spread of the 3D points projected on the best approximate plane gives the approximate “diameter” of the screen. The distance of the sweet spot in front of the screen can be selected to be proportional to this diameter, depending on the application and the desired field of view.

  A wide field of view is desirable for immersive applications. This distance should therefore be about half the diameter. For group browsing, this distance can be equal to the diameter.

In the present invention, the secondary transfer function Ψ _i between the virtual view image space and each projector output image is recalculated. This process is very similar to the calculation of Ψ _i Ψ _0i . First, find the projection of the 3D points on the quadric surface into the virtual view image space. Here, it is sufficient to update Ψ _i with the correspondence between the virtual view and the projector image, and the internal and external parameters of the virtual view and the projector.

Display Area The frustum of the virtual view is defined using the sweet spot and the OBB spread. The viewpoint vector is directed from the virtual viewpoint to the center of the OBB. Since the union of overlapping images from multiple projectors can illuminate a large area, the frustum can be “cropped” into a visually pleasing shape such as a rectangle or a circle. For 3D applications, render a set of black quadrilaterals and crop the area outside the desired display area. For example, in the case of a rectangular view, a viewport is created by four large quadrilaterals in the vicinity of the outer edge of the viewport in the projector image. The black quadrilateral, along with the rest of the 3D model, is rendered and warped as follows: For 2D applications, the area outside the input image to be displayed is considered black.

Image Transfer Given a 3D vertex M in the scene to render, find its screen space coordinate m in the virtual view. Next, the post-transmission pixel coordinates m _i in the output image of projector i are found using the secondary transfer function Ψ _oi = {A _i , E _i , e _i }. Then, vertex M is replaced by a vertex m _i, polygon in the scene is rendered. Therefore, the rendering process at each projector is very similar. Each projector's frame buffer automatically stores the appropriate portion of the virtual view image and does not need to explicitly determine the extent of the projector.

Therefore, for rendering, in each projector, for each vertex M,
Calculated pixel m via a virtual view projection (M), further obtains the warped pixel m _i through the secondary transfer function Ψ _{i (m),} and for each triangle T having an apex {M _j},
Render triangles with 2D vertices {m _ji }.

  There are two problems with this approach. First, only the vertices in the scene are warped accurately, not inside the polygon. Second, special processing is required for the visibility classification of polygons. After the secondary transmission, it is theoretically necessary to map the edges between the vertices of the polygons to the secondary curve in the projector image.

  However, the scan conversion converts the curve into a straight line segment between warped vertex positions. This problem is not recognized with a single projector. However, in the case of overlapping projectors, this leads to individual different deviations from the original curve and thus the edges appear out of alignment on the display screen. It is therefore necessary to use a sufficiently fine tessellation of triangles.

  Commercial systems are already available that split the input model on-the-fly and pre-distort so that the input model appears straight in perspective correct rendering on a curved screen. US Pat. No. 6,104,405, “Systems, methods and computer program products for converting image data to non-planar image data” granted to Idaszak et al. On Feb. 26, 1997, incorporated herein by reference. And U.S. Pat. No. 5,319,744 issued to Kelly et al. On June 7, 1994, "Polygon fragmentation method of distortion correction in computer image generating systems". The method of the present invention is compatible with the fine tessellation provided by such systems. Scene geometry pre-distortion in commercial systems is used to avoid two-pass rendering, including the first-pass texture mapping results. In the present invention, instead of pre-distorting the geometry, the image space projection is pre-distorted. As an advantage, the present invention can be implemented in part using a programmable graphics unit (GPU) vertex shader.

Scan Conversion When the pixel locations in the projection of the triangle are warped, information must also be passed along so that the depth buffer generates the appropriate visibility information. Furthermore, it is necessary to pass the appropriate weight value “w” for perspective correct color and texture coordinate interpolation. Therefore,
m (x, y, z, w) = virtual viewpoint projection (M (X))
m _i (x ′ _i , y ′ _i , w ′ _i ) = Ψ _i (m (x / w, y / w), 1)
m _i (x _i , y _i , z _i , w _i ) = [wx ′ _i / w ′ _i , wy ′ _i / w ′ _i , z, w]
Multiply the pixel coordinates by “w” from behind.

Therefore, the vertex _{m i} is the original depth and w values with the secondary transfer function, with the appropriate final pixel coordinates _{(x 'i / w' i} , y 'i / w' i).

FIG. 5 shows a vertex shader code 500. When rendering 2D images, the virtual view image space is subdivided into triangles and the image is mapped as a texture on these triangles. Vertex m of each triangle, as described above, is warped to be vertices (and pixels) m _i using quadratic transfer function. Scan conversion, a color attribute and texture attributes at vertex m automatically converts the vertex m _i, to interpolate between. Similarly, it is possible to render a 3D scene.

Note that warping of projector images using the quadratic transfer function of the present invention differs from rendering quadratic curves on a flat surface. See Watson et al. “A fast algorithm for rendering quadratic curves on raster displays” Proc. 27 ^th Annual SE ACM Conference, 1989.

Luminance Blending Pixel brightness in the overlapping image areas is attenuated using graphics hardware alpha blending. Using the parametric equation of the secondary transfer function, an alpha map is obtained as follows:

Every projector per pixel _{x i} in the projector i, formula:

  To find the corresponding pixel in projector j.

In the case of crossfading, pixels at the boundary of the projector image are attenuated. Therefore, the weight is proportional to the shortest distance from the frame boundary. The weights assigned to the pixels x _i represented by the normalized window pixel coordinates (u _i , v _i ) in the range [0, 1] are:

  It is. In the formula, d (x) is min (u, v, 1-u, 1-v) when 0 ≦ u and v ≦ 1, and d (x) = 0 in other cases. Because a parametric approach is used, it is possible to calculate the corresponding projector pixels and the weights at those locations with sub-pixel alignment accuracy. The sum of the weights at the corresponding projector pixels adds up to exactly 1.0.

  In each projector, the corresponding alpha map is stored as a texture map and rendered as a quadrilateral aligned to the screen at the final stage of rendering.

  The method of the present invention can also be used for projection onto a concave dome. This projection is particularly useful when the dome is made of a translucent or transparent material. When projected from behind, the viewer can have a completely immersive experience without blocking any projector.

The present invention can construct a projector system using a display surface curved for 2D or 3D visualization. The system of the present invention does not require an expensive infrastructure, and can operate in a state where the alignment between the plurality of projectors and the display surface is inadvertent. The automatic registration of the present invention utilizes a secondary image transfer function, eliminating the cumbersome setup and maintenance of the projector, thus reducing costs. The present invention can simplify the construction, calibration, and rendering processes of widely used applications, such as those used in flight simulators, planetariums, high-end visualization theaters, and the like. New applications that will be enabled include low-cost, flexible dome displays, shopping arcades, and projection onto cylinders or columns.

  The present invention provides a sophisticated solution to the problems that have been solved so far by discrete sampling. The advantage is that, unlike prior art systems, the projector of the present invention does not need to be placed in a sweet spot. This is important when applying in the real world, where the sweet spot is usually where the viewer wants to be.

  Although the invention has been described as a preferred exemplary embodiment, it should be understood that various other adaptations and modifications may be made within the spirit and scope of the invention. Accordingly, the scope of the appended claims is to cover within the scope of protection for all such variations and modifications that are within the true spirit and scope of this invention.

4 is a flowchart of preprocessing steps used in a method for projecting an image on a curved surface according to the present invention. 6 is a flow diagram of rendering steps used in a method for projecting an image on a curved surface according to the present invention. 1 is a diagram of a multi-projector system according to the present invention. 1 is a diagram of a multi-projector system according to the present invention. 3 is a vertex shader code for a second order transfer function according to the present invention. FIG. 3 shows homography according to the present invention. It is a block diagram of a some overlapping image.

Claims (18)

Projecting a predetermined image on the display surface;
Obtaining a stereoscopic image pair of the predetermined image;
Obtaining a secondary transfer function between the predetermined image and the stereoscopic image pair through the display surface;
Warping the output image according to the secondary transfer function;
And projecting the warped output image onto the display surface. The method according to claim 1, wherein the display surface is a quadric surface. The display surface is selected from the group consisting of a spherical surface, a hemispherical surface, a spheroidal surface, a dome, a cylindrical surface, a conical surface, a paraboloid, a hyperboloid, a hyperbolic paraboloid, and an ellipsoid. A method of projecting an image on a display surface. The method of projecting an image on a display surface according to claim 2, wherein the quadric surface is concave. The method according to claim 2, wherein the quadratic curved surface is a convex surface. Projecting a plurality of predetermined images onto the display surface from a plurality of viewpoints;
Obtaining a stereoscopic input image pair of the plurality of predetermined images;
Obtaining a secondary transfer function between each predetermined image and the stereoscopic input image pair via the display surface;
Warping a plurality of output images according to the corresponding secondary transfer function;
The method according to claim 1, further comprising: projecting the plurality of warped output images onto the display surface from the plurality of viewpoints. The method according to claim 6, wherein the display surface is a quadric surface. The method of projecting an image according to claim 1, wherein the plurality of warped output images are overlaid and viewed as one seamless image. Obtaining a three-dimensional correspondence between the features of the predetermined image and the corresponding features of the stereoscopic image pair;
Approximating a quadratic surface to the three-dimensional correspondence;
Obtaining a pose of the predetermined image with respect to the stereoscopic input image pair;
The method according to claim 1, further comprising: obtaining the secondary transfer function from the pose and the correspondence. The method according to claim 6, further comprising attenuating the plurality of warped output images according to blending weights before projection. The method according to claim 1, wherein the predetermined image is a checkerboard pattern. The method according to claim 1, further comprising: applying a nonlinear minimization to an objective function to obtain the second order transfer function. The method according to claim 12, wherein the objective function uses a total square transmission error of all pixels in a predetermined image. The method according to claim 1, wherein the output image is rendered with respect to an arbitrary virtual viewpoint. 11. The method of projecting an image on a display surface, further comprising cropping the attenuated and warped output images to an arbitrary viewing frustum. The method according to claim 1, wherein the secondary transfer function is obtained parametrically. The method of projecting an image on a display surface according to claim 1, wherein the warping is performed using a vertex shader of a graphic processing device. A projector configured to project a predetermined image onto a display surface;
A camera configured to obtain a stereoscopic image pair of the predetermined image;
Means for obtaining a secondary transfer function between the predetermined image and the stereoscopic image pair via the display surface;
Means for warping the output image according to the secondary transfer function;
A system for projecting an image on the display surface, the projector comprising: means for projecting the warped output image onto the display surface.

Images