US8201951B2  Catadioptric projectors  Google Patents
Catadioptric projectors Download PDFInfo
 Publication number
 US8201951B2 US8201951B2 US12488190 US48819009A US8201951B2 US 8201951 B2 US8201951 B2 US 8201951B2 US 12488190 US12488190 US 12488190 US 48819009 A US48819009 A US 48819009A US 8201951 B2 US8201951 B2 US 8201951B2
 Authority
 US
 Grant status
 Grant
 Patent type
 Prior art keywords
 light
 projector
 image
 camera
 matrix
 Prior art date
 Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
 Active, expires
Links
Images
Classifications

 H—ELECTRICITY
 H04—ELECTRIC COMMUNICATION TECHNIQUE
 H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
 H04N17/00—Diagnosis, testing or measuring for television systems or their details
 H04N17/04—Diagnosis, testing or measuring for television systems or their details for receivers

 H—ELECTRICITY
 H04—ELECTRIC COMMUNICATION TECHNIQUE
 H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
 H04N9/00—Details of colour television systems
 H04N9/12—Picture reproducers
 H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
 H04N9/3191—Testing thereof
Abstract
Description
This application claims the benefit of U.S. Provisional Application No. 61/116,152 filed Nov. 19, 2008 under 35 U.S.C. §119(e) and is hereby incorporated by reference in its entirety.
1. Field of Invention
The present invention is directed towards methods of creating and optimizing images in a catadioptric projector system. More specifically, the present invention presents methods of addressing defocus and distortion issues in projected images, and particularly in images projected using reflector and/or refractor materials.
2. Description of Related Art
High resolution and high contrast projectors are increasingly used in a wide array of commercial and scientific applications, ranging from shape acquisition, to virtual environments and Imax© theaters. In vision and graphics applications, cameras are often combined with projectors to form a projectorcamera, or ProCam, system, for creating socalled “intelligent projection systems”. Previous work puts forth a vision of intelligent projection systems serving as novel tools for problem solving. Using preauthored 3D models of objects, shader lamps have been used to add virtual texture and animation to real physical objects with nontrivial, complicated shapes. It is also possible to use spacetimemultiplexed illumination in a ProCam system to recover scene depth. Nonetheless, many obstacles still remain to be overcome.
Like pinhole cameras, a projector suffers from limited fieldofview, i.e., it can be difficult to produce a panoramic projection using a single projector. One possible solution is to mosaic multiple projectors, and significant advances have been made in automatic mosaicing of multiple projectors. However, these systems typically require accurate geometric and photometric calibrating between the multiple projectors and a camera.
One approach toward facilitating accurate geometric and photometric calibration of a camera and an individual projector is drawn from an article by Sen et al., entitled “Dual Photography”, published in the Proceedings ACM SIGGRRAPH, 2005, which is herein incorporated by reference in its entirety. Dual photography makes use of Helmholtz reciprocity to use images captured with real cameras to synthesize pseudo images (i.e. dual images) that simulate images “as seen” (or effectively “captured”) by projectors. That is, the pseudo image simulates a captured image as “viewed” by a projector, and thus represents what a projectorcaptured image would be if a projector could capture images.
Helmholtz reciprocity is based on the idea that the flow of light can be effectively reversed without altering its light transport properties. This reciprocity may be summarized by an equation describing the symmetry of the radiance transfer between incoming (ωi) and outgoing (ωo) directions as fr(ωi→ωo)=fr(ωo→ωi), where fr represents the bidirectional reflectance distribution function (BRDF) of a surface.
Thus, dual photography ideally takes advantage of this dual nature (i.e. duality relationship) of a projected image and a captured image to simulate one from the other. Dual photography (and more precisely Helmholtz reciprocity) requires the capturing of the light transport property between a camera and a projector. More specifically, dual photography requires determination of the light transport property (i.e. light transport coefficient) relating an emitted light ray to a captured light ray.
When dealing with a digital camera and a digital projector, however, dual photography requires capturing a separate light transport coefficient relating each projector pixel (i.e. every emitted light ray) to each, and every, camera pixel (i.e. every light sensor that captures part of the emitted light ray), at the resolution of both devices.
Since images projected by a digital projector and captured by a digital camera can be represented as a matrix of data, creation of a light transport matrix T relating the digital projector to the digital camera would facilitate the use of dual photography, which brings the discussion back to the subject of catadioptric optical systems.
Catadioptric optical systems are typically associated with catadioptric telescopes and catadioptric cameras, but an object of the present invention is to create a catadioptric projector.
A second object of the present invention is to provide an omnidirectional projection system that uses a single projector, as opposed to those that require multiple projectors.
Another object of the present invention is to improve images created by a catadioptric projector system.
The above objectives are met in a method for calibrating a digital projector to a digital camera, where the digital projector has an array of projector pixels and the digital camera has an array of sensor pixels. The method includes providing at least one processing unit to execute a series of steps including: (a) obtaining a light transport matrix correlating said array of projection pixels to said array of sensor pixels, said light transport matrix including a plurality of light transport datasets each corresponding to a separate projector pixel within said array of projector pixels and consisting of light transport data as determined by means of the digital camera; (b) generating a blur kernel from the light transport matrix, where the blur kernel is indicative of projection blur, and the blur kernel is further comprised of a plurality of different defocus datasets, each of the different defocus datasets corresponding to a different projector pixel within the projector pixel array; and (c) for each input image to be projected by the digital projector: (1) multiplying the input image by the matrix inverse of said light transport matrix to create an intermediate transformed image; (2) applying deconvolution on said intermediate transformed image using the blur kernel to create a final image; and (3) submitting the final image for projection by the digital projector.
In this method, it is preferred that in step (b), each defocus dataset within the blur kernel is customized to its corresponding projector pixel.
Preferably step (b) of generating the blur kernel from the light transport matrix includes: for each projector pixel within the projector pixel array: (i) identifying a current projector pixel's corresponding light transport dataset within said light transport matrix; (ii) within the identified light transport dataset, identifying all nonzero light transport data entries; (iii) fitting a 2D (i.e. twodimensional) Gaussian kernel to the identified nonzero light transport data entries to create the defocus dataset corresponding to the current projector pixel; and collecting all created defocus datasets to form the blur kernel.
Further preferably in step (2), deconvolution (using the blur kernel), is applied on the intermediate transformed image using a steepest descent iterative method.
Alternatively in step (2), deconvolution is applied on the intermediate transformed image using the blur kernel by the following formulation, where p is the final image, c is the input image, T is the light transport matrix, G is the defocus kernel, D is a current state of a processed image during each iteration, projector pixel coordinates are identified as (r, s), and symbol
represents a convolution operation:p=arg min/D{∥G D−(T ^{−1} ·c)∥^{2 }and ∀r,s 0≦Dr,s≦255}
In a preferred embodiment, the above formulation is achieved by defining Ĝ as a spatially varying kernel satisfying ĝ[r, s]=ĝ[t−r, t−s], where t is the window size of the blur kernel, and repeating the following substeps using c for both Din and D^{0 }in an initial iteration:
 (I) R=Ĝ(D_{in}−GD^{n});
 (II) update D^{n }using D^{n+1}=D^{n}+αR where α={(∥R^{2})/(GR)^{2}}, and individual pixel values within D^{n+1 }are clamped to between 0 and 255, inclusive, at the end of this step;
 (III) compute the following difference for all pixels [r, s] in consecutive iterations of D by
 (IV) Repeat substeps (I) to (III) until the computed difference in step (III) is less than a predefined threshold. Preferably, this threshold is set to βm^{2}, where m^{2 }is the number of pixels of the desired image c and β is a user definable constant between 0 and 1, exclusive. In the current best mode embodiment, β is set to between 0.1 and 0.2, inclusive.
In a currently preferred embodiment, image D is arranged as a column signal vector, and G and Ĝ are two sparse matrices, and the convolution is formulated as matrix multiplication.
In step 1, above, in a preferred embodiment, creation of the matrix inverse of the light transport matrix includes imposing a Display Constraint upon the light transport matrix, where the Display Constraint specifies that any two distinct light rays emitted from the projector will hit the camera's image sensor at distinct parts, and wherein the imposing of the Display Constraint includes (for each row in light transport matrix T): comparing matrix entries along a common row of the light transport matrix, and nullifying by assigning zero values to all but the highest valued matrix entry in the common row, the resultant matrix being a modified light transport matrix T*. In this approach, the inverse matrix of the modified light transport matrix T* is preferably generated by: identifying in turn each column in the modified light transport matrix as a target column, calculating normalized values for notnullified entry values in the target column with reference to the target column; creating an intermediate matrix of equal size as the modified light transport matrix; populating each column in the intermediate matrix with the calculated normalized values of its corresponding target column in the modified light transport matrix, each normalized value in each populated column in the intermediate matrix maintaining a onetoone correspondence with the notnullified entry values in its corresponding column in the modified light transport matrix; and applying a transpose matrix operation on the intermediate matrix. In this approach, the process of calculating normalized values for notnullified entry values in the target column with reference to the target column consists of generating a sum of the squares of only notnullified entry values in the target column and disregarding all nullified values in the target column, and dividing each notnullified entry value by the abovedescribed sum. In other words, if the intermediate matrix is denoted as {hacek over (T)}, a target column in modified light transport matrix T* is denoted as T*r and a corresponding column in {hacek over (T)} is denoted as {hacek over (T)}r, then the construction and population of {hacek over (T)} is defined as {hacek over (T)}r=T*r/(∥T*r∥)^{2}.
Other objects and attainments together with a fuller understanding of the invention will become apparent and appreciated by referring to the following description and claims taken in conjunction with the accompanying drawings.
In the drawings wherein like reference symbols refer to like parts.
The recent introduction and rapid adoption of consumer digital projectors has redefined the landscape for displaying images and videos. To model projection geometry, a projectorcamera system (i.e. a ProCam system), can be setup and the projector can be treated as a virtual pinhole camera. Classical perspective geometry can then be directly applied to analyze the projection process.
The basic construct of a projectorcamera system can be better understood with reference to
Preferably, real camera 25 is a digital camera having an image sensor including a camera sensor pixel array 29 (i.e. light receptor array or image sensor array), consisting of m rows by n columns of individual camera pixels i (i.e. image sensor elements or light receptor pixels). For simplicity, camera sensor pixel array 29 is shown on real camera 25, but it is to be understood that camera sensor pixel array 29 is internal to real camera 25.
Light rays emitted from individual projector pixels j within real projector 21 form an image on a scene (i.e. projection surface) 23 by bouncing off the projection surface 23 (i.e. display environment or scene), which may have an irregular or flat shape, and some of the light rays eventually reach image sensor array 29 within real camera 25. In general, each light ray is dispersed, reflected, and refracted in the scene and hits the camera's image sensor at a number of different locations throughout image sensor array 29. Thus, when a single light ray emitted from an individual projector pixel j reaches real camera 25, the individually projected light ray forms an mbyn image on camera sensor pixel array 29, with each individual camera sensor pixel i receiving a certain amount of light intensity contribution from the projected light ray.
Consequently, although it might appear that ideally a single light transport coefficient relating one projector pixel j to one camera pixel i might be determinable by individually turning ON a single projector pixel j to emit a single light ray to hit a single camera pixel i, this is not the case. In reality, the entire camera sensor pixel array 29 will receive a different contribution of light intensity from the individually emitted light ray. Therefore, each light ray emitted from each individual projector pixel j creates a different image (or light distribution) on image sensor array 29. In other words, each actuated projector pixel j generates a different set, or array, of individual light transport coefficients, one for each camera sensor pixel i within camera sensor pixel array 29. Consequently, each projector pixel's corresponding set (or array) of light transform coefficients will ideally consist of (m×n) [i.e. mmultipliedbyn] individual light transport coefficients, one for each camera pixel i.
If each projector pixel's corresponding set of light transport coefficients is arranged as a column of coefficients to form a composite light transport matrix, T, then the composite light transport matrix T will have a different column of light transport coefficients for each individual projector pixel j. Furthermore, since there is a onetoone correspondence between each light transport coefficient entry (i.e. matrix element) within each column and each camera pixel i, each column represents the entire image captured by camera 25 resulting from a single projector pixel j being turned ON. Accordingly, the entire (i.e. composite) light transport matrix T will consist of (p×q) [i.e. p multipliedby q] columns (one column [i.e. captured image] for each individually turned ON projector pixel j) and (m×n) rows (one row for each individual camera sensor pixel i).
In the following discussion, it is beneficial to view a real projected image as a first vector, p, (i.e. real projection vector) having (p×q) elements (one for each projector pixel j), and a resultant real captured image as a second vector, c, (i.e. real captured image vector) having (m×n) elements (one for each camera pixel i).
If a real projected image (i.e. an image projected using the entire projector pixel array 27) is represented as a (p×q) real projection vector “p” (i.e. a “pbyq vector”), and if its correspondingly captured real image (i.e. an image captured by the entire camera sensor pixel array 29) is represented as an (m×n) real captured image vector “c” (i.e. an “mbyn vector”), then the light transport relationship between real projector 21 and real camera 25 can be written as
c=Tp
where T is a composite light transport matrix that relates real projector 21 to real camera 25. It is to be understood that light transport matrix T would have been previously generated by, for example, individually and separately turning ON each projector pixel j and determining the corresponding light transport coefficient for each individual camera pixel i.
With reference to
In this case, a virtual image 23″ (as projected by virtual projector 25″) would be represented by an (m×n) virtual projection vector, p″. Similarly, a virtually captured image captured by virtual camera 21″ would be represented by a (p×q) virtual captured image vector, c″. Therefore in this dual setup, virtual captured image vector c″ relates to virtual projection vector c″ by a “dual” light transport matrix T″. By the principle of Helmholtz reciprocity, the light transport is equal in both directions. Therefore, the dual light transport matrix T″ for the dual setup (i.e. the duality transformation setup) relates virtual captured image vector c″ to virtual projection vector p″ by following relation:
c″=T″p″
This relationship, however, brings up the problem of how can one determine the multitudes of “virtual” light transport coefficients that make up dual light transport matrix T″? It has been found that a duality transformation from a real light transport matrix, T, to its dual light transport matrix, T″, can be achieved by submitting the real light transport matrix T to a transpose matrix operation. Thus, T″ is the transpose of T, such that T″=T^{T}, and thus c″T^{T }p″.
The present invention builds on the construct of
With reference to
Light emitted from any of projector pixels j from projector pixel array 27 is reflected or refracted by an optical unit (i.e. dome mirror 25) before reaching scene 23 (i.e. projection screen), and subsequently reaching image sensor pixels i within camera sensor pixel array 29. Light transport matrix T correlates each projector pixel j to a camera image (i.e. to sensor pixel array 29).
An alternate sample setup of the construct of
Thus, catadioptric projectors in accord with the present invention assume unknown reflector and/or refractor geometry of the optical units and do not require accurate alignment between the projector and the optical units. In fact, everyday reflective and refractive objects such as plastic mirrors, commodity security domes, and even wine glasses may be used.
The complex geometry of the reflective and refractive objects can cause severe distortions, scattering, and defocusing. Herein are proposed several new algorithms to effectively reduce these artifacts without recovering the reflector and/or refractor geometry(ies). In the presently preferred embodiment, the light transport matrix (LTM) T between a projector and a camera is directly modeled. It is further shown that every two rows of LTM T from the projector to the viewpoint of the camera are nearly orthogonal. A simple method to approximate a pseudo inverse of LTM T is then shown, and applied to finding the optimal input image that produces least projections.
After discussing methods of simplifying the generation and manipulation of LTM T, as well as increasing the circumstances (i.e. environments) wherein LTM T may be practically used, the present invention presents a projection defocus analysis for mirror and thin refractorbased catadioptric projectors. It is shown that defocus blur can be interpreted as spatiallyvarying Gaussian blurs on an input image. After showing how (Gaussian) kernels can be directly measured from LTM T, it is explained how deconvolution may be applied to optimize the input image. Thus, practical uses of catadioptric projectors in panoramic and omnidirectional projections are demonstrated. The presently preferred embodiment thereby achieves a wider fieldofview projection while maintaining sharpness and low geometric and photometric distortions.
To address the question of how to construct a catadioptric projector, it is helpful to reflect on catadioptric camera systems, which use curved reflectors to simulate a longer focal point and/or larger fieldofview. Thus, one may start by placing a virtual pinhole camera in front of specially shaped mirrors or reflectors to synthesize a wider field of view. Classical examples include single viewpoint catadioptric sensors based on hyperbolic or parabolic mirrors and multiple viewpoint sensors based on spherical, conical, and equiangular mirrors. Catadioptric cameras have also been used to capture highresolution 360 degree or cylindrical panoramas. Most of these systems, however, require accurate alignment of the viewing camera. When the camera moves off focus, the reflection rays form complex caustic surfaces that are difficult to analyze.
Below is presented a catadioptric projector that combines a commodity projector with similar optical units used in catadioptric cameras. The key difference, however, is that the presently preferred embodiment does not require knowing or recovering reflector/refractor geometry. Instead, given a projector and a viewpoint (i.e. a desired image as viewed from the camera's viewpoint), the present approach is to compute a transformed image the projector should use to illuminate the scene in order to reproduce the desired image with minimal distortion. The problem is also addressed by studying how light propagates between the projector and the desired viewpoint (i.e. the camera's fieldofvision).
Modeling light transport through a known 3D environment is a wellstudied problem in computer graphics. Consider emitting unit radiance along a light ray l towards a scene. The light ray l can be reflected, refracted, scattered, or simply pass through the scene. The full light transport t_{l }can then be captured in response to the impulse illumination of l. Light transport t_{l }may be referred to as the impulse scatter function, or ISF. The light transport for each incident ray can be measured, and all the light transport measurements can then be concatenated into a light transport matrix, or LTM, T. If the incident illumination is represented as L_{in}, then the outgoing light field can be expressed as a function of the illumination by L_{out}=TL_{in}.
To measure T, one can emit a 2D incident light field L_{in }(i.e., an image) using a projector and capture the 2D (or 4D) outgoing light fields using a camera or a camera array. As is explained above, it has been shown that by the principle of Helmholtz reciprocity, the light transport can be used to model a ‘dual’ setup where the projector is replaced by a virtual camera and the camera by a virtual projector. This enables Dual Photography, where one can synthesize images that appear as though they were captured by the projector, with the scene illuminated by light emitting from the camera.
A goal of the present invention is to find the input image to the catadioptric projector that when used to illuminate the scene, allows the camera to capture a given desired view. This is an inverse light transport problem. A closedform expression for computing the inverse LTM in a Lambertian environment is known, but for nonLambertian environments, such as the ones used in the present catadioptric projector setup, the question of how to effectively approximate the inverse LTM remains an open problem.
As is shown in
A first difference is the shape and size of the mirror used in the present catadioptric projector. In catadioptric cameras, it is often sufficient to use mirrors of approximately the same size as the lens (i.e., typically 35 cm in diameter) to synthesize a desired fieldofview. However, a projector uses a much bigger lens and has a longer focal length. Therefore to produce panoramic projections, it has traditionally been necessary to use larger sized mirrors (≈100 cm in diameter), which in the past typically had to be custom made.
In an effort to avoid the necessity for custom made components, the presently preferred setup may alternatively use a bent plastic mirror 35 to emulate a cylindric mirror (as illustrated in
A second way in which catadioptric projectors differ from catadioptric cameras is in their physical setup. In a catadioptric camera, the physical camera needs to be accurately positioned at the focal point of a hyperbolic or parabolic surface to synthesize a virtual pinhole camera. Hence, the physical camera, itself, will occupy the center part of the captured image. This does not cause much problem for catadioptric imaging as its goal is to capture the surrounding environment and the center portion of the image can be discarded especially if the physical camera is small. However, similar setups in a catadioptric projector can cause severe selfshadowing. Therefore, in the presently preferred setup, the projector is purposely moved away from the focal point of the mirror and is oriented so that the projector will not block the projected light path.
A third difference between catadioptric projectors and catadioptric cameras is that defocusing artifacts are more severe in catadioptric projectors than in catadioptric cameras. Most catadioptric cameras use a small aperture to increase the depthoffield while projectors use large apertures to produce bright images. Additionally, since the virtual projector in a Dual Photography transformation of a catadioptric projector has more than one center of projection (COP), and typically has multiple COPs, while the physical, real projector can only set a uniform focal length, the final projection will exhibit spatiallyvarying defocus blurs.
The abovedescribe three differences each pose a different challenge. To resolve these challenges, novel algorithms for correcting projection distortions by modeling the light transport matrix between the real projector and real camera are first presented. The defocus mode in catadioptric projectors is then analyzed, and effective methods to minimize the defocus blur by preprocessing the input image are then developed. Several applications of the presently preferred system are then demonstrated.
First, the correcting of geometric and photometric distortions is addressed. Given a catadioptric projector and a viewpoint represented by a camera, one seeks to find the optimal image to be displayed by the projector so that it will be viewed with minimal geometric and photometric distortions from the viewpoint of the camera.
This brings up the subject of light transports in catadioptric projectors. As was discussed above in reference to
c=Tp
where T is the light transport matrix (LTM).
The goal is to solve the inverse light transport problem, i.e., given the desired view c and LTM T, what should be the source image p. Intuitively, p can be found with
p=T ^{−1} c
provided one can determine T^{−1}. Thus, an aim of the present approach is to compute the inverse of the LTM T of the cameraprojector pair. However, the sheer size of LTM T makes computing T^{−1 }an extremely challenging task requiring tremendous computational resources. Moreover, it is not always possible to find the inverse of an arbitrary matrix. Fortunately, as is shown below, for most practical settings, it is possible to find an approximation for T^{−1 }that can be efficiently computed.
Thus, the first step towards this aim is to identify a fast method of approximating T^{−1}, the inverse of LTM T. Recalling that the present approach assumes that in a typical projectorcamera setup, any two distinct light rays l and k emanating from the projector will typically hit distinct regions of the camera sensor array, i.e., there will usually be little (or negligible) overlap in the camera pixels hit by light rays l and k. In the present discussion, this assumption is herein called the display constraint.
Furthermore, each column of light transport matrix T is the captured, projected image resulting from lightprojection by one projector pixel j. Since the number of camera sensor pixels hit by one light ray emanating from one projector pixel (i.e. the light footprint created on the camera sensor array due to activation of one projector pixel) is small relative to the pixelcount of the camera sensor array, most column entries in the LTM T will have zero values. Only those column entries in LTM T corresponding to camera sensor pixels hit by light emanating from the corresponding projector pixel j (i.e. corresponding to the light footprint created by the activation of the corresponding projector pixel j) will have nonezero values.
By the display constraint, light from different projector pixels will mostly hit different camera pixels. This means that if two columns of LTM T are placed next to each other, the nonzero entries in the two columns will not line up in the row direction, most of the time. Thus, their dot product will be close to zero. This implies that the columns of T are mostly orthogonal to each other.
The display constraint assumes little overlap of light footprints created by adjacent projector pixels, which is dependent upon the scene. However, the display constraint may be imposed on a captured light transport matrix T, irrespective of the scene, prior to approximating the inverse of the captured light transport matrix T. To impose the display constraint, for each row in LTM T, one compares matrix entries along a common row of LTM T, and nullifies (by assigning zero values) all but the highest valued matrix entry in the common row. The resultant matrix is a modified light transport matrix T*.
This can be better understood with reference to
An example of this is shown in
As was shown in
Returning to
Accordingly, the display constraint can be imposed on a captured LTM T if the dimmer perimeter camera pixels of a light footprint are ignored, and one creates resized light footprints consisting of only the brightest, center sections of the original light footprints. In this case, columns of LTM T become truly orthogonal to each other.
Thus, the orthogonal nature of captured LTM T may be assured by the way in which T is constructed in the abovedescribed process for generating a modified LTM T*. Furthermore, the amount of memory necessary for storing T* is likewise reduced by storing only nonzero values (i.e. the brightest sections of light footprint) within each column, with the understanding that any matrix entry not stored has a zero value by default.
In the modified light transport matrix T*, this process is simplified because the nonzero values to be stored can be quickly identified by storing only the brightest value within each row of captured LTM T, which automatically creates resized light footprints. For example in
If only the brightest valued entry within each row is retained, one obtains the structure of
It is to be understood that to fully define modified light transport matrix T*, one only needs a second matrix column, or second array, to hold index information indicating which groups of partial light footprint sections belong to the same resized light footprint. In the present case, for example, groups Pixel_1 c and Pixel_1 d are part of a single resized light footprint corresponding to a first projector pixel. Similarly, light footprint sections Pixel_2 b and Pixel_2 c together form a second resized light footprint for a second projector pixel, and so on.
Whether one chooses to use the LTM T or the modified LTM T* (upon which the display constraint is imposed to assure that columns are orthogonal to each other), the approximation of the inverse of LTM T (or inverse of modified LTM T*) follows a similar procedure. For the sake of brevity, the present discussion assumes that one is using LTM T, directly.
To determine an approximation to the inverse of LTM T (i.e. to determine an approximation to inverse matrix T^{−1}), it is beneficial to first note that AA^{−1}=I, and that the identity matrix I is comprised of a matrix having entry values set to a value of “one” (i.e. 1) along a diagonal from its top left corner (starting at matrix location (1,1)) to its bottom right corner (ending at last matrix location (w,v)), and having entry values set to “zero” (i.e. 0) everywhere else. In order to compute T^{−1}, one first defines an intermediate matrix {hacek over (T)} such that each column in T is comprised of normalized values, with each column in {hacek over (T)} corresponding to a column in LTM T (or equivalently, to a column in the modified light transport matrix T*). That is,
{hacek over (T)}r=Tr/(∥Tr∥)^{2} , r=1, 2, 3, . . . , pq
where {hacek over (T)}r is the r^{th }column of {hacek over (T)} and pq is the total number of projector pixels in the projector pixel array, which preferably consists of p rows and q columns of individual projector pixels. Since matrix operation ∥Tr∥ defines the square root of the sum of the squares of all values in column r of LTM T, the square of ∥Tr∥ is simply the sum of the squares of all the values in column r. That is,
By dividing each value entry in column r by the sum of the squares of all the values entries in column r, operation {Tr/(∥Tr∥)^{2}} has the effect of normalizing the value entries in column r of matrix T. If one now takes the transpose of {hacek over (T)}r, i.e. flips it on its side such that the first column becomes the top row and the last column becomes the bottom row, the result will be rows of elements that are the normalized values of corresponding columns of elements in T. Therefore, for every column in T, one has the following result:
({hacek over (T)}r ^{T})×(Tr)=1
and
({hacek over (T)}r ^{T})×(Tω)=0, for r≠ω
In other words, multiplying a column of T with a corresponding row in {hacek over (T)}r^{T }always results in a value of 1, and as one multiplies all the columns in T with the corresponding row in {hacek over (T)}r^{T}, one produces a matrix with numeral 1's along its diagonal, and one may place zeroes everywhere else to fully populate the produced matrix.
Therefore, in the case of LTM T, where columns are (or are made) orthogonal to each other, and given the specific construction of intermediate matrix {hacek over (T)}, it has been shown that the transpose of {hacek over (T)} is equivalent to (or is a good approximation of) the inverse of LTM T (or the inverse of modified LTM T*). This means that
{hacek over (T)} ^{T} ≈T ^{−1 }and p≈{hacek over (T)} ^{T} c
Note that the approximation to the inverse of LTM T can only applied to the projector pixels whose emitted light actually reached the camera's sensor array. For the projector pixels whose emitted light never reach any of the camera sensor pixels, the corresponding columns in LTM T will be all zeros, and the abovegiven equation for normalizing each column of LTM T would be undefined. In these cases, one may set the corresponding columns in intermediate matrix {hacek over (T)} to zero columns. Thus {hacek over (T)}^{T }is the inverse of the part of T that covers the overlapping area of the fieldsofviews (FOV) of the projector and the camera. It only recovers the projector pixels in p that fall in the overlapping FOV and blacks out other projector pixels. Once {hacek over (T)}^{T }is computed, it can be directly applied to the target image c to find the optimal projection image p (i.e. p={hacek over (T)}^{T }c).
In the following discussion, matrix {hacek over (T)}^{T }is called the View Projection matrix, such that given a desired view c, one can find an image p defined as p=({hacek over (T)}^{T }c) such that projecting image p produces a scene which, when viewed from the view point of the camera, has the same appearance as c. Since {hacek over (T)}^{T }is effectively an approximation of inverse matrix T^{−1 }(or a modified version of matrix T^{−1 }created by imposing upon it the display constraint) and is used as such herein, for the rest of the following discussion inverse matrix T^{−1 }and View Projection matrix {hacek over (T)}^{T }may be used interchangeably, unless otherwise indicated. In this case, it is to be understood that the symbol T^{−1 }generally represents the inverse of both the unmodified and the modified (i.e. approximated) forms of the light transport matrix T since its modification may be optional.
To acquire LTM T, an approach described in copending application Ser. No. 12/412,244 entitled Small Memory Footprint Light Transport Matrix Capture assigned to the same assignee as the present application and hereby incorporated by reference in its entirety, is preferably adopted. This approach activates (i.e. turns ON) only one row or one column of the projector pixels at a time with a white color of intensity of 255 to create a onerow image or onecolumn image, respectively, of each row or column of the projector pixel array. The camera individually captures each of the projector's onerow images and onecolumn images. Since in the present embodiment, it is assumed that the camera is of equal or higher resolution than the projector's resolution, each projected onerow image is captured by the camera as a corresponding camerarowset image consisting of one row of camera pixels with captured nonzero light transport values (if the camera's resolution is close to or equal to the projector's resolution) or consisting of multiple adjacent rows of camera pixels with captured nonzero light intensity values (if the camera pixel resolution is twice or more the projector's pixel resolution). Similarly, each projected onecolumn of projector pixels is captured by the camera as a cameracolumnset image consisting of one column of camera pixels with captured nonzero light intensity values (if the camera's resolution is equal to or close to the projector's resolution) or consisting of multiple adjacent columns of camera pixels with nonzero captured light transport values (if the camera's pixel resolution is twice or more the projector's pixel resolution). This process is repeated until the camera has captured a camerarowset image of every projected row in the projector pixel array, and captured a cameracolumnset image of every projected column in the projector pixel array. For discussion, within each camerarowset image, camera sensor pixels that capture light emanating from a projected row of projector pixels are identified as nonambient (or nonzero, NZ) pixels since they capture a nonzero light intensity value, and camera pixels that do not capture light emanating from a projected row are identified as zero pixels. Similarly within each captured cameracolumnset image, camera sensor pixels that capture light emanating from a projected column of projector pixels are identified as nonambient (or nonzero, NZ) pixels, and camera pixels that do not capture light emanating from a projected column are identified as zero pixels.
For each projector pixel j_{(r,s)}, (identifying a projector pixel having coordinates (r,s) within the projector pixel array 27), the common nonambient (i.e. nonzero or NZ) camera pixels in the r^{th }camerarowset image and the s^{th }cameracolumnset image are brightened by projector pixel j_{(r,s)}. The coordinates of the nonambient camera pixels common to both the r^{th }camerarowset image and the s^{th }cameracolumnset image, determine the indices of the nonzero values in the column of LTM T corresponding to projector pixel j_{(r,s)}, and the actual nonzero light transport values may be acquired as the average colors (or alternatively as the dimmer) of these individual, common camera pixels over the two images. This process goes through all pairs of projector onerow and projector onecolumn images, and LTM T is thereby acquired in its entirety.
Returning now the subject of catadioptric projectors, as a first example,
Since the present embodiment seeks to provide an affordable catadioptric projector, dome mirror 26 is not an ideal spherical mirror, and is preferably implemented using a commodity security dome. Unlike an ideal spherical mirror, the geometric distortions caused by dome mirror 26 are nonuniform due to microstructures on the mirror, such as bumps and dimples. They also introduce additional photometric distortions like interreflection. Additionally, since the dome mirror 26 used in the present embodiment is not perfectly specular, the center of the projection can be observed to be brighter than the boundary. If one were to project an image directly using dome mirror 26 without any light transport correction, one would observe geometric distortion, such as stretching and shearing of patterns in a final projection.
Two sets of examples of two projected images with and without light transport correction are shown in
Similarly,
In general, the display constraints may be violated if there is significant scatter in the light path to a projection scene. An example of this is demonstrated in a reflector experiment shown in
This approach, however, does not compensate for shadows caused by occlusions. Notice that near the occlusion boundaries of the wine glasses, the incident rays reach the grazing angle. The transparent glasses hence behave like opaque objects near the boundary and cast shadows on the final projection. It is conceived that combining multiple projectors may be used to remove the shadows.
The present light transport model assumes that each pixel from the projector is a single light ray. This corresponds to a small aperture model, but in practice, projectors use large apertures to produce bright images and multiple light rays may pass through the same projector pixel. Ideally, one should preferably use a 4D incident light field, rather than the currently used 2D image, to model the LTM T. Therefore, the inverse LTM of the current embodiment has difficulty correcting visual artifacts caused by the 4D incident light field, such as projection defocus artifacts.
Below is presented a solution for modeling projection defocusing. In the presently preferred embodiment, it is assumed that projection blur can be equivalently interpreted (i.e. modeled) as first blurring the input image by a blur kernel G and then transporting the blurred image by LTM T. This assumption model will be validated in both the reflection and refraction cases. The final light transport from the projector to the camera be then written as:
c=T*·p=T·G·p
where T* is the captured light transport image that includes projector defocus, G is the blur matrix, and T is the light transport matrix based on the small aperture model. The optimal input image p, hence, can be computed as:
p=T* ^{−1} ·c=G ^{−1} ·T ^{−1} ·c
To compute T^{−1}, it is assumed that the display constraint is still satisfied if G is applied on T. Therefore, the inverse LTM algorithm, described above, can be applied on the captured T* matrix to obtain T^{−1}. That is, in the case of defocus in a catadioptric projector, the captured LTM T* is used as an approximation of the LTM T that is based on the small aperture model. Next, the problem of how to compute G^{−1 }is addressed.
First, the cause of projection defocus in reflectorbased catadioptric projectors is analyzed with reference to
If specially shaped reflectors are used as the optical unit, such as hyperbolic or parabolic mirrors, and physical projector 21 is positioned at the focal point, all virtual projectors 21 a/21 b will lie at the same center of projection, COP. To minimize projection defocus, the focal plane of the virtual projectors 21 a/21 b can be set close to the display screen 23. For general mirrors, the loci of virtual projectors 21 a/21 b correspond to complex caustic surfaces. This implies that the virtual projectors 21 a/21 b will lie at different depths with respect to the display screen 23 and thus have different focal points, 32 and 34, for example. Since physical projector 21 can only have a single focal point, the virtual projectors 21 a/21 b will produce spatiallyvarying projection blurs.
A first example of this type of spatiallyvarying projection blurring is shown in
A method for applying defocus compensation on an image after it has received light transport compensation correction is explained below.
Returning now to
To acquire the kernel size, one can project dotted patterns and capture their defocused images. In the abovedescribed method of acquiring light transport matrix T, the nonzero terms in LTM T are a mixture of projection defocus and light transport data (e.g., interreflections). For curved mirrors (including dome mirrors), it preferably assumed that these nonzero terms are dominantly caused by defocus. Therefore, the presently preferred embodiment reuses the captured light transform matrix T* for estimating the blur kernels.
With reference to
The following steps are then repeated for each received input image c, and the process does not end (step 67) until the last input image c is received (step 63). Received input image c is multiplied by T^{−1 }(i.e. the inverse of LTM T) to create an intermediate transformed image c* (step 64). Deconvolution is then applied on intermediate transformed image c* using defocus kernel G to create final transformed image (step 65), which when projected should reproduce desired image “p”. This final transformed image “p” is then submitted to the projector for projection, or stored for later use (step 66).
A description of the preferred method of implementing step 62 of
Thus, for every projector pixel j_{(r,s) }(steps 71 and 7478), the nonzero terms in its corresponding column in LTM T* are first computed. Basically, this means that once LTM T (or equivalently T*) has been acquired, matrix G is populated by copying the nonzero light transport values in each column of T to a corresponding column in G (step 72). A 2D Gaussian kernel g_{r,s }is then fitted to each column in G to create a separate dataset for each projector pixel j (step 73).
Once the defocus kernel G and the inverse LTM T^{−1 }are obtained, the process returns to step 63 of
One method of fitting the 2D Gaussian kernel g_{r,s }to G (i.e. step 73 in
p=arg min/D{∥G
Where symbol is a convolution operator
A key point to be made here is that the defocus blur is applied to (T^{−1}·c) instead of the source image c. Preferably, the steepest descent method is used to approximate a nearly optimal D, where D is a current state of a processed image during each iteration. Further preferably, the desired image c is used as the initial guess D0 as well as Din during an initial iteration (step 82), and the following three steps are repeated:
Firstly (step 83), compute the residue R as:
R=Ĝ
where Ĝ is a spatially varying kernel satisfying ĝ[r, s]=ĝ[t−r, t−s], and t is the window size of the blur kernel (step 81). The image D is columnized as a single vector (i.e. arranged as a column vector), and G and Ĝ are used in sparse matrices format, directly. The convolution can then be formulated as matrix multiplication.
Secondly (step 84), update Dn using the steepest descent approach, as
D ^{n+1} =D ^{n} +αR where α={(R ^{2})/(G
Where (as illustrated in steps 8588) D^{n+1 }is clamped to [0, 255] at the end of step 84. D^{n+1 }is the current state of a processed image during each iteration, and the individual pixels are clamped between 0 and 255.
Thirdly, compute the sum of absolute differences for all projector pixels j(r, s) in consecutive iterations of D (step 89) by:
Step 8389 are repeated until the abovedetermined “Difference” in step 89 is less than a predefined threshold (steps 90, 91, and 92). Preferably, this threshold is set to βm^{2}, where m^{2 }is the number of pixels of the desired image c and β is a user definable constant between 0 and 1, exclusive. In the presently preferred embodiment, β is set between 0.1 and 0.2, inclusive.
Next, projection defocus in catadioptric projectors that use thin layer(s) of transparent surfaces is considered. A typical example is projecting through the wine glasses, as is discussed above in reference to
With reference to
(sin θ/sin α)=m and (sin α/sin β)=(1/m)
To reduce distortion, one should have θ=β, i.e., the exit direction (i.e. exiting angle β) of light ray 94 at the back face should be the same as the incident direction (i.e. incident angle θ) at the front face of refractor 95. However, light ray 94 is slightly shifted an amount Δd from its line of entry 96 by the refraction equation, as follows:
Δd=d tan θ−d tan α=d(tan θ−[sin θ/√(m ^{2}−sin^{2}θ)])
With reference to
Nevertheless, upon exiting the back face of the refractor 95, some portion of the light ray will also be reflected back into the refractor. This leads to scattering artifacts when projecting through thin refractors. However, these scattering artifacts are captured by the LTM T and can be effectively corrected using the inverse LTMbased correction algorithm described above. In the case of thin wine glasses, one need only repeat the analysis on singlelayered thin refractors to model both layers of the wine glasses. Scattering artifacts, however, will be more severe when projecting through multiple layers.
As is shown in
The present approach has successfully demonstrated the use of catadioptric projection systems for displaying panoramic and omnidirectional images.
Capturing and synthesizing panoramic views of complex scenes have attracted much attention in computer graphics and vision. However, very little work has been done to effectively display these panoramic images. The main difficulty lies in the limited aspect ratio of projectors. For example, it is common for panoramas to have an aspect ratio of 3:1 while a commodity projector can typically achieve at most a 1.2:1 aspect ratio. A simple solution is to enforce the same aspect ratio in the projected image (e.g., by displaying a 1024×200 image). However, this approach would waste a large amount of the projector's resolution. Furthermore, since a projector usually has a narrow fieldofview, it is difficult to display a panorama on a wide area.
Returning to
Another use of the present invention is in the area of omnidirectional projection. Developing inexpensive solutions for creating omnidirection projection effects in home or school environments, similar to IMAX© theater environments, can significantly benefit education, scientific visualization, and digital entertainment.
However, difficulty in achieving wide use of omnidirectional projection in various environments lies in several aspects. Firstly, IMAX© projectors use fisheye lenses to simulate 180 degree projection, whereas most commodity projectors use more typical lenses that have a much smaller fieldofview. Secondly, although it is possible to combine multiple projectors to synthesize omnidirectional projections, robust and efficient calibration between multiple projectors is still an open problem. Thirdly, IMAX© systems use domeshaped projection screens, whereas rooms in homes and schools typically provide only rectangularshaped projection areas, such as walls in a rectangular room.
With reference to
With reference to
Two examples of dome projection within a typical room were described above in reference to
Thus, a catadioptric projector analogous to a catadioptric camera has been presented by combining a commodity digital projector with additional optical units. The present system does not require recovering reflector/refractor geometries. In fact, the present setups used everyday objects such as commodity plastic mirrors and wine glasses, whose geometry is difficult to recover. The present approach models the light transport between the projector and a camera's viewpoint using the light transport matrix (LTM). It has further been shown, above, how to efficiently approximate a pseudo inverse LTM, and how to use it in finding the optimal input image that will incur minimum distortion.
Further presented above is a projection defocus analysis for catadioptric projectors. The defocus blur is interpreted as spatiallyvarying Gaussian blurs, where the kernels can be directly measured from the LTM. Deconvolution techniques are then applied to preprocess the input image to minimize the defocus blur in the final projection. The practical uses of catadioptric projectors in panoramic and omnidirectional projections have thus been demonstrated.
It was shown that by using specially shaped reflectors/refractors, catadioptric projectors can offer an unprecedented level of flexibility in aspect ratio, size, and fieldofview while maintaining sharpness and low geometric and photometric distortions.
A characteristic of the present approach is that it does not separate defocus blur from the actual light transport in the captured LTM. For reflectors and thin refractors, either the defocus blur or the scattering will dominate the nonzero terms in the LTM, and thus, they can be separated. However, for more general reflectors or refractors such as thick refractors, the captured LTM is a mixture of blurs and transport. In future developments, it is expected that the two terms may be effectively acquired and separated by using coded patterns. It is further envisioned to use the present LTMbased framework to recover the specular geometry, e.g., by using path tracing. Also under development is realtime distortion and defocusing compensation methods for displaying panoramic videos using commodity projectors. It is expected that catadioptric projectors will serve as conceptual inspiration for designing new generations of projection systems.
While the invention has been described in conjunction with several specific embodiments, it is evident to those skilled in the art that many further alternatives, modifications and variations will be apparent in light of the foregoing description. Thus, the invention described herein is intended to embrace all such alternatives, modifications, applications and variations as may fall within the spirit and scope of the appended claims.
Claims (13)
Priority Applications (2)
Application Number  Priority Date  Filing Date  Title 

US11615208 true  20081119  20081119  
US12488190 US8201951B2 (en)  20081119  20090619  Catadioptric projectors 
Applications Claiming Priority (2)
Application Number  Priority Date  Filing Date  Title 

US12488190 US8201951B2 (en)  20081119  20090619  Catadioptric projectors 
JP2009258618A JP2010122679A (en)  20081119  20091112  Method for calibrating digital projector to digital camera 
Publications (2)
Publication Number  Publication Date 

US20100123784A1 true US20100123784A1 (en)  20100520 
US8201951B2 true US8201951B2 (en)  20120619 
Family
ID=42171702
Family Applications (1)
Application Number  Title  Priority Date  Filing Date 

US12488190 Active 20310127 US8201951B2 (en)  20081119  20090619  Catadioptric projectors 
Country Status (2)
Country  Link 

US (1)  US8201951B2 (en) 
JP (1)  JP2010122679A (en) 
Cited By (2)
Publication number  Priority date  Publication date  Assignee  Title 

US20110010411A1 (en) *  20090711  20110113  KarHan Tan  View projection 
US20130155176A1 (en) *  20111216  20130620  Polycom, Inc.  Reflective and Refractive Solutions to Providing Direct Eye Contact Videoconferencing 
Families Citing this family (13)
Publication number  Priority date  Publication date  Assignee  Title 

US7636489B2 (en) *  20040416  20091222  Apple Inc.  Blur computation algorithm 
US7901093B2 (en) *  20060124  20110308  Seiko Epson Corporation  Modeling light transport in complex display systems 
US7901094B2 (en) *  20070124  20110308  Seiko Epson Corporation  View projection: onetouch setup of light displays on arbitrary surfaces 
US7901095B2 (en) *  20090327  20110308  Seiko Epson Corporation  Resolution scalable view projection 
US20140176592A1 (en) *  20110215  20140626  Lytro, Inc.  Configuring twodimensional image processing based on lightfield parameters 
US8134786B2 (en) *  20100630  20120313  Mitsubishi Electric Research Laboratories, Inc.  Method and system for determining projections in noncentral catadioptric optical systems 
US20120223972A1 (en) *  20110302  20120906  Hon Hai Precision Industry Co., Ltd.  Projecting system and method thereof 
CN102655578A (en) *  20110304  20120905  富泰华工业（深圳）有限公司  Projection system and projection method thereof 
US8976256B2 (en) *  20110321  20150310  The United States Of America As Represented By The Secretary Of The Air Force  Remote sensing of hidden objects 
US8432435B2 (en)  20110810  20130430  Seiko Epson Corporation  Ray image modeling for fast catadioptric light field rendering 
US8704895B2 (en) *  20110829  20140422  Qualcomm Incorporated  Fast calibration of displays using spectralbased colorimetrically calibrated multicolor camera 
US20140078260A1 (en) *  20120920  20140320  Brown University  Method for generating an array of 3d points 
US20160042553A1 (en) *  20140807  20160211  Pixar  Generating a Volumetric Projection for an Object 
Citations (6)
Publication number  Priority date  Publication date  Assignee  Title 

US7088426B2 (en)  20020301  20060808  Nikon Corporation  Projection optical system adjustment method, prediction method, evaluation method, adjustment method, exposure method and exposure apparatus, program, and device manufacturing method 
US7412159B2 (en)  20040721  20080812  Canon Kabushiki Kaisha  Image pickup apparatus and defocus amount correction value calculation method 
US20100034429A1 (en) *  20080523  20100211  Drouin MarcAntoine  Deconvolutionbased structured light system with geometrically plausible regularization 
US20110010411A1 (en) *  20090711  20110113  KarHan Tan  View projection 
US7901093B2 (en) *  20060124  20110308  Seiko Epson Corporation  Modeling light transport in complex display systems 
US20110176007A1 (en) *  20100115  20110721  Yuanyuan Ding  RealTime Geometry Aware Projection and Fast ReCalibration 
Patent Citations (6)
Publication number  Priority date  Publication date  Assignee  Title 

US7088426B2 (en)  20020301  20060808  Nikon Corporation  Projection optical system adjustment method, prediction method, evaluation method, adjustment method, exposure method and exposure apparatus, program, and device manufacturing method 
US7412159B2 (en)  20040721  20080812  Canon Kabushiki Kaisha  Image pickup apparatus and defocus amount correction value calculation method 
US7901093B2 (en) *  20060124  20110308  Seiko Epson Corporation  Modeling light transport in complex display systems 
US20100034429A1 (en) *  20080523  20100211  Drouin MarcAntoine  Deconvolutionbased structured light system with geometrically plausible regularization 
US20110010411A1 (en) *  20090711  20110113  KarHan Tan  View projection 
US20110176007A1 (en) *  20100115  20110721  Yuanyuan Ding  RealTime Geometry Aware Projection and Fast ReCalibration 
NonPatent Citations (23)
Title 

Baker, S., et al., "A Theory of Catadioptric Image Formation", Proceedings of the 6th International Conference on Computer Vision, Bombay, pp. 3542, Jan. 1998. 
Brown, M. S., et al., "Image Preconditioning for OutofFocus Projector Blur", Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Jan. 2006. 
Chen, H., et al., "Scalable Alignment of LargeFormat MultiProjector Displays Using Camera Homography Trees", Proceedings of IEEE Visualization 2002, pp. 339346, Nov. 2002. 
Fujii, K., et al., "A ProjectorCamera System with RealTime Photometric Adaptation for Dynamic Environments", Computer Vision and Pattern Recognition, IEEE Computer Society Conference, p. 1180, vol. 2, Jun. 2025, 2005. 
Grossberg, M. D., et al., "Making One Object Look Like Another: Controlling Appearance Using a ProjectorCamera System", Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. 
Gupta, R., et al., "Linear Pushbroom Cameras", IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, No. 9, pp. 963975, Sep. 1997. 
Lantz, E., "A Survey of LargeScale Immersive Displays", ACM SIGGRAPH in the Emerging Display Technology Conference Proceedings, Aug. 2007. 
Nayar, S. K., "Catadioptric Omnidirectional Camera" IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR, pp. 482, 1997. 
Nayar, S. K., et al., "360×360 Mosaics", IEEE Computer Vision and Pattern Recognition Conference, vol. 2, pp. 388395, 2000. 
Peri, V. N., et al., "Generation of Perspective and Panoramic Video from Omnidirectional Video", Proc. of DARPA Image Understanding Workshop, May 1997. 
Raskar, R., et al., "iLamps: Geometrically Aware and SelfConfiguring Projectors", Appears in ACM SIGGRAPH Conference Proceedings, 2003. 
Raskar, R., et al., "Nonphotorealistic Camera: Depth Edge Detection and Stylized Rendering using MultiFlash Imaging", International Conference on Computer Graphics and Interactive Techniques ACM SIGGRAPH 2005. 
Raskar, R., et al., "Shader Lamps: Animating Real Objects with ImageBased Illumination", Technical Report. UMI Order No. TR00027, 2000. 
Raskar, R., et al., "The Office of the Future: A Unified Approach to ImageBased Modeling and Spatially Immersive Displays", Computer Graphics Proceedings, Annual Conference Series, Jul. 1924, 1998. 
Seitz, S. M., et al., "A Theory of Inverse Light Transport", Computer Vision, 2005. ICCV 2005. Tenth IEEE International Conference, vol. 2, pp. 14401447, Oct. 1721, 2005. 
Sen, P., et al. "Dual Photography", ACM SIGGRAPH 2005 conference proceedings, 2005. 
Srinivasan, M. V., "A New Class of Mirrors for WideAngle Imaging", Computer Vision and Pattern Recognition Workshop Conference, Jun. 2003. 
Sukthankar, R., et al., "Smarter Presentations: Exploiting Homography in CameraProjector Systems", Proceedings of International Conference on Computer Vision, 2001. 
Swaminathan, R., et al., NonSingle Viewpoint Catadioptric Cameras: Geometry and Analysis, International Journal of Computer Vision, 2006. 
Underkoffler, J., et al., "Illuminating Light: An Optical Design Tool with a LuminousTangible Interface", Conference on Human Factors in Computing Systems, Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 542549, 1998. 
Yang, R., et al., "PixelFlex: A Reconfigurable MultiProjector Display System", Proceedings of IEEE Visualization, 2001. 
Yu, J., et al., "Focal Surfaces of Discrete Geometry", Eurographics Symposium on Geometry Processing, 2007. 
Zhang, L., et al., "Projection defocus analysis for scene capture and image display", ACM SIGGRAPH, pp. 907905, 2006. 
Cited By (4)
Publication number  Priority date  Publication date  Assignee  Title 

US20110010411A1 (en) *  20090711  20110113  KarHan Tan  View projection 
US9600855B2 (en) *  20090711  20170321  HewlettPackard Development Company, L.P.  View projection 
US20130155176A1 (en) *  20111216  20130620  Polycom, Inc.  Reflective and Refractive Solutions to Providing Direct Eye Contact Videoconferencing 
US8970655B2 (en) *  20111216  20150303  Polycom, Inc.  Reflective and refractive solutions to providing direct eye contact videoconferencing 
Also Published As
Publication number  Publication date  Type 

JP2010122679A (en)  20100603  application 
US20100123784A1 (en)  20100520  application 
Similar Documents
Publication  Publication Date  Title 

Narasimhan et al.  Chromatic framework for vision in bad weather  
Kruijff et al.  Perceptual issues in augmented reality revisited  
US6921172B2 (en)  System and method for increasing projector amplitude resolution and correcting luminance nonuniformity  
US7213926B2 (en)  Image projection system and method  
US7697750B2 (en)  Specially coherent optics  
Sen et al.  Dual photography  
US8290358B1 (en)  Methods and apparatus for lightfield imaging  
US20060157640A1 (en)  Apparatus and method for capturing still images and video using coded aperture techniques  
US20070024742A1 (en)  Method and apparatus for enhancing flash and ambient images  
Grossberg et al.  Making one object look like another: Controlling appearance using a projectorcamera system  
US20110019056A1 (en)  BiDirectional Screen  
US20070025717A1 (en)  Method and apparatus for acquiring HDR flash images  
US20090034086A1 (en)  Panoramic threedimensional adapter for an optical instrument and a combination of such an adapter and such an optical instrument  
US6304263B1 (en)  Threedimensional display system: apparatus and method  
Tocci et al.  A versatile HDR video production system  
US20070081200A1 (en)  Lensless imaging with controllable apertures  
Tarini et al.  3D acquisition of mirroring objects using striped patterns  
Grossberg et al.  The raxel imaging model and raybased calibration  
US20070291233A1 (en)  Mesh for rendering an image frame  
US7190461B2 (en)  Method and apparatus for determining a bidirectional reflectance distribution function, subsurface scattering or a bidirectional texture function of a subject  
Debevec et al.  Estimating surface reflectance properties of a complex scene under captured natural illumination  
US20060023105A1 (en)  Panoramic video system with realtime distortionfree imaging  
US20040080661A1 (en)  Camera that combines the best focused parts from different exposures to an image  
US20070286514A1 (en)  Minimizing image blur in an image projected onto a display surface by a projector  
Grossberg et al.  A general imaging model and a method for finding its parameters 
Legal Events
Date  Code  Title  Description 

AS  Assignment 
Owner name: EPSON RESEARCH AND DEVELOPMENT, INC.,CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:DING, YUANYUAN;XIAO, JING;SIGNING DATES FROM 20090615 TO20090619;REEL/FRAME:022858/0593 Owner name: EPSON RESEARCH AND DEVELOPMENT, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:DING, YUANYUAN;XIAO, JING;SIGNING DATES FROM 20090615 TO20090619;REEL/FRAME:022858/0593 

AS  Assignment 
Owner name: SEIKO EPSON CORPORATION,JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:EPSON RESEARCH AND DEVELOPMENT, INC.;REEL/FRAME:022965/0525 Effective date: 20090625 Owner name: SEIKO EPSON CORPORATION, JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:EPSON RESEARCH AND DEVELOPMENT, INC.;REEL/FRAME:022965/0525 Effective date: 20090625 

FPAY  Fee payment 
Year of fee payment: 4 