WO2000013423A1 - Procede et dispositif d'imagerie de synthese haute resolution utilisant une camera haute resolution et une camera a resolution plus faible - Google Patents

Procede et dispositif d'imagerie de synthese haute resolution utilisant une camera haute resolution et une camera a resolution plus faible Download PDF

Info

Publication number
WO2000013423A1
WO2000013423A1 PCT/US1999/019706 US9919706W WO0013423A1 WO 2000013423 A1 WO2000013423 A1 WO 2000013423A1 US 9919706 W US9919706 W US 9919706W WO 0013423 A1 WO0013423 A1 WO 0013423A1
Authority
WO
WIPO (PCT)
Prior art keywords
resolution
camera
image
parallax
imagery
Prior art date
Application number
PCT/US1999/019706
Other languages
English (en)
Other versions
WO2000013423A9 (fr
WO2000013423A8 (fr
Inventor
Keith J. Hanna
James R. Bergen
Rakesh Kumar
Harpreet Sawhney
Jeffrey Lubin
Original Assignee
Sarnoff Corporation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sarnoff Corporation filed Critical Sarnoff Corporation
Priority to EP99946672A priority Critical patent/EP1110414A1/fr
Priority to CA002341886A priority patent/CA2341886A1/fr
Priority to JP2000568261A priority patent/JP2002524937A/ja
Publication of WO2000013423A1 publication Critical patent/WO2000013423A1/fr
Publication of WO2000013423A8 publication Critical patent/WO2000013423A8/fr
Publication of WO2000013423A9 publication Critical patent/WO2000013423A9/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/111—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/239—Image signal generators using stereoscopic image cameras using two 2D image sensors having a relative position equal to or related to the interocular distance
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/243—Image signal generators using stereoscopic image cameras using three or more 2D image sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/25—Image signal generators using stereoscopic image cameras using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/296—Synchronisation thereof; Control thereof
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G06T2207/10012—Stereo images
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/15—Processing image signals for colour aspects of image signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/189—Recording image signals; Reproducing recorded image signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/246—Calibration of cameras
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/286—Image signal generators having separate monoscopic and stereoscopic modes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N2013/0074—Stereoscopic image analysis
    • H04N2013/0081—Depth or disparity estimation from stereoscopic image signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N2213/00—Details of stereoscopic systems
    • H04N2213/003—Aspects relating to the "2D+depth" image format

Definitions

  • the invention relates to an image processing apparatus and, more particularly, the invention relates to a method and apparatus for creating a high-resolution synthetic image from two or more cameras that differ by one or more characteristics or parameters.
  • the common practice of the prior art is to use two or more high-resolution devices or cameras, displaced from each other.
  • the first high-resolution camera captures an image or image sequence, that can be merged with other high-resolution images taken from a viewpoint different than the first high-resolution camera, creating a stereo image of the scene.
  • creating stereo imagery using multiple high-resolution cameras can be difficult and very expensive.
  • the number of high- resolution cameras used to record a scene can contribute significantly to the cost of producing the stereo imagery.
  • high-resolution cameras are large and unwieldy. Thus, the ease of which a scene is filmed can be burdensome. Further, some viewpoints may not be able to accommodate the size of such high-resolution cameras, thus limiting the viewpoints available for creating the stereo image.
  • One specific embodiment of the invention is apparatus that comprises a high- resolution camera for producing images at a high-resolution and a lower- resolution camera for producing images at a lower-resolution coupled to an image processor.
  • the image processor Performs various image flow and parallax estimation computations and warps the high-resolution image to a viewpoint of the lower-resolution camera.
  • the invention includes a method that is embodied as a software routine, or a combination of software and hardware.
  • the inventive method comprises the steps of supplying image data having a high- resolution, supplying image data having a lower-resolution, processing the imagery, then warping the high-resolution image to a viewpoint of the lower-resolution image data to form a synthetic image.
  • the original high-resolution image and the synthetic image can be used to form a high-resolution stereo image using only a single high-resolution camera.
  • Fig. 1 depicts a block diagram of an imaging apparatus incorporating the image analysis method and apparatus of the invention
  • Fig. 2 depicts a block schematic of an imaging apparatus and an image analysis method used to produce one embodiment of the subject invention
  • Fig. 3 is a flow chart of the parallax computation method; and, Fig. 4 is a flow chart of the image compositing method.
  • identical reference numerals have been used, where possible, to designate identical elements that are common to the figures.
  • FIG. 1 depicts a high-resolution synthetic image generation apparatus 100 of the present invention.
  • An input video sequence 112 is supplied to a computer 102.
  • the input sequence 112 may comprise of a pair of frames taken at a single instance, a series of frame pairs taken over time or a series of frames.
  • the computer 102 comprises a central processing unit (CPU) 104, support circuits 106, and memory 108. Residing within the memory 108 is a high-resolution synthetic image generation routine 110.
  • the high-resolution synthetic image generation routine 110 may alternately be readable from another source such as a floppy disk, CD, remote memory source or via a network.
  • the computer additionally is coupled to input/output accessories 118.
  • an input video sequence 112 is supplied to the computer 102, which after operation of the high-resolution synthetic image generation routine 110, outputs a synthetic high-resolution image 114.
  • An example embodiment of a transform related to the spatial positions of the sensors are the parallax recovery methods described below.
  • An example embodiment of a transform related to the spatial resolution of the sensor is described in Burt and Adelson "Laplacian Pyramid as a compact Image code", where images are transformed from one resolution to other resolutions in the process of computing an image pyramid.
  • An example embodiment of a transform relating to spectral characteristics of the sensors is a mapping from HSL (Hue,saturation,lightness) to RGB (Reg,Green,Blue) as described in “Graphics Gems", edited by Andrew Glassner, Academic Press, 1990.
  • An example of a transform that relates the spatial layout of imagery recorded from one sensor to another spatial layout is described in "A Theory of Catadioptric Image Formation” by S. Baker and S.K. Nayar in the Proceedings of the 6th International Conference on Computer Vision, Pages 35-42, Bombay, India, January, 1998.
  • the high-resolution synthetic image generation routine 110 hereinafter referred to as the routine 110, can be understood in greater detail by referring to Fig. 2.
  • the process of the present invention is discussed as being implemented as a software routine 110, some of the method steps that are disclosed therein may be performed in hardware as well as by the software controller. As such, the invention may be implemented in software as executed upon a computer system, in hardware as an application specific integrated circuit or other type of hardware implementation, or a combination of software and hardware.
  • each step of the routine 110 should also be construed as having an equivalent application specific hardware device (module), or hardware device used in combination with software.
  • the high-resolution synthetic image generation routine 110 receives the input 112 from a high resolution camera 206 and a lower resolution camera 206.
  • the high resolution camera 206 views a scene 200 from a first viewpoint 216 while the lower resolution camera 206 views the scene 200 from a second viewpoint 218.
  • the high resolution camera 206 has an image resolution higher than that of the lower resolution camera 206.
  • the high resolution camera 206 may comprise a number of different devices having a number of different data output formats, as one skilled in the art will readily be able to adapt the process described by the teachings herein to any number of devices and data formats and/or protocols.
  • the high resolution camera 206 is a high-definition camera, i.e., a model MSM9801 camera, available from IMAX ® Corporation.
  • the lower resolution camera 206 may also comprise a varied number of devices, since one skilled in the art can readily adapt the routine 110 to various devices as discussed above.
  • the low-resolution device is a camera having a resolution lower than the resolution of the high-resolution device, i.e., a standard definition video camera.
  • the resolution imagery may have at least 8000 by 6000 pixels/cm 2 and the lower resolution image may have 1000 by 1000 pixels/cm 2 .
  • the routine 110 receives input data from the high resolution camera 206 and corrects the spatial, intensity and chromanence (chroma) distortions in step 202.
  • the chroma distortions are caused by, for example, lens distortion. This correction is desired in order to improve the accuracy of subsequent steps executed in the routine 110.
  • Methods are known in the art for computing a parametric function that describes the lens distortion function. For example, the parameters are recovered in step 202 using a calibration procedure as described in H. S. Sawhney and R. Kumar, True Multi-Image Alignment and its Application to Mosaicing and Lens Distortion, Computer Vision and Pattern Recognition Conference proceedings, pages 450-456, 1997, incorporated by reference in its entirety herein.
  • Chroma and intensity corrections are also performed in step 202. These correction are necessary since image data from the lower resolution camera 206 is merged with data from the high resolution camera 206, and any differences in the device response to scene color and intensity or due to lens vignetting, for example, results in image artifacts in the synthesized image 114.
  • the correction is performed by pre-calibrating the devices (i.e., the high resolution camera 206 and the lower resolution camera 206) such that the mapping of chroma and intensity from one device to the next is known.
  • the measured chroma and intensity correction information from each device is stored in look-up tables or as a parametric function.
  • Input data from the lower resolution camera 206 is also corrected for spatial, intensity and chroma distortions in step 204.
  • the process for correcting the low-resolution distortions in step 204 follow the same process as the corrections performed in step 202.
  • step 210 The corrected high-resolution data from step 202 is subsequently filtered and subsampled in step 210 .
  • the purpose of step 210 is to reduce the resolution of the high-resolution imagery such that it matches the resolution of the low-resolution image.
  • Step 210 is necessary since features that appear in the high-resolution imagery may not be present in the lower-resolution imagery, and cause errors in a depth recovery process (step 306 detailed in Fig. 3 below). Specifically, these errors are caused since the depth recovery process 306 attempts to determine the correspondence between the high-resolution imagery and the low- resolution imagery, and if features are present in one image and not the other, then the correspondence process is inherently error-prone.
  • the step 210 is performed by first calculating the difference in spatial resolution between the high resolution camera 206 and low resolution camera 208. This is performed as a pre-calibration step in which the relative scale of pixels/cm 2 between the two cameras is computed. For example, this relative scale is given by the ratio of lengths or square root of the ratio of areas of a fixed shape that is viewed by the two cameras. From the difference in spatial resolution, a convolution kernel can be computed that reduces the high-frequency components in the high- resolution imagery such that the remaining frequency components match those components in the low-resolution imager. This can be performed using standard, sampling theory (e.g., see P. J. Burt and E. H. Adelson, The Laplacian Pyramid as a Compact Image Code, IEEE Transactions on Communication, Vol. 31, pages 532-540, 1983, incorporated by reference herein in its entirety).
  • standard, sampling theory e.g., see P. J. Burt and E. H. Adelson, The Laplacian Pyramid as a Compact Image Code, IEEE
  • an appropriate filter kernel is [1,4,6,4,1]/16.
  • the filter kernel is applied first vertically, then horizontally.
  • the high-resolution image can then be sub-sampled by a factor of 2 so that the spatial sampling of the image data derived from the high-resolution imager matches that of the low-resolution imager.
  • the parallax is computed in step 212 at each frame time to determine the relationship between viewpoint 216 and viewpoint 218 in the high-resolution and low-resolution data sets. More specifically, the parallax computation of step 212 computes the displacement of image pixels between the images taken from view point 216 and viewpoint 218 due to their difference in viewpoint of the scene 200.
  • this parallax information depends on the relationship between the two input images recorded at a common instance in time and having different viewpoints (216 and 218, respectively) of a scene 200, it is initially computed at the spatial resolution of the lower resolution image. This is accomplished by resampling the high-resolution input image using an appropriate filtering and sub-sampling process, as described above in step 210.
  • step 212 is performed using more or less constrained algorithms depending on the assumptions made about the availability and accuracy of calibration information. In the extremely unconstrained case, a two-dimensional flow vector is computed for each pixel in the image for which alignment is being performed. If it is known that the epipolar geometry is stable and accurately known, then the computation reduces to a single value for each image point.
  • parallax computations are, in effect, a constrained computation of image flow.
  • One method of parallax computation is known as "plane plus parallax".
  • the plane plus parallax representation can be used to reduce the size of per- pixel quantities that need to be estimated. For example, in the case where scene 200 comprises an urban scene with a lot of approximately planar facets, parallax may be computed in step 212 as a combination of planar layers with additional out-of-plane component of structure.
  • the procedure for performing the plane plus parallax method is detailed in United State Patent Application No.
  • step 212 can be satisfied by simply computing parallax using the plane plus parallax method described above, there are a number of techniques that can be used to make the basic two-frame stereo parallax computation of step 212 more robust and reliable. These techniques may be performed singularly or in combination to improve the accuracy of step 212.
  • the techniques are depicted in the block diagram of Fig. 3 and comprise of augmentation routines 302, sharpening routines 304, routines that compute residual parallax 306, occlusion detection routines 308, and motion analysis routines 310. Although these processes are discussed as being useful in improving a parallax computation, the same augmentation processes can be applied to an image flow computation to enhance the accuracy of an image flow estimation.
  • the augmentation routines 302 make the basic two-frame stereo parallax computation robust and reliable.
  • One approach divides the images into tiles and, within each tile, the parameterization is of a dominant plane and parallax.
  • the dominant plane could be a frontal plane.
  • the planar parameterization for each tile is constrained through a global rotation and translation (which is either known through pre-calibration of the stereo set up or can be solved for using a direct method).
  • a single epipolar constraint is applied to all the parallax vectors for any planar tile.
  • Another augmentation routine 302 handles occlusions and textureless areas that may induce errors into the parallax computation.
  • depth matching across two frames is performed using varying window sizes, and from coarse to fine spatial frequencies. Multiple window sizes are used at any given resolution level to test for consistency of depth estimate and the quality of the correlation. Depth estimate is considered reliable only if at least two window sizes produce acceptable correlation levels with consistent depth estimates. Otherwise, the depth at that level is not updated. If the window under consideration does not have sufficient texture, the depth estimate is ignored and a consistent depth estimate from a larger window size is preferred if available.
  • Areas in which the depth remains undefined are labeled as such as to that they can be filled in either using preprocessing, i.e., data from the previous synthetic frame or through temporal predictions using the low-resolution data, i.e., up-sampling low-resolution data to fill in the labeled area in the synthetic image 114.
  • preprocessing i.e., data from the previous synthetic frame or through temporal predictions using the low-resolution data, i.e., up-sampling low-resolution data to fill in the labeled area in the synthetic image 114.
  • JND Just Noticeable Difference
  • An additional augmentation routine 302 provides an algorithm for computing image location correspondences. First, all potential correspondences at image locations are defined by a given camera rotation and translation at the furthest possible range, and then correspondences are continuously checked at point locations corresponding to successively closer ranges. Consistency between correspondences recovered between adjacent ranges gives a measure of the accuracy of the correspondence.
  • Another augmentation routine 302 avoids blank areas around the perimeter of the synthesized image. Since the high-resolution imagery is being warped such that it appears at a different location, the image borders of the synthesized image may not have a correspondence in the original synthesized image. Such areas may potentially be left blank. This problem is solved using three approaches. The first approach is to display only a central window of the original and high-resolution imagery, such that the problem area is not displayed. The second approach is to use data from previous synthesized frames to fill in the region at the boundary. The third approach is to filter and up-sample the data from the low-resolution device, and insert that data at the image boundary.
  • An additional augmentation routine 302 provides an algorithm that imposes global 3D and local (multi-) plane constraints Specifically, the approach is to represent flow between frame pairs as tiled parametric (with soft constraints across tiles) and smooth residual flow. In addition, even the tiles can be represented in terms of a small number of parametric layers per tile. In the case when there is a global 3D constraint across the two frames (stereo), then the tiles are represented as planar layers where within a patch more than one plane may exist.
  • Another method for improving the quality of the parallax computation of step 212 is to employ a sharpening routine 304. For example, in the neighborhood of range discontinuities or other rapid transitions, there is typically a region of intermediate estimated parallax due to the finite spatial support used in the computation process 212. Explicit detection of such transitions and subsequent "sharpening" of the parallax field minimize these errors.
  • information from earlier (and potentially later) portions of the image sequence is used to improve synthesis of the high-resolution image 114. For example, image detail in occluded areas may be visible from the high-resolution device in preceding or subsequent frames. Use of this information requires computation of motion information from frame to frame as well as computation of parallax. However, this additional computation is performed as needed to correct errors rather than on a continual basis during the processing of the entire sequence.
  • the parallax computation of step 212 can be improved by computing the residual parallax (depth) using a method described as follows or an equivalent method that computes residual parallax 306.
  • depth residual parallax
  • One method computes depth consistency over time to further constrain depth/disparity computation when a motion stereo sequence is available as is the case, for example, with a 15-65 formatted hi-resolution still image.
  • rigidity constraint is valid and is exploited in the two-frame computation of depth outlined above.
  • optical flow is computed between the corresponding frames over time. The optical flow serves as a predictor of depth in the new frames.
  • depth computation is accomplished between the pair while being constrained with soft constraints coming from the predicted depth estimate.
  • Another method of computing residual parallax 306 is to use the optical flow constraint along with a rigidity constraint for simultaneous depth/disparity computation over multiple stereo pairs.
  • the temporal rigidity constraint is parameterized in the depth computation in exactly the same manner as the rigidity constraint between the two frames at the same time instant.
  • the optical flow constraint over time may be employed as a soft constraint as a part of the multi-time instant depth computation.
  • Another method of computing residual parallax 306 is to constrain depth as consistent over time to improve alignment and maintain consistency across the temporal sequence. For example, once depth is recovered at one time instant, the depth at the next frame time can be predicted by shifting the depth by the camera rotation and translation recovered between the old and new frames. This approach can also be extended by propagating the location of identified contours or occlusion boundaries in time to improve parallax or flow computation.
  • An additional approach for computing residual parallax 306 is to directly solve for temporally smooth stereo, rather than solve for instantaneous depth, and impose subsequent constraints to smooth the result.
  • This can be implemented using a combined epipolar and flow constraint. For example, assume that previous synthesized frames are available. The condition imposed on the newly synthesized frame is that it is consistent with the instantaneous parallax computation and that it is smooth in time with respect to the previously generated frames. This latter condition can be imposed by making a flow-based prediction based on the previous frames and making the difference from that prediction part of the error term.
  • the parallax-based frame i.e., the warped high-resolution image
  • the flow based temporally interpolated frame can be compared with the flow based temporally interpolated frame. This comparison can be used either to detect problem areas or to refine the parallax computation.
  • This approach can be used without making rigidity assumptions or in conjunction with a structure/parallax constraint. In this latter case, the flow-based computation can operate with respect to the residual motion after the rigid part has been compensated.
  • An extension of this is to apply the planar constraint across frames along with the global rigid motion constraint across all the pixels in one frame.
  • a method of computing residual parallax 306 which avoids a potential problem with instability in the synthetic stereo sequence in three dimensional structure composed using the synthetic image 114 is to limit the amount of depth change between frames. To reduce this problem, it is important to avoid temporal fluctuations in the extracted parallax structure using temporal smoothing. A simple form of this smoothing can be obtained by simply limiting the amount of change introduced when updating a previous estimate. To do this in a systematic way requires inter-frame motion analysis as well as intra-frame parallax computation to be performed.
  • Occlusion detection 308 is helpful in situations in which an area of the view to be synthesized is not visible from the position of the high- resolution camera. In such situations, it is necessary to use a different source for the image information in that area. Before this can be done, it is necessary to detect that such a situation has occurred. This can be accomplished by comparing results obtained when image correspondence is computed bi-directionally. That is, in areas in which occlusion is not a problem, the estimated displacements from computing right-left correspondence and from computing left-right correspondence agree. In areas of occlusion, they generally do not agree. This leads to a method for detecting occluded regions. Occlusion conditions can also be predicted from the structure of the parallax field itself. To the extent that this is stable over time areas of likely occlusion can be flagged in the previous frame. The bi-directional technique can then be used to confirm the condition. Motion analysis 310 also improves the parallax computation of step
  • Motion analysis 310 involves analyzing frame-to-frame motion within the captured sequence. This information can be used to solve occlusion problems because regions not visible at one point in time may have been visible (or may become visible) at another point in time. Additionally, the problem of temporal instability can be reduced by requiring consistent three-dimensional structure across several frames of the sequence.
  • Analysis of frame-to-frame motion generally involves parsing the observed image change into components due to viewpoint change (i.e., camera motion), three dimensional structure and object motion.
  • viewpoint change i.e., camera motion
  • techniques for performing this decomposition and estimating the respective components include direct camera motion estimation, motion parallax estimation, simultaneous motion and parallax estimation, and layer extraction for representation of moving objects or multiple depth surfaces.
  • a key component of these techniques is the "plane plus parallax" representation.
  • parallax structure is represented as the induced motion of a plane (or other parametric surface) plus a residual per pixel parallax map representing the variation of induced motion due to local surface structure.
  • the parallax estimation techniques referred to above are essentially special cases of motion analysis techniques for the case in which camera motion is assumed to be given by the fixed stereo baseline.
  • the parallax computation (or flow computation) can be performed at the resolution of the low resolution image. Then, the parallax information can be projected to generate a correspondence map at the higher resolution. The subsequent image warping and/or compositing process is then performed using the projected parallax information.
  • the parallax field has been computed in step 212, it is used to produce the high-resolution synthesized image 114 in step 214.
  • the compositing and warping step 214 is depicted in Fig. 2 and in greater detail in Fig. 4.
  • this process involves two steps: parallax interpolation and image warping. In practice these two steps are usually combined into one operation as represented by step 214.
  • step 214 for each pixel in the to-be-synthesized image, the computation of step 214 involves accessing a displacement vector specifying a location in the high- resolution source image from the high resolution camera 206 (step 502), accessing the pixels in some neighborhood of the specified location and computing, based on those pixels (step 504), an interpolated value for the synthesized pixels that comprise the synthetic image 114 (step 506).
  • Step 214 should be performed at the full target image resolution.
  • the interpolation step 506 should be done using at least a bilinear or bicubic interpolation function.
  • step 508 Even more effective warping and compositing algorithms can make use of motion, parallax, other information (step 508).
  • the location of depth discontinuities from the depth recovery process can be used to prevent spatial interpolation in the warping across such discontinuities. Such interpolation can cause blurring in such regions.
  • occluded areas can be filled in with information from previous or following frames using flow based warping. The technique describe above in the discussion of plane plus parallax is applicable for accomplishing step 508.
  • temporal scintillation of the synthesized imagery can be reduced using flow information to impose temporal smoothness (step 510).
  • This flow information can be both between frames in the synthesize sequence, as well as between the original and synthesized imagery.
  • Scintillation can also be reduced by adaptively peaking pyramid-based appearance descriptors for synthesized regions with the corresponding regions of the original high resolution frames. These can be smoothed over time to reduce "texture flicker.”
  • the compositing and warping step 214 can also be performed using data collected over an image patch, rather than just a small neighborhood of pixels. For example, the image can be split up into a number of separate regions, and the resampling is performed based on the area covered by the region in the target image (step 512).
  • the depth recovery may not produce completely precise depth estimates at each image pixel. This can result in a difference between the desired intensity or chroma value and the values produced from the original high-resolution imagery.
  • the warping module can then choose to select one or more of the following options as a correction technique (step 514), either separately, or in combination:
  • JND Just Noticeable Difference
  • the JND measure is performed on the synthesized sequence by comparing the difference between a low-resolution form of the synthesized data and data from the low-resolution camera to create a JND map representing a quality-of-parallax computation measure.
  • JND measures are described in United States Patent Application No.'s 09/055,076, filed April 3, 1989, 08/829,540, filed March 28, 1997, 08/829,516, filed March 28, 1997, and 08/828,161, filed March 28, 1997 and United States Patent No.'s 5,738,430 and 5,694,491, all of which are incorporated herein by reference in their entireties.
  • the JND can be performed between the synthesized high-resolution image data, and the previous synthesized high-resolution image after being warped by the flow field computed from the parallax computation in step 212.
  • the original high resolution image and the synthetic image can be used to form a high resolution stereo image.

Abstract

L'invention concerne un dispositif et un procédé permettant la transformation d'images enregistrées par une caméra en images différentes des premières. Ce disposition et ce procédé utilisent des images collectées par une ou plusieurs caméras additionnelles qui présentent des caractéristiques ou des paramètres différents de la première caméra. Ces différences portent par exemple sur la position spatiale, la résolution spatiale, les caractéristiques spectrales et la disposition spatiale de la caméra. Ce dispositif génère une image de synthèse haute résolution au moyen d'une caméra (206) haute résolution et d'une caméra (208) à résolution plus faible. Les données image haute résolution sont déformées au moyen des données faible résolution afin de produire une image (114) de synthèse haute résolution présentant l'angle de vue de la caméra (208) à résolution plus faible. La routine (110) de génération d'images de synthèse haute résolution comprend les étapes suivantes : correction des distorsions spatiales, d'intensité et de chrominance des données image relevées par la caméra (206) haute résolution et la caméra (206) faible résolution (étape 202), puis filtrage et sous-échantillonnage des données haute résolution corrigées (étape 210), calcul de la parallaxe entre les données haute résolution et les données faible résolution (étape 212) et déformation de l'image haute résolution pour créer une image (114) de synthèse de la scène (200) présentant l'angle de vue de la caméra (208) à résolution plus faible (étape 214).
PCT/US1999/019706 1998-08-28 1999-08-30 Procede et dispositif d'imagerie de synthese haute resolution utilisant une camera haute resolution et une camera a resolution plus faible WO2000013423A1 (fr)

Priority Applications (3)

Application Number Priority Date Filing Date Title
EP99946672A EP1110414A1 (fr) 1998-08-28 1999-08-30 Procede et dispositif d'imagerie de synthese haute resolution utilisant une camera haute resolution et une camera a resolution plus faible
CA002341886A CA2341886A1 (fr) 1998-08-28 1999-08-30 Procede et dispositif d'imagerie de synthese haute resolution utilisant une camera haute resolution et une camera a resolution plus faible
JP2000568261A JP2002524937A (ja) 1998-08-28 1999-08-30 高解像度カメラと低解像度カメラとを用いて高解像度像を合成する方法および装置

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US9836898P 1998-08-28 1998-08-28
US60/098,368 1998-08-28
US38439699A 1999-08-27 1999-08-27
US09/384,396 1999-08-27

Publications (3)

Publication Number Publication Date
WO2000013423A1 true WO2000013423A1 (fr) 2000-03-09
WO2000013423A8 WO2000013423A8 (fr) 2000-05-25
WO2000013423A9 WO2000013423A9 (fr) 2000-08-17

Family

ID=26794682

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US1999/019706 WO2000013423A1 (fr) 1998-08-28 1999-08-30 Procede et dispositif d'imagerie de synthese haute resolution utilisant une camera haute resolution et une camera a resolution plus faible

Country Status (4)

Country Link
EP (1) EP1110414A1 (fr)
JP (1) JP2002524937A (fr)
CA (1) CA2341886A1 (fr)
WO (1) WO2000013423A1 (fr)

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1354292A1 (fr) * 2000-12-01 2003-10-22 Imax Corporation Techniques et systemes pour la mise au point d'imagerie haute resolution
JP2009020879A (ja) * 2002-03-25 2009-01-29 Trustees Of Columbia Univ In The City Of New York データ品位を向上する方法及びシステム
US7856055B2 (en) 2002-03-13 2010-12-21 Imax Corporation Systems and methods for digitally re-mastering or otherwise modifying motion pictures or other image sequences data
US8000521B2 (en) 2004-06-25 2011-08-16 Masataka Kira Stereoscopic image generating method and apparatus
WO2016097470A1 (fr) * 2014-12-15 2016-06-23 Nokia Technologies Oy Système multi-caméra comprenant des caméras étalonnées différemment
EP3068124A4 (fr) * 2013-12-06 2017-01-04 Huawei Device Co., Ltd. Procédé, dispositif et terminal de traitement d'image
WO2017121058A1 (fr) * 2016-01-13 2017-07-20 南京大学 Système d'acquisition d'informations entièrement optiques
CN109155854A (zh) * 2016-05-27 2019-01-04 松下电器（美国）知识产权公司 编码装置、解码装置、编码方法及解码方法
KR101937673B1 (ko) 2012-09-21 2019-01-14 삼성전자주식회사 3ｄ 디스플레이에 대한 ｊｎｄｄ 모델을 생성, 상기 ｊｎｄｄ 모델을 이용하여 깊이 영상을 개선하는 방법 및 시스템
CN117036987A (zh) * 2023-10-10 2023-11-10 武汉大学 一种基于小波域交叉配对的遥感影像时空融合方法及系统

Families Citing this family (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001266128A (ja) * 2000-03-21 2001-09-28 Nippon Telegr & Teleph Corp <Ntt> 奥行き情報取得方法，装置および奥行き情報取得プログラムを記録した記録媒体
JP2005210217A (ja) * 2004-01-20 2005-08-04 Olympus Corp ステレオカメラ
JP2008511080A (ja) * 2004-08-23 2008-04-10 サーノフ コーポレーション 融合画像を形成するための方法および装置
JP4612835B2 (ja) * 2004-12-20 2011-01-12 キヤノン株式会社 画像表示装及び画像表示システム
EP1851727A4 (fr) * 2005-02-23 2008-12-03 Craig Summers Modelisation automatique de scenes pour camera 3d et video 3d
CA2636858C (fr) 2006-01-27 2015-11-24 Imax Corporation Procedes et systemes pour le rematricage numerique de films bi- et tridimensionnels pour une presentation avec une qualite visuelle amelioree
CA2884702C (fr) 2006-06-23 2018-06-05 Samuel Zhou Procedes et systemes de conversion d'images cinematographiques 2d pour une representation stereoscopique 3d
ES2578022T3 (es) * 2009-02-17 2016-07-20 Koninklijke Philips N.V. Combinación de datos de imágenes 3D y gráficos
WO2012029299A1 (fr) * 2010-08-31 2012-03-08 パナソニック株式会社 Dispositif de capture d'images, dispositif de lecture et procédé de traitement d'images
WO2012029298A1 (fr) 2010-08-31 2012-03-08 パナソニック株式会社 Dispositif de capture d'images et procédé de traitement d'images
WO2012029301A1 (fr) * 2010-08-31 2012-03-08 パナソニック株式会社 Appareil de capture d'image, appareil de lecture et procédé de traitement d'image
US10200671B2 (en) 2010-12-27 2019-02-05 3Dmedia Corporation Primary and auxiliary image capture devices for image processing and related methods
US8274552B2 (en) * 2010-12-27 2012-09-25 3Dmedia Corporation Primary and auxiliary image capture devices for image processing and related methods
KR20130001541A (ko) * 2011-06-27 2013-01-04 삼성전자주식회사 다시점 영상의 해상도 복원 방법 및 장치
JP5940812B2 (ja) * 2012-01-12 2016-06-29 日本放送協会 多視点画像処理装置、方法及びプログラム
CN105528793A (zh) * 2016-01-12 2016-04-27 北京建筑大学 一种高空间分辨率遥感影像的分割方法及系统

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0641132A1 (fr) * 1993-08-26 1995-03-01 Matsushita Electric Industrial Co., Ltd. Appareil d'affichage et de prise de vues stéréoscopiques
WO1998021690A1 (fr) * 1996-11-13 1998-05-22 Sarnoff Corporation Alignement d'images a partir de vues multiples avec application au mosaiquage et a la correction de distorsions de lentilles

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0641132A1 (fr) * 1993-08-26 1995-03-01 Matsushita Electric Industrial Co., Ltd. Appareil d'affichage et de prise de vues stéréoscopiques
WO1998021690A1 (fr) * 1996-11-13 1998-05-22 Sarnoff Corporation Alignement d'images a partir de vues multiples avec application au mosaiquage et a la correction de distorsions de lentilles

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1354292A1 (fr) * 2000-12-01 2003-10-22 Imax Corporation Techniques et systemes pour la mise au point d'imagerie haute resolution
US7260274B2 (en) 2000-12-01 2007-08-21 Imax Corporation Techniques and systems for developing high-resolution imagery
EP1354292A4 (fr) * 2000-12-01 2009-04-22 Imax Corp Techniques et systemes pour la mise au point d'imagerie haute resolution
US7856055B2 (en) 2002-03-13 2010-12-21 Imax Corporation Systems and methods for digitally re-mastering or otherwise modifying motion pictures or other image sequences data
JP2009020879A (ja) * 2002-03-25 2009-01-29 Trustees Of Columbia Univ In The City Of New York データ品位を向上する方法及びシステム
US8000521B2 (en) 2004-06-25 2011-08-16 Masataka Kira Stereoscopic image generating method and apparatus
KR101937673B1 (ko) 2012-09-21 2019-01-14 삼성전자주식회사 3ｄ 디스플레이에 대한 ｊｎｄｄ 모델을 생성, 상기 ｊｎｄｄ 모델을 이용하여 깊이 영상을 개선하는 방법 및 시스템
US9870602B2 (en) 2013-12-06 2018-01-16 Huawei Device (Dongguan) Co., Ltd. Method and apparatus for fusing a first image and a second image
EP3068124A4 (fr) * 2013-12-06 2017-01-04 Huawei Device Co., Ltd. Procédé, dispositif et terminal de traitement d'image
WO2016097470A1 (fr) * 2014-12-15 2016-06-23 Nokia Technologies Oy Système multi-caméra comprenant des caméras étalonnées différemment
WO2017121058A1 (fr) * 2016-01-13 2017-07-20 南京大学 Système d'acquisition d'informations entièrement optiques
CN109155854A (zh) * 2016-05-27 2019-01-04 松下电器（美国）知识产权公司 编码装置、解码装置、编码方法及解码方法
CN109155854B (zh) * 2016-05-27 2022-06-07 松下电器（美国）知识产权公司 编码装置、解码装置、编码方法及解码方法
CN117036987A (zh) * 2023-10-10 2023-11-10 武汉大学 一种基于小波域交叉配对的遥感影像时空融合方法及系统
CN117036987B (zh) * 2023-10-10 2023-12-08 武汉大学 一种基于小波域交叉配对的遥感影像时空融合方法及系统

Also Published As

Publication number Publication date
JP2002524937A (ja) 2002-08-06
EP1110414A1 (fr) 2001-06-27
WO2000013423A9 (fr) 2000-08-17
CA2341886A1 (fr) 2000-03-09
WO2000013423A8 (fr) 2000-05-25

Similar Documents

Publication Publication Date Title
US6269175B1 (en) Method and apparatus for enhancing regions of aligned images using flow estimation
EP1418766A2 (fr) Verfahren und Gerät zur Verarbeitung von Bildern
WO2000013423A1 (fr) Procede et dispositif d&#39;imagerie de synthese haute resolution utilisant une camera haute resolution et une camera a resolution plus faible
CA2430591C (fr) Techniques et systemes pour la mise au point d&#39;imagerie haute resolution
Ndjiki-Nya et al. Depth image-based rendering with advanced texture synthesis for 3-D video
US9398215B2 (en) Stereoscopic panoramas
JP2019502295A (ja) アーチファクトを意識したビュー合成のための補助データ
KR20110093829A (ko) 깊이 맵을 생성하기 위한 방법 및 디바이스
KR20000023784A (ko) 모자이크 이미지 구성 방법 및 장치
WO2007083204A2 (fr) Procédé pour l&#39;estimation de champ de flux optique mettant en oeuvre le filtrage adaptatif
US10586345B2 (en) Method for estimating aggregation results for generating three dimensional images
EP2259599A1 (fr) Procédé et agencement pour le traitement d&#39;une image stéréo à partir d&#39;une vue tridimensionnelle
Knorr et al. Stereoscopic 3D from 2D video with super-resolution capability
Wang et al. Improved super-resolution reconstruction from video
Al Ismaeil et al. Dynamic super resolution of depth sequences with non-rigid motions
CA2463162C (fr) Procede et dispositif de traitement d&#39;images
Szeliski et al. Dense motion estimation
CN113139999B (zh) 一种透明物体单视图多尺度深度估计方法和系统
Lukac Image Resizing Solutions for Single-Sensor Digital Cameras
Jin Depth-Map-Assisted Texture and Depth Map Super-Resolution
Zakhor et al. Supplement to Exploiting Chaos in Oversampled A/D Converters (FY 94)
Zakhor REPORT (DOCUMENTATION PAGE AFRL-SR-BL-TR-98-> d

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): CA JP

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE

121 Ep: the epo has been informed by wipo that ep was designated in this application
AK Designated states

Kind code of ref document: C1

Designated state(s): CA JP

AL Designated countries for regional patents

Kind code of ref document: C1

Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE

CFP Corrected version of a pamphlet front page
CR1 Correction of entry in section i

Free format text: PAT. BUL. 10/2000 UNDER (30) REPLACE "NOT FURNISHED" BY "09/384396"

DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
AK Designated states

Kind code of ref document: C2

Designated state(s): CA JP

AL Designated countries for regional patents

Kind code of ref document: C2

Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE

COP Corrected version of pamphlet

Free format text: PAGES 1/4-4/4, DRAWINGS, REPLACED BY NEW PAGES 1/4-4/4; DUE TO LATE TRANSMITTAL BY THE RECEIVING OFFICE

ENP Entry into the national phase

Ref document number: 2341886

Country of ref document: CA

Ref country code: CA

Ref document number: 2341886

Kind code of ref document: A

Format of ref document f/p: F

ENP Entry into the national phase

Ref country code: JP

Ref document number: 2000 568261

Kind code of ref document: A

Format of ref document f/p: F

WWE Wipo information: entry into national phase

Ref document number: 1999946672

Country of ref document: EP

WWP Wipo information: published in national office

Ref document number: 1999946672

Country of ref document: EP

WWW Wipo information: withdrawn in national office

Ref document number: 1999946672

Country of ref document: EP