US20100111444A1 - Method and system for fast dense stereoscopic ranging - Google Patents

Method and system for fast dense stereoscopic ranging Download PDF

Info

Publication number
US20100111444A1
US20100111444A1 US12/451,064 US45106408A US2010111444A1 US 20100111444 A1 US20100111444 A1 US 20100111444A1 US 45106408 A US45106408 A US 45106408A US 2010111444 A1 US2010111444 A1 US 2010111444A1
Authority
US
United States
Prior art keywords
pixel
reference frame
frame
method
pair
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
US12/451,064
Other versions
US8467628B2 (en
Inventor
Thayne R. Coffman
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
21 CT Inc
Original Assignee
21 CT Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to US90795507P priority Critical
Application filed by 21 CT Inc filed Critical 21 CT Inc
Priority to PCT/US2008/005271 priority patent/WO2009023044A2/en
Priority to US12/451,064 priority patent/US8467628B2/en
Assigned to 21 CT, INC. reassignment 21 CT, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: COFFMAN, THAYNE R.
Publication of US20100111444A1 publication Critical patent/US20100111444A1/en
Publication of US8467628B2 publication Critical patent/US8467628B2/en
Application granted granted Critical
Application status is Active legal-status Critical
Adjusted expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/50Depth or shape recovery
    • G06T7/55Depth or shape recovery from multiple images
    • G06T7/593Depth or shape recovery from multiple images from stereo images

Abstract

A stochastic method and system for fast stereoscopic ranging includes selecting a pair of images for stereo processing, in which the pair of images are a frame pair and one of the image is a reference frame, seeding estimated values for a range metric at each pixel of the reference frame, initializing one or more search stage constraints, stochastically computing local influence for each valid pixel in the reference frame, aggregating local influences for each valid pixel in the reference frame, refining the estimated values for the range metric at each valid pixel in the reference frame based on the aggregated local influence, and post-processing range metric data. A valid pixel is a pixel in the reference frame that has a corresponding pixel in the other frame of the frame pair. The method repeats n iterations of the stochastically computing through the post-processing.

Description

    GOVERNMENT LICENSE RIGHTS
  • This invention was made with Government support under Contract Nos. FA8651-04-C-0233 and FA8651-05-C-0117 awarded by the United States Air Force Research Laboratory. The Government has certain rights in the invention.
  • BACKGROUND
  • Since the early 1970's, estimating the three-dimensional (3D) structure of a scene from two-dimensional (2D) imagery has been one of the most actively researched areas in the fields of digital image processing and computer vision. Thirty years of active research indicates the difficulty of developing efficient computational stereo techniques that are able to reconstruct dense scene structure estimates from stereo or monocular imagery.
  • The first step in bounding the problem is to define which sensors are considered imaging or visual sensors that generate the “visual images” for input. Typically sensors are categorized as active or passive. Active sensors include radar, synthetic aperture radar, ladar, sonar, and sonograms, which recover 3D information directly by sending out energy and analyzing the timing and/or content of reflections or returns. Also considered active are structured light sensors, which actively transmit a known lighting pattern to illuminate the target scene and then analyze images of the augmented scene. Active sensors generate depth estimates directly and thus do not need to estimate depth. Active sensors stand in contrast with passive staring visual sensors that analyze incoming energy that they did not generate. The common visible-light camera and video camera are passive visual sensors, as are electro-optic (EO) sensors that operate on other wavelengths like infrared, ultraviolet, multispectral, or hyperspectral.
  • Even within the more constrained realm of passive visual sensors, a wide variety of sensor configurations have been explored. Some approaches use a single camera or viewpoint (monocular), while others use two or more synchronized cameras capturing images from different viewpoints (binocular). Some approaches use video sequences (again, monocular, binocular, or even trinocular) with incremental changes in camera position between frames, and others operate on sets of a few images captured from widely varying viewpoints.
  • A number of different approaches can be followed to extract information on 3D structure from one or more images of a scene. “Shape from focus” techniques estimate depth by varying a camera's focal length or other intrinsic parameters, and identifying which parts of the image are sharply in focus at which set of parameters. “Shape from shading” techniques analyze changes in image intensity over space in a single image to infer the gradient of is the surface being imaged. “Semantic information” can also be used—if the real-world size and geometry of an Abrams M-1 tank is known and an Abrams M-1 tank is recognized in an image, the known size and appearance of its projection can be used to infer its distance and pose relative to the camera. Finally, direct per-pixel depth estimates can be extracted by using “structure from stereo” and “structure from motion” techniques, collectively known as computational stereo techniques. “Structure from stereo” refers to approaches based on two or more cameras, and “structure from motion” refers to approaches that use a single camera and the motion of that camera relative to the scene to simulate the existence of two cameras. The final output of these passive structure recovery systems is almost always depth (or range).
  • Computational stereo approaches generate depth estimates at some set of locations (or directions) relative to a reference frame. For two-camera approaches, these estimates are often given relative to the first camera's coordinate system. Sparse reconstruction systems generate depth estimates at a relatively small subset of possible locations, where dense reconstruction systems attempt to generate estimates for most or all pixels in the imagery.
  • Computational stereo techniques estimate a range metric such as depth by determining corresponding pixels in two images that show the same entity (scene object, element, location or point) in the 3D scene. Given a pair of corresponding pixels and knowledge of the relative position and orientation of the cameras, depth can be estimated by triangulation to find the intersecting point of the two camera rays. Once depth estimates are computed, knowledge of intrinsic and extrinsic camera parameters for the input image frame is used to compute equivalent 3D positions in an absolute reference frame (e.g., global positioning system (GPS) coordinates), thereby producing, for example, a 3D point cloud for each frame of imagery, which can be converted into surface models for further analysis using volumetric tools.
  • While it is “depth” which provides the intuitive difference between a 2D and a 3D image, it is not necessary to measure or estimate depth directly. “Disparity” is another range metric that is analytically equivalent to depth when other parameters are known. Disparity refers, generally, to the difference in pixel locations (i.e., row and column positions) between a pixel in one image and the corresponding pixel in another image. More precisely, a disparity vector L(i,j) stores the difference in pixel indices between matching pixels in image IA and image IB. If pixel IA(10,20) matches pixel IB(15,21), then the disparity is L(10,20)=(15,21)−(10,20)=(5,1), assuming L is computed relative to reference frame IA. Zero disparity means that pixel IA(m,n) corresponds to pixel IB(m,n), so L(m,n)=(0,0). If camera position and orientation are known for two frames being processed, then quantities such as correspondences, disparity, and depth hold equivalent information: depth can be calculated from disparity by triangulation.
  • A disparity vector field stores a disparity vector at each pixel, and thus tells how to find the match (or correspondences) for each pixel in the two images. When intrinsic and extrinsic camera parameters are known, triangulation converts those disparity estimates into depth estimates and thus 3D positions relative to the camera's frame of reference.
  • An important problem in dense computational stereo is to determine the correspondences between all the pixels in the two (or more) images being analyzed. This computation, which at its root is based on a measure of local match quality between pixels, remains a challenge, and accounts for the majority of complexity and runtime in computational stereo approaches.
  • Academia and industry have provided many advances in automated stereo reconstruction, but the domain still lacks a general solution that is robust and deployable in real-world scenarios. A number of facets of potential general solutions remain open research problems. Runtime and efficiency continue to be challenges, as well as finding match quality metrics that are robust to low-quality imagery or changing scene conditions. Robustness to camera path changes and scene orientation are also issues.
  • Calibrated monocular aerial modeling is an application that has received somewhat less attention than other areas in computational stereo, and it lacks a generally applicable solution. In these applications, the camera typically follows an aerial platform's known but independently controlled path, with position and orientation changing incrementally between frames. A standard stereo geometry is not available and stereo groupings must be selected from within a set of buffered frames. Intrinsic and extrinsic camera parameters are typically known to a high degree of accuracy. Unlike some other applications, expected characteristics include large absolute ranges to the scene (hundreds or thousands of meters), large absolute disparities (tens or hundreds of pixels), and large disparity search ranges. Approaches encounter complex and uncontrolled outdoor scenes that may contain moving objects and are imaged under uncontrolled outdoor lighting. Images may also contain other various artifacts.
  • Reliable solutions in these areas would enable a wide variety of applications in the commercial, military, and government domains. Rapid passive modeling of urban or rural areas is valuable in itself for virtual training and virtual tourism, but that capability also enables improved tracking, surveillance, and change detection, supports disaster response, and facilitates more robust autonomous systems through visually-aided navigation, object recognition, and other follow-on processing.
  • SUMMARY
  • Embodiments described herein overcome the disadvantages described above. This advantage and others are achieved by a stochastic method for fast stereoscopic ranging. The stochastic method includes selecting a pair of images for stereo processing, in which the pair of images are a frame pair and one of the image is a reference frame, seeding estimated values for a range metric at each pixel of the reference frame, initializing one or more search stage constraints, stochastically computing local influence for each valid pixel in the reference frame, aggregating local influences for each valid pixel in the reference frame, refining the estimated values for the range metric at each valid pixel in the reference frame based on the aggregated local influence, and post-processing range metric data. A valid pixel is a pixel in the reference frame that has a corresponding pixel in the other frame of the frame pair. The method repeats n iterations of the stochastically computing through the post-processing. The value of n is selected in the initializing.
  • These advantages and others are also achieved by a computer readable medium including instructions for performing a stochastic method for fast stereoscopic ranging. The instructions perform the method by selecting a pair of images for stereo processing, in which the pair of images are a frame pair and one of the image is a reference frame, seeding estimated values for a range metric at each pixel of the reference frame, initializing one or more search stage constraints, stochastically computing local influence for each valid pixel in the reference frame, aggregating local influences for each valid pixel in the reference frame, refining the estimated values for the range metric at each valid pixel in the reference frame based on the aggregated local influence, and post-processing range metric data. A valid pixel is a pixel in the reference frame that has a corresponding pixel in the other frame of the frame pair. The instructions repeat n iterations of the stochastically computing through the post-processing. The value of n is selected in the initializing.
  • These advantages and others are also achieved by a system for stochastic fast stereoscopic ranging. The system includes a visual sensor capable of taking images and a computing device coupled to the visual sensor. The computing device includes a memory. The memory includes instructions for performing a stochastic method for fast stereoscopic ranging by selecting a pair of images for stereo processing, in which the pair of images are a frame pair and one of the image is a reference frame, seeding estimated values for a range metric at each pixel of the reference frame, initializing one or more search stage constraints, stochastically computing local influence for each valid pixel in the reference frame, aggregating local influences for each valid pixel in the reference frame, refining the estimated values for the range metric at each valid pixel in the reference frame based on the aggregated local influence, and post-processing range metric data. A valid pixel is a pixel in the reference frame that has a corresponding pixel in the other frame of the frame pair. The instructions repeat n iterations of the stochastically computing through the post-processing. The value of n is selected in the initializing.
  • These advantages and others are also achieved by a method for initializing depth estimates in an iterative process for stereoscopic ranging. The method includes projecting a first depth estimate for each one of a plurality of valid pixels in an image, the image having an implicit camera origin, generating a 3D representation of the plurality of valid pixels using the first depth estimate, relocating the implicit camera origin, reprojecting the 3D representation of the plurality of valid pixels to the relocated implicit camera origin, using Z-buffering to render a second depth estimate for each one of the plurality of valid pixels according to the relocated implicit camera origin, and using the second depth estimate for each of the plurality of valid pixels as an initial value in a subsequent iteration of a process for stereoscopic ranging.
  • DESCRIPTION OF THE DRAWINGS
  • The detailed description will refer to the following drawings, wherein:
  • FIG. 1 is a flowchart illustrating an exemplary method for fast stereoscopic ranging.
  • FIG. 2 a illustrates a standard two-camera geometry, and FIG. 2 b illustrates the epipolar constraint.
  • FIGS. 3 a-3 c are images illustrating example images masked for artifacts and image rectification.
  • FIG. 4 a illustrates an exemplary method of initializing depth estimates in an iterative process for stereoscopic ranging.
  • FIGS. 4 b-4 c are images that illustrate seeding depth estimates.
  • FIG. 5 is a flowchart illustrating an exemplary method for computing local influence based on direct estimates of depth.
  • FIGS. 6 a through 6 d show a series of image components illustrating a region of random depth perturbations, resulting changes in match quality, resulting pixel-wise influences, and aggregated influences.
  • FIG. 7 is a flowchart illustrating the steps of an exemplary method of computing local influence based on direct estimates of disparity.
  • FIG. 8 illustrates an exemplary system for fast dense stereoscopic ranging.
  • DETAILED DESCRIPTION
  • Prior research has identified a number of related techniques. Cooperative techniques divide the complex correspondence matching problem into identical sub-problems at each pixel, and transfer information between the sub-problem solutions to achieve a global solution that is constrained by assumptions of continuity and uniqueness among the sub-problem solutions.
  • Stochastic techniques such as simulated annealing and microcanonical annealing search the solution space in a nondeterministic manner. Multi-view stereo systems process a sequence of many images in non-standard stereo geometries simultaneously. Finally, there are other existing approaches targeting real-time operation, which tend to use parallel architectures or other custom architectures. The inspirations provided by these related techniques are combined in the embodiments described herein in a new way to yield an approach with particular advantages. Existing cooperative approaches repeatedly apply local and non-local effects to iterate towards their solution. However, the cooperative approaches typically compute local match quality exhaustively (with respect to integer-valued disparities), they can consume considerable amounts of memory, and their memory and runtime can change with stereo geometry and scene geometry. Simulated annealing and other stochastic search approaches avoid an exhaustive sampling of potential local solutions, but still often quantize the disparity search space and can also be slow to converge. The embodiments described herein reduce computation by using an approach that is both cooperative and stochastic, and has advantages from both.
  • Cooperative and stochastic techniques are not obviously compatible. Cooperative approaches universally assume an exhaustive sampling of the solution space on which to base their calculations, but stochastic approaches intentionally avoid this exhaustive sampling. Stochastic approaches may consider non-local relationships in the solution when determining whether or not to explore a portion of the search space, but they do not directly incorporate the effects of one local sample of the search space into neighboring solutions. Using a combined cooperative and stochastic approach to gain the benefits of both, therefore, represents a non-obvious extension of the state of the art.
  • Existing multi-view systems are designed for operation on many images, and can be applied to monocular video imagery, but they typically process all available frames collectively in a batch, and thus are of little help for applications that require a more incremental and streaming solution. Existing is real-time stereo systems often make assumptions that are unattractive in some domains, for example assuming the use of two or more cameras in a fixed geometry (which would provide an insufficient stereo baseline on all but the largest of aerial platforms) or specialized hardware. By exploiting the piecewise continuity and continuity of matching likelihood constraints and by initializing each frame's estimate based on previous results, the embodiments described herein avoid exploration of large portions of the search space and allow range estimates to converge over multiple frame pairs, reducing runtime and providing an incremental structural estimate.
  • The embodiments described herein present a general approach that is easily applied to a variety of specific applications. One application area that is particularly well-suited to exploit these advances is the incremental dense reconstruction of man-made structures (buildings, roads, stationary vehicles, etc.) and natural backgrounds (e.g., hills, waterways, and trees) from sequences of monocular aerial video accompanied by intrinsic and extrinsic camera parameters. In more common applications (e.g., robotic navigation), the approach suffers no loss.
  • FIG. 1 is a flowchart illustrating an exemplary method 10 for fast stereoscopic ranging.
  • In block 105, a pair of images (or frames) IA(i,j) and IB(i,j) (where i=1, 2, . . . , imax and j=1, 2, . . . , jmax are discrete pixel indices) are selected for stereo processing from a video stream. Given this image pair, the more recent of the two, IA, is defined to be the reference frame. These stereo pairs are formed by selecting non-adjacent frames in the input stream in a way that optimizes accuracy and insulates the algorithm from changes in platform motion. Using a technique described in R. Vidal and J. Oliensis, Structure form Planar Motions with Small Baselines, Proc. European Conference on Computer Vision, pp. 383-398 (2002), frames are paired to maintain a target ratio τ0 between the stereo baseline T and the expected minimum depth to the scene
  • min i , j { D ( i , j ) } ,
  • defined by
  • τ 0 = T min i , j { D ( i , j ) } . ( 1 )
  • This target baseline ratio provides resilience from changes in platform speed or direction, making it possible to tune other parameters to a more consistent stereo geometry. The stereo baseline T is taken as the Euclidean distance between the two camera origins (without considering camera orientation or other more complex geometric factors that also affect the mapping from depth to disparity).
  • FIG. 2 a illustrates a general geometry 210 formed between two cameras, A and B, referred to as the left and right camera respectively. The camera origins OA and OB (215, 220) are shown, along with the principal camera axes Z+ A and Z+ B (217, 222), and image planes IA and IB (230, 240). The embodiments described herein apply the projective camera model, in which a point P=(X,Y,Z)′ (225) in 3D Euclidean space is projected onto a point uA=(xA,yA)′ (235) in image IA through the relationship
  • ( x A , y A ) = ( X Z , Y Z ) ,
  • and similarly for camera B and image IB (point 245) This projection is typically modeled as a linear operator in a 3D projective geometry through the use of homogeneous coordinates Ph=(x, y, z, w)′ which are unique only up to the scaling factor w.
  • Within this geometry, given noise-free values for the camera positions, orientations, and the projections of the point onto the two image planes, the location of P in 3D Euclidean space can be recovered by computing the intersection of the two camera rays {right arrow over (v)}A and {right arrow over (v)}B.
  • FIG. 2 b illustrates a fundamental characteristic 250 of this geometry called the epipolar constraint. The epipolar constraint is an important constraint in computational stereo because it removes a full dimension from the search space for each pixel's correspondence. Given two camera origins OA and OB (255, 260) and the projection of a point P (265) in 3D space onto one of their image planes, the projection of P onto the other image plane is constrained to lie on a specific line. The epipolar constraint states that the projections uA and uB (275, 285) of P onto image planes IA and IB (270, 280) are bound by the constraint u′AFuB=0, where F is the 3×3 fundamental matrix, which can be computed directly from knowledge of the camera positions (OA and OB), camera orientations, and intrinsic camera parameters such as field of view, resolution, et cetera. Further details regarding the fundamental matrix and the geometries and epipolar constraint discussed herein can be found in Hartley & Zisserman, Multiple View Geometry in Computer Vision, 2d ed. (Cambridge University Press 2003), which is incorporated here by reference.
  • The epipolar constraint restricts the projections of any point P such that independent of its depth or 3D position, given either uA or uB, the other projection must like on the corresponding epipolar line. Given uA, uB must lie on the line LB=u′AF, and given uB, uA must lie on the line LA=FuB. The projections of one camera's origin onto the other's image plane is called an epipole (EA and EB) (272, 282), and all epipolar lines go through the epipole in their image plane.
  • Most two-camera computational stereo systems assume the use of a standard stereo geometry. The standard geometry requires that the rotations RA and RB of the two cameras with respect to world coordinates are identical, so there is no relative rotation between the two camera reference frames. This means the X+ axes of the two cameras are parallel, as are their Y+ and Z+ axes. Further, the standard geometry requires that the stereo baseline T or (OB−OA) is parallel to the X+ axis and is positive. As a result, in the standard stereo geometry, all epipolar lines coincide with the horizontal scanlines of the images.
  • Computational stereo estimates depth by determining corresponding pixels in two images that show the same point in the 3D scene and exploiting the epipolar geometry to compute depth. Given a pair of corresponding pixels and knowledge of the relative position and orientation of the cameras, depth can be estimated by triangulation to find the intersecting point of the two camera rays. Computational stereo approaches are based on triangulation of corresponding pixels, features, or regions within an epipolar geometry between two cameras. Triangulation is straightforward under certain stereo geometries and in the absence of errors in the correspondence estimate.
  • The standard stereo geometry is generally only physically achievable by using two cameras mounted on a common chassis. Single-camera approaches must achieve this geometry or another desirable stereo geometry via rectification. While rectification is not required in a computational stereo system, it can reduce overall computational complexity and/or improve the conditioning of the problem, and it is preferable for use in a single-camera approach. Rectification can improve conditioning by facilitating additional constraints on the solution or making the solution more robust to noise or error. To perform rectification, the epipolar geometry described above must be either known or estimated.
  • In a preferred embodiment, rectification is used to eliminate only the relative rotations between the two cameras, without transforming to a completely standard geometry, thereby creating the rectified paired image IBR. This can be accomplished through standard coordinate transforms for rotation, given knowledge of the relative orientation between the two cameras. As a result of this type of rectification, the direction of disparity for matching pixels becomes independent of depth, and the epipolar line for a left image pixel IL(r,c) will always pass through the right image pixel IBR(r,c). As another a result, scene elements at infinite depth exhibit zero disparity. This type of rectification can reduce computational complexity during the correspondence search. Other potential rectification strategies include two-frame projective rectification, which applies projective transforms to both images in a non-standard stereo geometry in order to convert them into a standard stereo geometry, and a vergent stereo geometry with non-parallel principal camera axes that intersect at a point of interest in the scene.
  • In an embodiment, a scalar floating-point depth value {circumflex over (D)}(i,j) is directly estimated at each pixel (i,j) in the domain of IA. In the pre-processing block 110 of an embodiment shown in FIG. 1, unit vectors {right arrow over (e)}(i,j) are pre-computed for each pixel in IA in the epipolar direction for the geometry between IA and IBR. The length of the disparity vector (k) for any estimated depth can be easily calculated, which makes it possible to convert between estimated depth {circumflex over (D)}(i,j) and an equivalent estimated disparity {circumflex over ({right arrow over (L)}(i,j)=k{right arrow over (e)}(i,j) very quickly, in order to efficiently compute match quality samples Q(IA,IB;i,j,{circumflex over (D)}(i,j)) for any particular choice of local match quality metric Q(•) and any particular {circumflex over (D)}(i,j). This process also implicitly enforces the constraint that correspondences lie on the epipolar lines.
  • Following rectification in block 110 in FIG. 1, masks are constructed to identify pixels in IA(i,j) and IBR(i,j) that have been invalidated by any of a number of constraints, including: imagery artifacts, the constraint that IBR have the same pixel domain as IB (i=1, 2, . . . , imax and j=1, 2, . . . jmax), that both pixels in a correspondence lie within the paired images, and also any known or assumed bounds on scene structure.
  • FIGS. 3 a through 3 c illustrate the results of pre-processing and rectification on an exemplary image. FIG. 3 a shows a reference image IA, masked for image artifacts. FIG. 3 b shows paired image IB masked for image artifacts. FIG. 3 c shows rectified paired image IBR, masked for image artifacts and rectification as described in the preceding paragraph.
  • Conservative application-specific assumptions about scene structure can considerably improve performance. Scene structure constraints and changes in stereo geometry cause the valid range of potential depths (and disparities) to be different for each frame pair, and also different at each pixel in a given frame pair. As an example, aerial modeling applications can benefit from loose assumptions on the minimum and maximum elevation in the scene to establish minimum and maximum depths, {circumflex over (D)}min(i,j) and {circumflex over (D)}max(i,j), at each pixel. Other applications with different categories of scenes, camera orientations, or camera motion will benefit from analogous but different constraints. Regardless of the specific scene structure assumptions, the depth and/or disparity ranges being estimated by the approach can vary frame-by-frame or pixel-by-pixel with no effect on runtime or memory consumption.
  • In block 110 (see FIG. 1), the images are preprocessed and certain data points are initialized. In block 115 of FIG. 1, each pixel (i,j) in the domain of IA is directly initialized (seeded) with an estimated floating point depth value. For each new frame pair, depth estimation at each pixel is initialized based on the final estimated value computed for the corresponding pixel in frame IA of the previous frame pair. Seeding allows the depth estimates to converge over multiple frame pairs in a style of recursive estimation. On each frame but the first, existing estimates are refined instead of generating entirely new estimates. As a result, the number of search iterations and runtime for each frame are reduced even further, and search schedules with smaller perturbations and aggregation neighborhoods can be used to retain detail in the estimates.
  • Depth estimates can be seeded very aggressively since the camera position and orientation are assumed known and will differ little between adjacent frames. For each new frame pair, the most recent depth estimates are re-projected to the new reference frame and adjusted for changes in camera origin. Preferably this is done using a Z-buffering algorithm to address occlusions or de-occlusions introduced by camera motion. The vast majority of pixels can be initialized with this process. Small gaps in coverage are filled using nearest-neighbor interpolation with a small structuring element. Where there are no initial values with which to seed a pixel (such as in the initial frames when the algorithm begins), the pixel can be initialized using an appropriate application-specific heuristic (e.g., a default elevation). Aggressively seeding the stochastic cooperative search lets the estimates converge over multiple stereo image pairs, so the number of iterations for each frame pair can be reduced further, again reducing the number of evaluations of match quality and reducing runtime and improving accuracy.
  • FIG. 4 a illustrates an exemplary method 40 of initializing depth estimates in an iterative process for stereoscopic ranging. In block 410 the most recent (or most current) depth estimates are selected. Preferably the most recent depth estimate is the final estimated value for the corresponding pixel in the reference frame I′A for the previous frame pair. In block 420 a 3D representation of the location for each pixel is generated using the most recent depth estimate. Exemplary 3D representations include a point cloud or arrays of 3-tuples representing coordinates of the pixel in 3D space. The previous reference frame I′A defines a camera origin, and in block 430 the camera origin is relocated to the origin of the current reference frame IA. One potential consequence of the movement of the camera is that objects in the 2D image may become occluded or de-occluded, and preferably the depth estimates can be adjusted to account for such occlusion or de-occlusion.
  • In block 440, the 3D representation for the scene location for each pixel is re-projected to its 2D location in reference frame IA via the projective camera model described above. In block 450 a Z-buffering algorithm is used to generate a new set of depth estimates at each pixel in the current reference frame IA from the depths of the 3D scene locations and their 2D locations in reference frame IA. Z-buffering is a technique from the domain of computer graphics that is used to generate realistic images of scenes that contain many objects, some of which occlude others and thus prevent them from being visible. Z-buffering conditionally draws the color of each scene element pixel-by-pixel to the image, displaying it at a given pixel only if it is located at a range less than that of the currently-drawn scene element at that pixel. The approach is named for the record of the ranges of currently-drawn scene elements at each pixel, which is called the “Z buffer”. These ranges are not drawn to the screen in existing applications of Z-buffering. The Z-buffering algorithm is applied in a novel way to “draw” the seed values for a given frame pair such that new occlusions and de-occlusions that are generated by the camera's motion between adjacent frames are reflected accurately in the seed values. This is conceptually the equivalent of “drawing” the Z buffer to the screen directly.
  • The depth estimates for some pixels in the new 3D representation may have been changed as a result of the application of Z-buffering. These new depth estimates are used, in block 450 of FIG. 4 a, to seed the initial depth estimate values in the reference frame IA in block 115 of FIG. 1,
  • FIGS. 4 b and 4 c illustrate the seeding process. FIG. 4 b shows example depth estimates from a previous frame pair that will be used for seeding via Z-buffering, where example depth values in reference frame I′A are pseudocolored by depth (larger depths are colored more brightly). FIG. 4 c shows those depths re-projected to a new reference frame IA through Z-buffering. Most but not all pixels in reference frame IA are initialized by selecting and adjusting the appropriate estimate from frame I′A. The examples I′A and IA in FIGS. 4 b and 4 c are 10 frames distant for better visualization of the is parallax effects that cause gaps in the set of pixels that can be initialized through Z-buffering. Seeding via Z-buffering is preferably applied to adjacent reference frames so these gaps will be smaller or nonexistent.
  • After the pre-processing and seeding of initial estimates comes a novel combination of stochastic and cooperative search techniques used to explore the space of potential depth estimates. In an embodiment the method proceeds through a finite number of search stages, preferably four.
  • Block 120 is the beginning of the loop that iterates through the different search stages, beginning with stage 1. After the process has completed iterating through all search stages, control passes to block 125 (QUIT). At this point there is a refined depth estimate for each valid pixel in the reference frame.
  • Optionally, the estimated depth value or other image data can be visually displayed on a display device such as a monitor. An exemplary method of displaying depth values is a point cloud, i.e., a set of points in 3D space (or simulated 3D space) in some coordinate frame. Another exemplary method is a depth image, where depth estimates are displayed at their corresponding pixel location in the camera's reference frame and are pseudocolored by the estimated depth. Yet another embodiment is a surface model, in which structures in the scene are represented as a collection of piecewise-smooth surfaces.
  • For each search stage the method proceeds through N iterations, where N preferably is different in each stage and is selected to allow estimates to converge in each stage. In block 130 a value for N is determined. Also initialized in block 130 are other constraints that vary depending on the search stage, including preferably maximum depth perturbation magnitude δmax and aggregation neighborhood W, described in more detail below.
  • Block 135 of FIG. 1 is the first block of each iteration of the search stage. If the N iterations have been completed for the search stage, control returns to block 120 to begin the next search stage. If N iterations have not yet been completed, the process performs the next iteration, beginning with block 140.
  • In an embodiment, the method stores a range metric for each pixel in the form of an estimated point depth value. In block 140, a local influence is computed for each pixel. In an embodiment a random (or stochastic) perturbation is injected into each pixel's depth estimate. The local match quality corresponding to the perturbed estimate is computed, as well as the influence that the new sample of the local match quality function should exert on the local region (which may be either towards or away from the perturbed depth estimate). In block 150 local influences are aggregated in a cooperative approach to eliminate noise and extract consistent trends in the match quality values, driving the new set of depth estimates towards the correct depth values. An embodiment of the approach uses relatively straightforward iterations of random perturbation and smoothing with a novel non-linear formulation of per-pixel and aggregated influences that pull the depth estimates towards a better solution without evaluating local stereo match quality at every possible depth.
  • Using a floating point estimated depth value as the range metric in the method 10 is in contrast with other approaches that estimate discrete disparity values for the reference frame and then post-process to recover sub-pixel disparity estimates and finally depths. Estimating floating-point depths directly is preferable in cases involving large depth or disparity ranges, or ranges that vary significantly pixel-to-pixel. It also avoids quantizing the estimated values and also facilitates the use of local match quality metrics defined on continuous domains.
  • FIG. 5 shows the steps of computing a local influence (i.e., block 140 in FIG. 1) for each valid pixel in image IA. The “valid” pixels are those pixels in image IA that have corresponding pixels in image IBR, meaning those pixels which have not been invalidated for any reason. Possible reasons for invalidation of corresponding pixels in image IBR include image artifacts in IBR and disparity values that cause the corresponding pixel to lie outside of the rectified image.
  • Block 505 in FIG. 5 computes match quality at pixel (i,j). Generally speaking, match quality is a local metric that measures how closely a pixel location in image IA matches a pixel in image IBR. Match quality should be high when the two pixels are showing the same object, and low when the two pixels are showing different objects. The match quality function on a particular IA, I BR, pixel at (i,j), depth estimate and local match function Q ( ) is denoted:

  • q n(i,j)=Q(I A ,I BR ;i,j,{circumflex over (D)}(i,j))  (2)
  • Preferably the match quality function used is the “sum of absolute differences” (SAD), which is a simplified measure of correlation between the local regions surrounding the pixels IA and IBR. To compute the SAD value, a region is selected about the pixel of interest in frame IA preferably between 3×3 and 5×5 pixels; the similar-sized region surrounding the corresponding pixel in IBR is located; and the absolute value of the differences between each pixel in the region in IA and the corresponding pixel in the region in IBR are summed. Other match quality functions that can be used include the sum of squared differences, local correlation, local mutual information, and similarity of responses to oriented Gaussian filters.
  • In order to evaluate the match quality function, the depth estimates for each pixel must be converted to disparity values. Disparity typically is not used inside the match quality function Q, but the disparity information is required to identify which pixels in IA and IBR are to be compared with the match quality function.
  • In block 510 of FIG. 5, the depth estimate at pixel (i,j) is stochastically perturbed with additive noise

  • {tilde over (D)} n(i,j)={circumflex over (D)} n-1(i,j)+ΔD n (i,j).  (3)
  • On each iteration in an embodiment, the noise ΔD n (i,j) that is applied to each pixel's depth estimate is drawn from a uniform distribution subject to a set of pixel-specific constraints. In alternative embodiments, the noise may be drawn from other distributions, e.g., a normal distribution or an exponential distribution. The search schedule (described below) defines a maximum depth perturbation magnitude δmax for each iteration. This limit is preferably large for early iterations and gradually shrinks in later iterations. The limit δmax is defined on the normalized range [0,1] and is scaled by the depth search range for each pixel to establish the constraint

  • ΔD n (i,j)ε└−ΔD max (i,j),ΔD max (i,j)┘,  (4)

  • where

  • ΔD max (i,j)=δmax({circumflex over (D)} max(i,j)−{circumflex over (D)} min(i,j)).  (5)
  • Perturbations are also constrained such that the perturbed depth estimate {tilde over (D)}n(i,j) is guaranteed to lie within the pixel-specific depth estimate bounds:

  • ΔD n (i,j)ε└{circumflex over (D)}min(i,j)−{circumflex over (D)}n-1(i,j),{circumflex over (D)}max(i,j)−{circumflex over (D)}n-1(i,j)┘.  (6)
  • These constraints are introduced before sampling from the noise distribution, instead of sampling from the distribution and then clipping, to avoid sampling {circumflex over (D)}min(i,j) and {circumflex over (D)}max(i,j) more often than other depth values, which introduces a bias towards depth estimates on extreme ends of the search range.
  • In block 515 of FIG. 5, the induced local match quality {tilde over (q)}n(i,j) is evaluated according to

  • {tilde over (q)} n(i,j)=Q(I A ,I BR ;i,j,{tilde over (D)} n(i,j)),  (7)
  • The induced local match quality is the result of the match quality function applied at pixel (i,j) with perturbed depth estimate {tilde over (D)}n(i,j).
  • In block 520 the resulting per-pixel influences J*n(i,j) of the perturbed estimate are computed. Each new sample of local match quality has the potential to exert influence on the local solution. The sign and magnitude of J*n(i,j) are determined considering the change in local match quality

  • Δq n (i,j)={tilde over (q)} n(i,j)−Q(I a ,I BR ;i,j,{circumflex over (D)} n-1(i,j))  (8)
  • resulting from the random perturbation, as well as the sign and magnitude of the perturbation, as described further below.
  • For each pixel (i,j), per-pixel influence J*n is defined as
  • J n * = Δ D n ( w 1 + w 2 ) 2 ( 9 ) w 1 = max ( C 1 q ~ n - ( C 1 - 1 ) , 0 ) with perferably C 1 10 ( 10 ) w 2 = Δ q n max i , j Δ q n . ( 11 )
  • This definition of J*n(i,j) has proven effective. The maximum operation in (10) ensures that only samples with high match quality are given influence. The normalization in (17) ensures that w2 is bound from above by 1.0 and achieves that value for at least one pixel in each iteration. Thus this definition of J*n aligns it appropriately with the depth perturbation, selectively weights high-quality matches, preserves influence magnitudes when approaches zero, and prevents large influences from causing instability in the estimate. Preferably the per-pixel influence is calculated for each valid pixel in IA before proceeding to aggregation of the per-pixel values in block 150 in FIG. 1.
  • Returning to FIG. 1, in block 150 per-pixel influences J*n(i,j) are aggregated over a local support region W to form the aggregated influence Jn(i,j); and in block 160 the aggregated influence Jn(i,j) is used to refine the depth estimates of each pixel towards a better solution according to

  • {circumflex over (D)} n(i,j)={circumflex over (D)} n-1(i,j)+J n(i,j).  (12)
  • To aggregate influence, simple averaging over the local window W performs acceptably and can be implemented very efficiently. Anisotropic filtering approaches have also been applied to prevent loss of detail along boundaries. Averaging enforces a smoothness constraint while anisotropic filtering enforces a piecewise-smoothness constraint. Robust aggregation approaches, including order-statistic filtering or voting, could also be applied.
  • The aggregation neighborhood is preferably defined in terms of the average of the resolution in rows and columns in the images,
  • μ res = R + C 2 . ( 13 )
  • In block 170 of FIG. 1, the frame data is post-processed prior to beginning a new iteration (or new search stage). Preferably depth estimates are smoothed in each iteration over the same local support region, W, to help propagate influences through the solution more quickly.
  • Control returns to block 135 to begin a new iteration. Preferably blocks 150 and 160 are completed for all valid pixels in IA before beginning block 170.
  • Standard image processing algorithms such as smoothing and edge detection can be used on both 2D depth images in block 170 and with the per-pixel influences J*(i,j) in block 150, in order to remove noise in the 3D structure estimate or to otherwise enhance results. These operations can be applied more easily to depth and influence than to disparity because depth has a more direct relationship with scene structure than disparity (which is also heavily influenced by stereo geometry).
  • In contrast with the vast majority of approaches, the approach described herein avoids computing Q(•) exhaustively by stochastically sampling it and using local aggregation to extract consistent trends caused by the piecewise-continuous structure of the scene and the continuity of matching likelihood constraint. Even if it were desirable, it would be impossible to truly compute Q(•) exhaustively over all possible floating-point values. Instead, non-exhaustive and non-uniform sampling reduces the number of evaluations of Q(•) required to compute an accurate estimate, and it facilitates the use of match quality metrics defined on non-integer disparities.
  • Influence has a small set of general requirements. When an acceptable solution has not been reached, influence (both per-pixel and aggregated) should tend to have sign and magnitude such that adding it to {circumflex over (D)}n-1(i,j) moves {circumflex over (D)}ni,j) closer to the true solution. Aggregated influence Jn(i,j) must tend towards zero when an acceptable solution is reached. Consistent trends in Q(•) should be captured and inconsistent noise rejected by the influence formation and aggregation process.
  • FIGS. 6 a-6 d illustrate the progression of the method described herein. FIG. 6 a shows an example region of random depth perturbations ΔD n (i,j) (pseudo-colored in grayscale); FIG. 6 b shows corresponding changes in match quality Δq n (i,j); FIG. 6 c shows per-pixel influences J*n(i,j); and FIG. 6 d shows aggregated influences Jn(i,j). Pre-perturbation depth estimates {circumflex over (D)}n-1(i,j) are typically smooth, with smoothly-varying match qualities. Randomly perturbing the depth estimates generates a “noisy” {tilde over (q)}n(i,j) and Δq n (i,j). The role of per-pixel influence is to reflect the information gained by the new sample of the match quality function. Per-pixel influence are positive for perturbations that increase both match quality and the depth estimate, and negative for perturbations that increase match quality by decreasing the depth estimate. When perturbation decreases match quality, per-pixel influence is either zero or of the opposite sign of the perturbation.
  • By selectively weighting perturbations that improve the solution and aggregating over a local neighborhood, the approach refines the estimates towards a better solution. Randomness nearly ensures that some perturbations will increase depth and some will decrease depth. It is similarly very likely that some match quality values will increase due to the depth perturbation, and others will decrease. The role of influence aggregation is to extract from the noisy Δq n (i,j) consistent trends that reflect the scene's underlying structure.
  • In an alternative embodiment, the range metric which is locally perturbed is disparity estimates rather than depth estimates. Preferably the direction of each pixel's epipolar line is known through knowledge of camera positions and orientations. As a result, the embodiment can pre-compute unit vectors along the epipolar directions and represent the estimated correspondences by a scalar-valued disparity magnitude field D(r,c) (where (r,c) refer to (row,column)). Infinite depth results in zero disparity as a result of the preferred rectification scheme described earlier. The search is bound by conservative assumptions on the minimum and maximum elevation of elements in the scene. As a result, each pixel can have different disparity bounds.
  • Except for seeding estimated values and computing local influence for each valid pixel (blocks 115 and 140), the method of the alternative embodiment is the same as shown in FIG. 1, except using a scalar-valued disparity estimate instead of a depth estimate. The initial values are seeded with estimated disparity magnitude at each pixel. FIG. 7 illustrates the computation of local influence for each valid pixel in the alternative embodiment based on the disparity estimate Di(r,c). In block 755, match quality is computed using a match quality function qi(r,c,Di(r,c)). In block 760, the disparity estimate Di(r,c) is randomly perturbed, e.g., as described above, with an independent uniformly distributed random perturbation Δdi, where Δdi(r,c)ε[−δmaxmax]∩[−Di(r,c),Dmax(r,c)−Di(r,c)] at each pixel to generate {tilde over (D)}i(r,c)=Di(r,c)+Δdi(r,c). In block 765 the induced match quality of the perturbed disparities is generated, {tilde over (q)}i(r,c,{tilde over (D)}i(r,c)). In block 770 the local influence I*i(r,c) to be exerted by each new sample {tilde over (q)}i(r,c,{tilde over (D)}i(r,c)) is computed. The sign and magnitude of the influence are based on the sign and magnitude of the random perturbation, as well as its resulting effect on the match quality at that pixel. These pixel-wise influences are then aggregated over a local region W and applied to Di(r,c) to move it towards the correct solution as described in block 150 of FIG. 1.
  • Yet another embodiment uses normalized disparity (or “percent disparity”) estimates as the range metric. Use of normalized disparity value retains most of the advantages of estimating depth directly and can execute marginally faster. Instead of estimating disparity magnitude at each pixel, the minimum and maximum possible disparity at each pixel (which does not change in a given frame pair) are stored; and then a value between 0.0 and 1.0 is estimated for every pixel that is a percentage of the range between the minimum and maximum possible disparity values. Converting between the two forms of disparity is a straightforward calculation of:

  • D(m,n)=D min(m,n)+ND(m,n)*(D max(m,n)−D min(m,n)), and

  • ND(m,n)=(D(m,n)−D min(m,n))/(D max(m,n)−D min(m,n))
  • where D( ) is disparity, Dmin ( ) is minimum disparity, Dmax ( ) is maximum disparity, and ND( ) is normalized disparity (a.k.a. “percent disparity”). This alternative embodiment normalizes D(r,c) over the allowed bounds at each pixel, yielding a normalized disparity on the range [0.0,1.0]. This normalized disparity is also equal to a “percent elevation” over the assumed minimum and maximum elevation range.
  • The stochastic search of the embodiments described herein is regulated by a search schedule analogous to the annealing schedule in simulated annealing, or the training schedule in self-organizing feature map construction. The schedule defines values for the maximum depth perturbation magnitude δmax, aggregation neighborhood W, and number of iterations N for each search stage.
  • Search schedules preferably have a set of characteristics driven by their need to be robust while preserving detail in the scene. Maximum depth perturbation magnitude δmax is preferably specified as a percentage of the absolute depth search range (which varies for each pixel). Aggregation neighborhood size W is preferably specified as a fraction of image resolution. Schedules preferably tend to have few stages, with larger perturbations and aggregation neighborhoods in early stages and smaller values in later stages. Using a fixed number of iterations in each stage also allows explicit control of the tradeoff between accuracy and runtime.
  • An example of a preferable search schedule for all frame pairs but the first in is as follows:
  • Stage 1 Stage 2 Stage 3 Stage 4
    N 4 6 8 8
    W Square(μres/30) Square(μres/40) Square(μres/60) Square(μres/120)
    δmax 0.25 0.15 0.10 0.05

    Where the neighborhood W is defined in teems of a image processing function “Square” that takes as input a single length and generates a neighborhood function (also commonly called a structuring element) defining a square with each side of the specified length. A different schedule is preferably applied to the first frame pair in order to emphasize larger perturbations and aggregations.
  • FIG. 8 illustrates a high-level view of components of an exemplary system 80 for fast stereoscopic ranging, comprising one or more visual sensors 820, computing device 870 and optional platform 850
  • Also depicted in FIG. 8 is a three-dimensional scene 810. Three-dimensional scene 810 can include any scene containing three-dimensional objects. An exemplary and non-limiting list of examples includes: planetary and geographic features such as continents, oceans, islands, ice formations, volcanoes, forests and jungles, mountains and rivers; population areas such as cities or towns; ports, tunnels, airports, towers, bunkers, streets, highways, roads, bridges, houses, buildings, roads, and other man-made structures; ships, planes, rockets, cars, trucks and other vehicles; structures, people or objects in an internal space (e.g., a room, cave or body cavity) or external space (e.g., a plaza); underwater objects, including submarines or reefs; and facial and other identifying features of people.
  • Visual sensor 820 includes an sensor that can produce a 2D image of a 3D object and includes passive staring visual sensors such as, for example, the common visible-light camera and video camera and other sensors that capture images in the visible light spectrum, and electro-optic (EO) sensors that operate on other wavelengths like infrared, ultraviolet, multispectral, or hyperspectral. Visual sensor 820 can be adapted and configured to transmit live images in real time or to store images (such as on digital video tape, flash memory or other media) for non-real-time processing. Each visual sensor 820 will have associated intrinsic parameters (e.g., focal length, resolution, field of view, principal point, skew, radial distortion, et cetera) and extrinsic parameters (e.g., absolute position in terms of global positioning system (GPS), and orientation), which preferably are monitored and stored in system 80.
  • The system 80 includes one or more visual sensors 820. The methods described herein work by synthesizing information from two or more different (but related) 2-D images of the same three-dimensional scene 810. The two or more different images can be captured by a single (monocular) visual sensor which is displaced, relative to three-dimensional scene 810, in time and/or space, such as, for example, a single video camera mounted on an unmanned surveillance drone that captures a stream of video image frames of an object during a flyover. Preferably any visual sensors used as a monocular is visual sensor is adapted and configured to generate sequential 2-D image frames with incremental changes in viewpoints such as would be generated, for example, by a video camera. Alternatively the two or more different images can be captured by two or more different visual sensors that are displaced relative to each other and three dimensional scene 810 in time and/or space, such as, for example two separate video cameras pointed at the same object at a 45 degree offset. If two or more different visual sensors are used, preferably the visual sensors are aligned so as to avoid the need for rectification.
  • System 80 includes one or more computing devices 870 preferably implemented as a general-purpose computing device in the form of a computer comprising one or more processing units 872, a system memory 874 comprising computer storage media, a communications unit 876, optional storage device 878 and display device 879, and a system bus that couples the various system components including the system memory 874 to the processing units 872.
  • The system 80 optionally includes a platform 850. Platform 850 preferably comprises one or more computing devices 851 comprising one or more processing units 852, system memory 854 comprising computer storage media, communications unit 856, optional storage device 858 comprising computer storage media, and a system bus that couples the various system components including the system memory 854 to the processing units 852.
  • Alternatively computing devices 870 and 851 can be implemented as a special-purpose computer designed to implement the method for fast dense stereoscopic ranging described herein.
  • The selection and conglomeration of the components of computing device 870 and platform 850 is within the knowledge of the person of ordinary skill in the art. What follows is a non-limiting description of exemplary embodiments of the system components.
  • The computing devices 870 and 851 typically include a variety of computer readable media. Computer readable media can be any available media that can be accessed by computer and includes both volatile and nonvolatile media, removable and non-removable media. Computer readable media provide storage of software, computer readable instructions, data structures, program modules and other data for the computer, including an operating system, application programs, other program modules and program data. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media.
  • The phrase “computer storage media” is intended to include both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, USB drives, memory sticks, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, hard disks, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the computer.
  • The term “communication media” typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as satellite, microwave, acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer readable media.
  • Processing units 872 and 852 can include one or more conventional processors, configured in a single-core or multi-core arrangement. Alternatively, computing devices 870 and 851 can include multiple processors configured to operate in parallel; indeed, the method described herein is well-suited to parallel implementation because use of nonlocal methods (such as inhibition) can be avoided or minimized in embodiments.
  • System memories 874 and 854 preferably comprise RAM and/or other computer storage media coupled to processor(s) 872 and 852 and adapted and configured for storing the software instructions embodying the method described herein, the data used in the method, including the per-pixel estimated range metrics and other meta-data regarding each pixel of frame data, intrinsic and extrinsic parameters of visual sensors 820, and other information used in the system. Preferably there is sufficient memory to simultaneously store all estimated range metric estimations for at least two iterations of the method.
  • Communications modules 876 and 856 comprise communications media including but not limited to the hardware and software required to implement the communications protocols used to communicate and to exchange data with visual sensors 820, computing device(s) 870, and/or platform 850. Exemplary communications protocols supported by communications module 876 and 856 include TCP/IP, HTTP, Ethernet, video connection (e.g., digital video or IEEE 1394), and wireless networking protocols (e.g. 802.11). If visual sensor 820 is an analog device (such as a composite video camera), communications module 876 and 856 include a video card operative to digitize and store the incoming video frames.
  • System 80 optionally comprises a communications link 840 for communicating image data and control data between visual sensors 820 and computing device 870. Communications link 840 can include a wired or wireless connection between visual sensors 820 and computing device 870, such as, for example, a digital video connection under the IEEE 1394 protocol, to enable real-time transmission of image data from visual sensor 820. Alternatively, communications link 840 can be as simple as an input device operative to read images stored on media by visual sensor 820, such as, for example, a digital videotape created by a digital videotape recorder.
  • Computing device 870 optionally is coupled via link 882 to one or more storage devices 878. Storage device 878 comprises computer storage media and can be used to store image date generated by visual sensor 820. Computing device 870 optionally is coupled via link 884 to one or more display devices 879. Storage device 878 comprises a standalone PC, a workstation, a dumb terminal, and can be used to view image data generated or used by system 80. Links 882 and 884 can comprise a local connection or remote connection via a local area network, wide area network or the Web using protocols known to those of skill in the art.
  • Computing devices 870 and 851 can operate as one computing device in a standalone environment or in a networked environment using logical connections to one or more remote computing devices, which may be, by way of example, a server, a router, a network PC, personal computer, workstation, a hand-held device, a peer device or other common network node. All or part of the functionality of computing devices 870 and 851 could be implemented in application specific integrated circuits, field programmable gate arrays or other special purpose hardware. In a networked environment, program modules depicted relative to the computing devices 870 and 851, or portions thereof, may be stored on one or more remote computing devices.
  • System memory 874 and 854 stores the program and date instructions for implementing the method described herein. The software implementation of the method described herein can be encoded via conventional programming means known to those of ordinary skill in the art, e.g., Java language implementations. The method described herein also can be implemented in commercially available numerical computing environment and programming language such as Matlab.
  • Platform 850 optionally is adapted and configured to mount and/or transport visual sensor(s) 820, with or without an on-board computing device 851. In another embodiment platform 850 comprises one or more substructures, with visual sensors 820 mounted on substructure and a computing device 851 mounted on another substructure. When visual sensor(s) 820 are mounted on platform 850, platform 850 should be adapted and configured to monitor, store and update positioning and orientation parameters for visual sensors 820. Exemplary platforms 850 include, for example, unmanned vehicles (e.g., airborne surveillance drones), satellites, manned vehicles (e.g., cars, tanks, aircraft, spacecraft, submarines), robots, and a video camera mounted on an endoscope.
  • System 80 optionally comprises a communications link 830 which uses communications media to communicating image data and control data between visual sensors 820 and computing device 851 on platform 850. Communications link 830 can include a wired or wireless connection between visual sensors 820 and computing device 851, such as, for example, a digital video connection under the IEEE 1394 protocol, to enable real-time transmission of image data from visual sensor 820. System 80 optionally includes a communications link 860 which uses communication media to exchange image data and/or control information between platform 850 and computing device 870.
  • Those of skill will recognize that the techniques of the embodiments described herein may be implemented to advantage in a variety of sequential orders and that the present invention may be generally implemented in magnetic media such as memory, disks, tapes and CD-ROMs or other storage media for introduction into a system for fast dense stereoscopic ranging. In such cases, instructions for executing the steps described herein for fast dense stereoscopic ranging will be embedded in the media.
  • The terms and descriptions used herein are set forth by way of illustration only and are not meant as limitations. Those skilled in the art will recognize that many variations are possible within the spirit and scope of the invention as defined in the following claims, and their equivalents, in which all terms are to be understood in their broadest possible sense unless otherwise indicated. The described embodiments illustrate the scope of the claims but do not restrict the scope of the claims.

Claims (23)

1. A stochastic method for fast stereoscopic ranging comprising:
(a) selecting a pair of images for stereo processing, wherein the pair of images are a frame pair and one of the image is a reference frame;
(b) seeding estimated values for a range metric at each pixel of the reference frame;
(c) initializing one or more search stage constraints;
(d) stochastically computing local influence for each valid pixel in the reference frame, wherein a valid pixel is a pixel in the reference frame that has a corresponding pixel in the other frame of the frame pair;
(e) aggregating local influences for each valid pixel in the reference frame;
(f) refining the estimated values for the range metric at each valid pixel in the reference frame based on the aggregated local influence; and
(g) post-processing range metric data; and
(h) repeating n iterations of (d) through (g), wherein n is a value selected in (c).
2. The method of claim 1 further comprising repeating (c) through (h) one or more times.
3. The method of claim 2 further comprising visually displaying the estimated depth values.
4. The method of claim 1 further comprising pre-processing image data from the frame pair.
5. The method of claim 4 in which pre-processing image data from the frame pair includes rectifying the reference frame.
6. The method of claim 5 in which the rectifying eliminates the relative rotations between camera positions at which each frame of the frame pair were taken.
7. The method of claim 4 in which pre-processing image data from the frame pair includes computing unit disparity vectors for each pixel in the reference frame of the frame pair.
8. The method of claim 4 in which pre-processing image data from the frame pair includes constructing masks to identify invalid pixels in the reference frame.
9. The method of claim 1 in which post-processing image data includes smoothing depth estimates.
10. The method of claim 1 in which the range metric is an estimated floating point depth value and seeding estimated values seeds estimated floating point depth values.
11. The method of claim 10 in which the estimated floating point depth value is based on a previously estimated depth value.
12. The method of claim 10 in which the outputted ranges are the range metric output from the last iteration of the search stage.
13. The method of claim 1 in which the range metric is an estimated floating point disparity value and seeding estimated values seeds estimated floating point disparity values.
14. The method of claim 13 in which the estimated floating point disparity value is based on a previously estimated disparity value.
15. The method of claim 14 in which the outputted ranges are converted from the range metric output from the last iteration of the search stage.
16. The method of claim 1 in which selecting a pair of images selects two, non-adjacent image frames from a video stream.
17. The method of claim 1 in which computing local influence for each valid pixel in the reference frame comprises:
computing match quality at the valid pixel;
perturbing a depth estimate at the valid pixel with additive noise;
computing an induced local match quality, wherein the computing applies a match quality function to the corresponding pixels determined by the perturbed depth estimate; and
computing resulting per pixel influences with the original and induced local match quality.
18. A method for initializing depth estimates in an iterative process for stereoscopic ranging, the method comprising:
projecting a first depth estimate for each one of a plurality of valid pixels in an image, the image having an implicit camera origin;
generating a 3D representation of the plurality of valid pixels using the first depth estimate;
relocating the implicit camera origin;
reprojecting the 3D representation of the plurality of valid pixels to the relocated implicit camera origin;
using Z-buffering to render a second depth estimate for each one of the plurality of valid pixels according to the relocated implicit camera origin; and
using the second depth estimate for each of the plurality of valid pixels as an initial value in a subsequent iteration of a process for stereoscopic ranging.
19. The method of claim 18 in which the first 3D representation is a point cloud.
20. A computer readable medium comprising instructions for performing a stochastic method for fast stereoscopic ranging by:
(a) selecting a pair of images for stereo processing, wherein the pair of images are a frame pair and one of the image is a reference frame;
(b) seeding estimated values for a range metric at each pixel of the reference frame;
(c) initializing one or more search stage constraints;
(d) stochastically computing local influence for each valid pixel in the reference frame, wherein a valid pixel is a pixel in the reference frame that has a corresponding pixel in the other frame of the frame pair;
(e) aggregating local influences for each valid pixel in the reference frame;
(f) refining the estimated values for the range metric at each valid pixel in the reference frame based on the aggregated local influence;
(g) post-processing range metric data; and
(h) repeating (d) through (g) for n iterations, wherein n is a value selected in (c).
21. A system for stochastic fast stereoscopic ranging comprising:
a visual sensor capable of taking images;
a computing device coupled to the visual sensor, comprising a memory, wherein the memory includes instructions for performing a stochastic method for fast stereoscopic ranging by:
(a) selecting a pair of images for stereo processing, wherein the pair of images are a frame pair and one of the image is a reference frame;
(b) seeding estimated values for a range metric at each pixel of the reference frame;
(c) initializing one or more search stage constraints;
(d) stochastically computing local influence for each valid pixel in the reference frame, wherein a valid pixel is a pixel in the reference frame that has a corresponding pixel in the other frame of the frame pair;
(e) aggregating local influences for each valid pixel in the reference frame;
(f) refining the estimated values for the range metric at each valid pixel in the reference frame based on the aggregated local influence;
(g) post-processing range metric data; and
(h) repeating (d) through (g) for n iterations, wherein n is a value selected in (c).
22. The system of claim 21 in which the computing device is a server.
23. The system of claim 22 in which the network is a wireless network.
US12/451,064 2007-04-24 2008-04-24 Method and system for fast dense stereoscopic ranging Active 2030-04-27 US8467628B2 (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
US90795507P true 2007-04-24 2007-04-24
PCT/US2008/005271 WO2009023044A2 (en) 2007-04-24 2008-04-24 Method and system for fast dense stereoscopic ranging
US12/451,064 US8467628B2 (en) 2007-04-24 2008-04-24 Method and system for fast dense stereoscopic ranging

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US12/451,064 US8467628B2 (en) 2007-04-24 2008-04-24 Method and system for fast dense stereoscopic ranging

Publications (2)

Publication Number Publication Date
US20100111444A1 true US20100111444A1 (en) 2010-05-06
US8467628B2 US8467628B2 (en) 2013-06-18

Family

ID=40351329

Family Applications (1)

Application Number Title Priority Date Filing Date
US12/451,064 Active 2030-04-27 US8467628B2 (en) 2007-04-24 2008-04-24 Method and system for fast dense stereoscopic ranging

Country Status (2)

Country Link
US (1) US8467628B2 (en)
WO (1) WO2009023044A2 (en)

Cited By (54)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100091095A1 (en) * 2008-10-15 2010-04-15 Samsung Electronics Co., Ltd. Method for driving glasses-type stereoscopic display preventing visual fatigue and refractive index-variable shutter glasses
US20100149330A1 (en) * 2008-12-12 2010-06-17 Honeywell International Inc. Security system with operator-side privacy zones
US20110007138A1 (en) * 2008-01-04 2011-01-13 Hongsheng Zhang Global camera path optimization
US20120127171A1 (en) * 2009-05-21 2012-05-24 Jianguo Li Techniques for rapid stereo reconstruction from images
US20120150573A1 (en) * 2010-12-13 2012-06-14 Omar Soubra Real-time site monitoring design
US20130162768A1 (en) * 2011-12-22 2013-06-27 Wen-Nung Lie System for converting 2d video into 3d video
US20130257851A1 (en) * 2012-04-01 2013-10-03 Chao-Hua Lee Pipeline web-based process for 3d animation
US20140002675A1 (en) * 2012-06-28 2014-01-02 Pelican Imaging Corporation Systems and methods for detecting defective camera arrays and optic arrays
US20140064602A1 (en) * 2012-09-05 2014-03-06 Industrial Technology Research Institute Method and apparatus for object positioning by using depth images
US9025895B2 (en) 2011-09-28 2015-05-05 Pelican Imaging Corporation Systems and methods for decoding refocusable light field image files
US9041829B2 (en) 2008-05-20 2015-05-26 Pelican Imaging Corporation Capturing and processing of high dynamic range images using camera arrays
US9049381B2 (en) 2008-05-20 2015-06-02 Pelican Imaging Corporation Systems and methods for normalizing image data captured by camera arrays
US9100586B2 (en) 2013-03-14 2015-08-04 Pelican Imaging Corporation Systems and methods for photometric normalization in array cameras
US9123118B2 (en) 2012-08-21 2015-09-01 Pelican Imaging Corporation System and methods for measuring depth using an array camera employing a bayer filter
US9124864B2 (en) 2013-03-10 2015-09-01 Pelican Imaging Corporation System and methods for calibration of an array camera
US9128228B2 (en) 2011-06-28 2015-09-08 Pelican Imaging Corporation Optical arrangements for use with an array camera
US9143711B2 (en) 2012-11-13 2015-09-22 Pelican Imaging Corporation Systems and methods for array camera focal plane control
US9177381B2 (en) 2010-12-22 2015-11-03 Nani Holdings IP, LLC Depth estimate determination, systems and methods
US9185276B2 (en) 2013-11-07 2015-11-10 Pelican Imaging Corporation Methods of manufacturing array camera modules incorporating independently aligned lens stacks
US9210392B2 (en) 2012-05-01 2015-12-08 Pelican Imaging Coporation Camera modules patterned with pi filter groups
US9214013B2 (en) 2012-09-14 2015-12-15 Pelican Imaging Corporation Systems and methods for correcting user identified artifacts in light field images
US9247117B2 (en) 2014-04-07 2016-01-26 Pelican Imaging Corporation Systems and methods for correcting for warpage of a sensor array in an array camera module by introducing warpage into a focal plane of a lens stack array
US9253380B2 (en) 2013-02-24 2016-02-02 Pelican Imaging Corporation Thin form factor computational array cameras and modular array cameras
US9264610B2 (en) 2009-11-20 2016-02-16 Pelican Imaging Corporation Capturing and processing of images including occlusions captured by heterogeneous camera arrays
US9361412B1 (en) * 2012-03-26 2016-06-07 The United Sates Of America As Represented By The Secretary Of The Navy Method for the simulation of LADAR sensor range data
US9412206B2 (en) 2012-02-21 2016-08-09 Pelican Imaging Corporation Systems and methods for the manipulation of captured light field image data
US9426361B2 (en) 2013-11-26 2016-08-23 Pelican Imaging Corporation Array camera configurations incorporating multiple constituent array cameras
US9438888B2 (en) 2013-03-15 2016-09-06 Pelican Imaging Corporation Systems and methods for stereo imaging with camera arrays
US9497370B2 (en) 2013-03-15 2016-11-15 Pelican Imaging Corporation Array camera architecture implementing quantum dot color filters
US9497429B2 (en) 2013-03-15 2016-11-15 Pelican Imaging Corporation Extended color processing on pelican array cameras
US9516222B2 (en) 2011-06-28 2016-12-06 Kip Peli P1 Lp Array cameras incorporating monolithic array camera modules with high MTF lens stacks for capture of images used in super-resolution processing
US9521319B2 (en) 2014-06-18 2016-12-13 Pelican Imaging Corporation Array cameras and array camera modules including spectral filters disposed outside of a constituent image sensor
US9578259B2 (en) 2013-03-14 2017-02-21 Fotonation Cayman Limited Systems and methods for reducing motion blur in images or video in ultra low light with array cameras
US9633442B2 (en) 2013-03-15 2017-04-25 Fotonation Cayman Limited Array cameras including an array camera module augmented with a separate camera
US9638883B1 (en) 2013-03-04 2017-05-02 Fotonation Cayman Limited Passive alignment of array camera modules constructed from lens stack arrays and sensors based upon alignment information obtained during manufacture of array camera modules using an active alignment process
US9733486B2 (en) 2013-03-13 2017-08-15 Fotonation Cayman Limited Systems and methods for controlling aliasing in images captured by an array camera for use in super-resolution processing
US9741118B2 (en) 2013-03-13 2017-08-22 Fotonation Cayman Limited System and methods for calibration of an array camera
US9766380B2 (en) 2012-06-30 2017-09-19 Fotonation Cayman Limited Systems and methods for manufacturing camera modules using active alignment of lens stack arrays and sensors
US9774789B2 (en) 2013-03-08 2017-09-26 Fotonation Cayman Limited Systems and methods for high dynamic range imaging using array cameras
US9794476B2 (en) 2011-09-19 2017-10-17 Fotonation Cayman Limited Systems and methods for controlling aliasing in images captured by an array camera for use in super resolution processing using pixel apertures
US9800856B2 (en) 2013-03-13 2017-10-24 Fotonation Cayman Limited Systems and methods for synthesizing images from image data captured by an array camera using restricted depth of field depth maps in which depth estimation precision varies
US9813616B2 (en) 2012-08-23 2017-11-07 Fotonation Cayman Limited Feature based high resolution motion estimation from low resolution images captured using an array source
US9866739B2 (en) 2011-05-11 2018-01-09 Fotonation Cayman Limited Systems and methods for transmitting and receiving array camera image data
US9888194B2 (en) 2013-03-13 2018-02-06 Fotonation Cayman Limited Array camera architecture implementing quantum film image sensors
US9898856B2 (en) 2013-09-27 2018-02-20 Fotonation Cayman Limited Systems and methods for depth-assisted perspective distortion correction
US9936148B2 (en) 2010-05-12 2018-04-03 Fotonation Cayman Limited Imager array interfaces
US9942474B2 (en) 2015-04-17 2018-04-10 Fotonation Cayman Limited Systems and methods for performing high speed video capture and depth estimation using array cameras
US9955070B2 (en) 2013-03-15 2018-04-24 Fotonation Cayman Limited Systems and methods for synthesizing high resolution images using image deconvolution based on motion and depth information
US10009538B2 (en) 2013-02-21 2018-06-26 Fotonation Cayman Limited Systems and methods for generating compressed light field representation data using captured light fields, array geometry, and parallax information
US10089740B2 (en) 2014-03-07 2018-10-02 Fotonation Limited System and methods for depth regularization and semiautomatic interactive matting using RGB-D images
US10122993B2 (en) 2013-03-15 2018-11-06 Fotonation Limited Autofocus system for a conventional camera that uses depth information from an array camera
US10119808B2 (en) 2013-11-18 2018-11-06 Fotonation Limited Systems and methods for estimating depth from projected texture using camera arrays
US10250871B2 (en) 2014-09-29 2019-04-02 Fotonation Limited Systems and methods for dynamic calibration of array cameras
US10311649B2 (en) 2017-09-01 2019-06-04 Fotonation Limited Systems and method for performing depth based image editing

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102124742B (en) 2008-08-20 2013-09-11 汤姆逊许可公司 Refined depth map
CN102272778B (en) 2009-01-07 2015-05-20 汤姆森特许公司 Joint depth estimation
US8654181B2 (en) * 2011-03-28 2014-02-18 Avid Technology, Inc. Methods for detecting, visualizing, and correcting the perceived depth of a multicamera image sequence
US9619933B2 (en) 2014-06-16 2017-04-11 Occipital, Inc Model and sizing information from smartphone acquired image sequences
US9536320B1 (en) * 2014-12-23 2017-01-03 John H. Prince Multiple coordinated detectors for examination and ranging
EP3236286A1 (en) 2016-04-18 2017-10-25 Otis Elevator Company Auto commissioning system and method
US10192111B2 (en) 2017-03-10 2019-01-29 At&T Intellectual Property I, L.P. Structure from motion for drone videos

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6757445B1 (en) * 2000-10-04 2004-06-29 Pixxures, Inc. Method and apparatus for producing digital orthophotos using sparse stereo configurations and external models
US6862364B1 (en) * 1999-10-27 2005-03-01 Canon Kabushiki Kaisha Stereo image processing for radiography
US6911995B2 (en) * 2001-08-17 2005-06-28 Mitsubishi Electric Research Labs, Inc. Computer vision depth segmentation using virtual surface
US7035892B2 (en) * 1999-12-10 2006-04-25 Broadcom Corporation Apparatus and method for reducing precision of data
US7317812B1 (en) * 2002-11-15 2008-01-08 Videomining Corporation Method and apparatus for robustly tracking objects
US7522186B2 (en) * 2000-03-07 2009-04-21 L-3 Communications Corporation Method and apparatus for providing immersive surveillance
US7860301B2 (en) * 2005-02-11 2010-12-28 Macdonald Dettwiler And Associates Inc. 3D imaging system

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040104935A1 (en) * 2001-01-26 2004-06-03 Todd Williamson Virtual reality immersion system

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6862364B1 (en) * 1999-10-27 2005-03-01 Canon Kabushiki Kaisha Stereo image processing for radiography
US7035892B2 (en) * 1999-12-10 2006-04-25 Broadcom Corporation Apparatus and method for reducing precision of data
US7522186B2 (en) * 2000-03-07 2009-04-21 L-3 Communications Corporation Method and apparatus for providing immersive surveillance
US6757445B1 (en) * 2000-10-04 2004-06-29 Pixxures, Inc. Method and apparatus for producing digital orthophotos using sparse stereo configurations and external models
US6911995B2 (en) * 2001-08-17 2005-06-28 Mitsubishi Electric Research Labs, Inc. Computer vision depth segmentation using virtual surface
US7317812B1 (en) * 2002-11-15 2008-01-08 Videomining Corporation Method and apparatus for robustly tracking objects
US7860301B2 (en) * 2005-02-11 2010-12-28 Macdonald Dettwiler And Associates Inc. 3D imaging system

Cited By (129)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110007138A1 (en) * 2008-01-04 2011-01-13 Hongsheng Zhang Global camera path optimization
US8803958B2 (en) * 2008-01-04 2014-08-12 3M Innovative Properties Company Global camera path optimization
US9060124B2 (en) 2008-05-20 2015-06-16 Pelican Imaging Corporation Capturing and processing of images using non-monolithic camera arrays
US10027901B2 (en) 2008-05-20 2018-07-17 Fotonation Cayman Limited Systems and methods for generating depth maps using a camera arrays incorporating monochrome and color cameras
US10142560B2 (en) 2008-05-20 2018-11-27 Fotonation Limited Capturing and processing of images including occlusions focused on an image sensor by a lens stack array
US9124815B2 (en) 2008-05-20 2015-09-01 Pelican Imaging Corporation Capturing and processing of images including occlusions captured by arrays of luma and chroma cameras
US9576369B2 (en) 2008-05-20 2017-02-21 Fotonation Cayman Limited Systems and methods for generating depth maps using images captured by camera arrays incorporating cameras having different fields of view
US9191580B2 (en) 2008-05-20 2015-11-17 Pelican Imaging Corporation Capturing and processing of images including occlusions captured by camera arrays
US9077893B2 (en) 2008-05-20 2015-07-07 Pelican Imaging Corporation Capturing and processing of images captured by non-grid camera arrays
US9485496B2 (en) 2008-05-20 2016-11-01 Pelican Imaging Corporation Systems and methods for measuring depth using images captured by a camera array including cameras surrounding a central camera
US9188765B2 (en) 2008-05-20 2015-11-17 Pelican Imaging Corporation Capturing and processing of images including occlusions focused on an image sensor by a lens stack array
US9060121B2 (en) 2008-05-20 2015-06-16 Pelican Imaging Corporation Capturing and processing of images captured by camera arrays including cameras dedicated to sampling luma and cameras dedicated to sampling chroma
US9060142B2 (en) 2008-05-20 2015-06-16 Pelican Imaging Corporation Capturing and processing of images captured by camera arrays including heterogeneous optics
US9712759B2 (en) 2008-05-20 2017-07-18 Fotonation Cayman Limited Systems and methods for generating depth maps using a camera arrays incorporating monochrome and color cameras
US9060120B2 (en) 2008-05-20 2015-06-16 Pelican Imaging Corporation Systems and methods for generating depth maps using images captured by camera arrays
US9749547B2 (en) 2008-05-20 2017-08-29 Fotonation Cayman Limited Capturing and processing of images using camera array incorperating Bayer cameras having different fields of view
US9235898B2 (en) 2008-05-20 2016-01-12 Pelican Imaging Corporation Systems and methods for generating depth maps using light focused on an image sensor by a lens element array
US9049367B2 (en) 2008-05-20 2015-06-02 Pelican Imaging Corporation Systems and methods for synthesizing higher resolution images using images captured by camera arrays
US9041829B2 (en) 2008-05-20 2015-05-26 Pelican Imaging Corporation Capturing and processing of high dynamic range images using camera arrays
US9041823B2 (en) 2008-05-20 2015-05-26 Pelican Imaging Corporation Systems and methods for performing post capture refocus using images captured by camera arrays
US9049391B2 (en) 2008-05-20 2015-06-02 Pelican Imaging Corporation Capturing and processing of near-IR images including occlusions using camera arrays incorporating near-IR light sources
US9049390B2 (en) 2008-05-20 2015-06-02 Pelican Imaging Corporation Capturing and processing of images captured by arrays including polychromatic cameras
US9055233B2 (en) 2008-05-20 2015-06-09 Pelican Imaging Corporation Systems and methods for synthesizing higher resolution images using a set of images containing a baseline image
US9049381B2 (en) 2008-05-20 2015-06-02 Pelican Imaging Corporation Systems and methods for normalizing image data captured by camera arrays
US9049411B2 (en) 2008-05-20 2015-06-02 Pelican Imaging Corporation Camera arrays incorporating 3×3 imager configurations
US9055213B2 (en) 2008-05-20 2015-06-09 Pelican Imaging Corporation Systems and methods for measuring depth using images captured by monolithic camera arrays including at least one bayer camera
US9094661B2 (en) 2008-05-20 2015-07-28 Pelican Imaging Corporation Systems and methods for generating depth maps using a set of images containing a baseline image
US20100091095A1 (en) * 2008-10-15 2010-04-15 Samsung Electronics Co., Ltd. Method for driving glasses-type stereoscopic display preventing visual fatigue and refractive index-variable shutter glasses
US8908017B2 (en) * 2008-10-15 2014-12-09 Samsung Electronics Co., Ltd. Method for driving glasses-type stereoscopic display preventing visual fatigue and refractive index-variable shutter glasses
US8576282B2 (en) * 2008-12-12 2013-11-05 Honeywell International Inc. Security system with operator-side privacy zones
US20100149330A1 (en) * 2008-12-12 2010-06-17 Honeywell International Inc. Security system with operator-side privacy zones
US9053550B2 (en) * 2009-05-21 2015-06-09 Intel Corporation Techniques for rapid stereo reconstruction from images
US9652849B2 (en) * 2009-05-21 2017-05-16 Intel Corporation Techniques for rapid stereo reconstruction from images
US20120127171A1 (en) * 2009-05-21 2012-05-24 Jianguo Li Techniques for rapid stereo reconstruction from images
US20150348266A1 (en) * 2009-05-21 2015-12-03 Intel Corporation Techniques for Rapid Stereo Reconstruction from Images
US10306120B2 (en) 2009-11-20 2019-05-28 Fotonation Limited Capturing and processing of images captured by camera arrays incorporating cameras with telephoto and conventional lenses to generate depth maps
US9264610B2 (en) 2009-11-20 2016-02-16 Pelican Imaging Corporation Capturing and processing of images including occlusions captured by heterogeneous camera arrays
US9936148B2 (en) 2010-05-12 2018-04-03 Fotonation Cayman Limited Imager array interfaces
US20120150573A1 (en) * 2010-12-13 2012-06-14 Omar Soubra Real-time site monitoring design
US9177381B2 (en) 2010-12-22 2015-11-03 Nani Holdings IP, LLC Depth estimate determination, systems and methods
US10218889B2 (en) 2011-05-11 2019-02-26 Fotonation Limited Systems and methods for transmitting and receiving array camera image data
US9866739B2 (en) 2011-05-11 2018-01-09 Fotonation Cayman Limited Systems and methods for transmitting and receiving array camera image data
US9128228B2 (en) 2011-06-28 2015-09-08 Pelican Imaging Corporation Optical arrangements for use with an array camera
US9578237B2 (en) 2011-06-28 2017-02-21 Fotonation Cayman Limited Array cameras incorporating optics with modulation transfer functions greater than sensor Nyquist frequency for capture of images used in super-resolution processing
US9516222B2 (en) 2011-06-28 2016-12-06 Kip Peli P1 Lp Array cameras incorporating monolithic array camera modules with high MTF lens stacks for capture of images used in super-resolution processing
US9794476B2 (en) 2011-09-19 2017-10-17 Fotonation Cayman Limited Systems and methods for controlling aliasing in images captured by an array camera for use in super resolution processing using pixel apertures
US9031335B2 (en) 2011-09-28 2015-05-12 Pelican Imaging Corporation Systems and methods for encoding light field image files having depth and confidence maps
US9129183B2 (en) 2011-09-28 2015-09-08 Pelican Imaging Corporation Systems and methods for encoding light field image files
US20180197035A1 (en) 2011-09-28 2018-07-12 Fotonation Cayman Limited Systems and Methods for Encoding Image Files Containing Depth Maps Stored as Metadata
US9025895B2 (en) 2011-09-28 2015-05-05 Pelican Imaging Corporation Systems and methods for decoding refocusable light field image files
US9036931B2 (en) 2011-09-28 2015-05-19 Pelican Imaging Corporation Systems and methods for decoding structured light field image files
US9042667B2 (en) 2011-09-28 2015-05-26 Pelican Imaging Corporation Systems and methods for decoding light field image files using a depth map
US9025894B2 (en) 2011-09-28 2015-05-05 Pelican Imaging Corporation Systems and methods for decoding light field image files having depth and confidence maps
US9864921B2 (en) 2011-09-28 2018-01-09 Fotonation Cayman Limited Systems and methods for encoding image files containing depth maps stored as metadata
US9811753B2 (en) 2011-09-28 2017-11-07 Fotonation Cayman Limited Systems and methods for encoding light field image files
US9031343B2 (en) 2011-09-28 2015-05-12 Pelican Imaging Corporation Systems and methods for encoding light field image files having a depth map
US9536166B2 (en) 2011-09-28 2017-01-03 Kip Peli P1 Lp Systems and methods for decoding image files containing depth maps stored as metadata
US10019816B2 (en) 2011-09-28 2018-07-10 Fotonation Cayman Limited Systems and methods for decoding image files containing depth maps stored as metadata
US10275676B2 (en) 2011-09-28 2019-04-30 Fotonation Limited Systems and methods for encoding image files containing depth maps stored as metadata
US20130162768A1 (en) * 2011-12-22 2013-06-27 Wen-Nung Lie System for converting 2d video into 3d video
US9167232B2 (en) * 2011-12-22 2015-10-20 National Chung Cheng University System for converting 2D video into 3D video
US9754422B2 (en) 2012-02-21 2017-09-05 Fotonation Cayman Limited Systems and method for performing depth based image editing
US9412206B2 (en) 2012-02-21 2016-08-09 Pelican Imaging Corporation Systems and methods for the manipulation of captured light field image data
US9361412B1 (en) * 2012-03-26 2016-06-07 The United Sates Of America As Represented By The Secretary Of The Navy Method for the simulation of LADAR sensor range data
US20130257851A1 (en) * 2012-04-01 2013-10-03 Chao-Hua Lee Pipeline web-based process for 3d animation
US9210392B2 (en) 2012-05-01 2015-12-08 Pelican Imaging Coporation Camera modules patterned with pi filter groups
US9706132B2 (en) 2012-05-01 2017-07-11 Fotonation Cayman Limited Camera modules patterned with pi filter groups
US9100635B2 (en) * 2012-06-28 2015-08-04 Pelican Imaging Corporation Systems and methods for detecting defective camera arrays and optic arrays
US20140002675A1 (en) * 2012-06-28 2014-01-02 Pelican Imaging Corporation Systems and methods for detecting defective camera arrays and optic arrays
US9807382B2 (en) 2012-06-28 2017-10-31 Fotonation Cayman Limited Systems and methods for detecting defective camera arrays and optic arrays
US9766380B2 (en) 2012-06-30 2017-09-19 Fotonation Cayman Limited Systems and methods for manufacturing camera modules using active alignment of lens stack arrays and sensors
US10261219B2 (en) 2012-06-30 2019-04-16 Fotonation Limited Systems and methods for manufacturing camera modules using active alignment of lens stack arrays and sensors
US9129377B2 (en) 2012-08-21 2015-09-08 Pelican Imaging Corporation Systems and methods for measuring depth based upon occlusion patterns in images
US9147254B2 (en) 2012-08-21 2015-09-29 Pelican Imaging Corporation Systems and methods for measuring depth in the presence of occlusions using a subset of images
US9240049B2 (en) 2012-08-21 2016-01-19 Pelican Imaging Corporation Systems and methods for measuring depth using an array of independently controllable cameras
US9235900B2 (en) 2012-08-21 2016-01-12 Pelican Imaging Corporation Systems and methods for estimating depth and visibility from a reference viewpoint for pixels in a set of images captured from different viewpoints
US9123117B2 (en) 2012-08-21 2015-09-01 Pelican Imaging Corporation Systems and methods for generating depth maps and corresponding confidence maps indicating depth estimation reliability
US9858673B2 (en) 2012-08-21 2018-01-02 Fotonation Cayman Limited Systems and methods for estimating depth and visibility from a reference viewpoint for pixels in a set of images captured from different viewpoints
US9123118B2 (en) 2012-08-21 2015-09-01 Pelican Imaging Corporation System and methods for measuring depth using an array camera employing a bayer filter
US9813616B2 (en) 2012-08-23 2017-11-07 Fotonation Cayman Limited Feature based high resolution motion estimation from low resolution images captured using an array source
US20140064602A1 (en) * 2012-09-05 2014-03-06 Industrial Technology Research Institute Method and apparatus for object positioning by using depth images
US9117138B2 (en) * 2012-09-05 2015-08-25 Industrial Technology Research Institute Method and apparatus for object positioning by using depth images
US9214013B2 (en) 2012-09-14 2015-12-15 Pelican Imaging Corporation Systems and methods for correcting user identified artifacts in light field images
US9143711B2 (en) 2012-11-13 2015-09-22 Pelican Imaging Corporation Systems and methods for array camera focal plane control
US9749568B2 (en) 2012-11-13 2017-08-29 Fotonation Cayman Limited Systems and methods for array camera focal plane control
US10009538B2 (en) 2013-02-21 2018-06-26 Fotonation Cayman Limited Systems and methods for generating compressed light field representation data using captured light fields, array geometry, and parallax information
US9374512B2 (en) 2013-02-24 2016-06-21 Pelican Imaging Corporation Thin form factor computational array cameras and modular array cameras
US9253380B2 (en) 2013-02-24 2016-02-02 Pelican Imaging Corporation Thin form factor computational array cameras and modular array cameras
US9774831B2 (en) 2013-02-24 2017-09-26 Fotonation Cayman Limited Thin form factor computational array cameras and modular array cameras
US9743051B2 (en) 2013-02-24 2017-08-22 Fotonation Cayman Limited Thin form factor computational array cameras and modular array cameras
US9638883B1 (en) 2013-03-04 2017-05-02 Fotonation Cayman Limited Passive alignment of array camera modules constructed from lens stack arrays and sensors based upon alignment information obtained during manufacture of array camera modules using an active alignment process
US9774789B2 (en) 2013-03-08 2017-09-26 Fotonation Cayman Limited Systems and methods for high dynamic range imaging using array cameras
US9917998B2 (en) 2013-03-08 2018-03-13 Fotonation Cayman Limited Systems and methods for measuring scene information while capturing images using array cameras
US9986224B2 (en) 2013-03-10 2018-05-29 Fotonation Cayman Limited System and methods for calibration of an array camera
US9124864B2 (en) 2013-03-10 2015-09-01 Pelican Imaging Corporation System and methods for calibration of an array camera
US10225543B2 (en) 2013-03-10 2019-03-05 Fotonation Limited System and methods for calibration of an array camera
US9800856B2 (en) 2013-03-13 2017-10-24 Fotonation Cayman Limited Systems and methods for synthesizing images from image data captured by an array camera using restricted depth of field depth maps in which depth estimation precision varies
US9733486B2 (en) 2013-03-13 2017-08-15 Fotonation Cayman Limited Systems and methods for controlling aliasing in images captured by an array camera for use in super-resolution processing
US9741118B2 (en) 2013-03-13 2017-08-22 Fotonation Cayman Limited System and methods for calibration of an array camera
US9888194B2 (en) 2013-03-13 2018-02-06 Fotonation Cayman Limited Array camera architecture implementing quantum film image sensors
US10127682B2 (en) 2013-03-13 2018-11-13 Fotonation Limited System and methods for calibration of an array camera
US9787911B2 (en) 2013-03-14 2017-10-10 Fotonation Cayman Limited Systems and methods for photometric normalization in array cameras
US9578259B2 (en) 2013-03-14 2017-02-21 Fotonation Cayman Limited Systems and methods for reducing motion blur in images or video in ultra low light with array cameras
US10091405B2 (en) 2013-03-14 2018-10-02 Fotonation Cayman Limited Systems and methods for reducing motion blur in images or video in ultra low light with array cameras
US9100586B2 (en) 2013-03-14 2015-08-04 Pelican Imaging Corporation Systems and methods for photometric normalization in array cameras
US9800859B2 (en) 2013-03-15 2017-10-24 Fotonation Cayman Limited Systems and methods for estimating depth using stereo array cameras
US10122993B2 (en) 2013-03-15 2018-11-06 Fotonation Limited Autofocus system for a conventional camera that uses depth information from an array camera
US9497370B2 (en) 2013-03-15 2016-11-15 Pelican Imaging Corporation Array camera architecture implementing quantum dot color filters
US9438888B2 (en) 2013-03-15 2016-09-06 Pelican Imaging Corporation Systems and methods for stereo imaging with camera arrays
US9955070B2 (en) 2013-03-15 2018-04-24 Fotonation Cayman Limited Systems and methods for synthesizing high resolution images using image deconvolution based on motion and depth information
US9633442B2 (en) 2013-03-15 2017-04-25 Fotonation Cayman Limited Array cameras including an array camera module augmented with a separate camera
US9497429B2 (en) 2013-03-15 2016-11-15 Pelican Imaging Corporation Extended color processing on pelican array cameras
US9602805B2 (en) 2013-03-15 2017-03-21 Fotonation Cayman Limited Systems and methods for estimating depth using ad hoc stereo array cameras
US10182216B2 (en) 2013-03-15 2019-01-15 Fotonation Limited Extended color processing on pelican array cameras
US9898856B2 (en) 2013-09-27 2018-02-20 Fotonation Cayman Limited Systems and methods for depth-assisted perspective distortion correction
US9426343B2 (en) 2013-11-07 2016-08-23 Pelican Imaging Corporation Array cameras incorporating independently aligned lens stacks
US9264592B2 (en) 2013-11-07 2016-02-16 Pelican Imaging Corporation Array camera modules incorporating independently aligned lens stacks
US9185276B2 (en) 2013-11-07 2015-11-10 Pelican Imaging Corporation Methods of manufacturing array camera modules incorporating independently aligned lens stacks
US9924092B2 (en) 2013-11-07 2018-03-20 Fotonation Cayman Limited Array cameras incorporating independently aligned lens stacks
US10119808B2 (en) 2013-11-18 2018-11-06 Fotonation Limited Systems and methods for estimating depth from projected texture using camera arrays
US9456134B2 (en) 2013-11-26 2016-09-27 Pelican Imaging Corporation Array camera configurations incorporating constituent array cameras and constituent cameras
US9813617B2 (en) 2013-11-26 2017-11-07 Fotonation Cayman Limited Array camera configurations incorporating constituent array cameras and constituent cameras
US9426361B2 (en) 2013-11-26 2016-08-23 Pelican Imaging Corporation Array camera configurations incorporating multiple constituent array cameras
US10089740B2 (en) 2014-03-07 2018-10-02 Fotonation Limited System and methods for depth regularization and semiautomatic interactive matting using RGB-D images
US9247117B2 (en) 2014-04-07 2016-01-26 Pelican Imaging Corporation Systems and methods for correcting for warpage of a sensor array in an array camera module by introducing warpage into a focal plane of a lens stack array
US9521319B2 (en) 2014-06-18 2016-12-13 Pelican Imaging Corporation Array cameras and array camera modules including spectral filters disposed outside of a constituent image sensor
US10250871B2 (en) 2014-09-29 2019-04-02 Fotonation Limited Systems and methods for dynamic calibration of array cameras
US9942474B2 (en) 2015-04-17 2018-04-10 Fotonation Cayman Limited Systems and methods for performing high speed video capture and depth estimation using array cameras
US10311649B2 (en) 2017-09-01 2019-06-04 Fotonation Limited Systems and method for performing depth based image editing

Also Published As

Publication number Publication date
WO2009023044A2 (en) 2009-02-19
WO2009023044A3 (en) 2009-04-09
US8467628B2 (en) 2013-06-18

Similar Documents

Publication Publication Date Title
Scharstein et al. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms
Goesele et al. Multi-view stereo for community photo collections
Xiao et al. Reconstructing the world’s museums
Hirschmuller Stereo processing by semiglobal matching and mutual information
Bylow et al. Real-time camera tracking and 3D reconstruction using signed distance functions.
Salas-Moreno et al. Slam++: Simultaneous localisation and mapping at the level of objects
Hähnel et al. Learning compact 3D models of indoor and outdoor environments with a mobile robot
US6487304B1 (en) Multi-view approach to motion and stereo
US7187809B2 (en) Method and apparatus for aligning video to three-dimensional point clouds
Basha et al. Multi-view scene flow estimation: A view centered variational approach
Golparvar-Fard et al. Evaluation of image-based modeling and laser scanning accuracy for emerging automated performance monitoring techniques
Dolson et al. Upsampling range data in dynamic environments
Pizzoli et al. REMODE: Probabilistic, monocular dense reconstruction in real time
Furukawa et al. Multi-view stereo: A tutorial
US6476803B1 (en) Object modeling system and process employing noise elimination and robust surface extraction techniques
Whelan et al. Real-time large-scale dense RGB-D SLAM with volumetric fusion
Zhao et al. Alignment of continuous video onto 3D point clouds
US20090324059A1 (en) Method for determining a depth map from images, device for determining a depth map
Eigen et al. Depth map prediction from a single image using a multi-scale deep network
Vanegas et al. Building reconstruction using manhattan-world grammars
Geiger et al. Stereoscan: Dense 3d reconstruction in real-time
US9361723B2 (en) Method for real-time face animation based on single video camera
Vogiatzis et al. Video-based, real-time multi-view stereo
Vineet et al. Incremental dense semantic stereo fusion for large-scale semantic scene reconstruction
Whelan et al. Deformation-based loop closure for large scale dense RGB-D SLAM

Legal Events

Date Code Title Description
AS Assignment

Owner name: 21 CT, INC.,TEXAS

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:COFFMAN, THAYNE R.;REEL/FRAME:023437/0451

Effective date: 20091022

Owner name: 21 CT, INC., TEXAS

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:COFFMAN, THAYNE R.;REEL/FRAME:023437/0451

Effective date: 20091022

STCF Information on status: patent grant

Free format text: PATENTED CASE

REMI Maintenance fee reminder mailed
FPAY Fee payment

Year of fee payment: 4

SULP Surcharge for late payment