Classifications

• • G—PHYSICS
  • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
  • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
  • G03B37/00—Panoramic or wide-screen photography; Photographing extended surfaces, e.g. for surveying; Photographing internal surfaces, e.g. of pipe
  • G03B37/04—Panoramic or wide-screen photography; Photographing extended surfaces, e.g. for surveying; Photographing internal surfaces, e.g. of pipe with cameras or projectors providing touching or overlapping fields of view
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
  • G01C11/00—Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
  • G01C11/02—Picture taking arrangements specially adapted for photogrammetry or photographic surveying, e.g. controlling overlapping of pictures

Definitions

• the present invention relates, generally, to the field of remote imaging techniques and, more particularly, to a system for rendering high-resolution, high accuracy, low distortion digital images over very large fields of view.
• Remote sensing and imaging are broad-based technologies having a number of diverse and extremely important practical applications - such as geological mapping and analysis, and meteorological forecasting.
• Aerial and satellite-based photography and imaging are especially useful remote imaging techniques that have, over recent years, become heavily reliant on the collection and processing of data for digital images, including spectral, spatial, elevation, and vehicle or platform location and orientation parameters.
• Spatial data - characterizing real estate improvements and locations, roads and highways, environmental hazards and conditions, utilities infrastructures (e.g., phone lines, pipelines), and geophysical features - can now be collected, processed, and communicated in a digital format to conveniently provide highly accurate mapping and surveillance data for various applications (e.g., dynamic GPS mapping).
• Elevation data may be used to improve the overall system's spatial and positional accuracy and may be acquired from either existing Digital Elevation Model (DEM) data sets or collected with the spectral sensor data from an active, radiation measuring Doppler based devices, or passive, stereographic calculations.
• DEM Digital Elevation Model
• Photographic issues such as spherical aberrations, astigmatism, field curvature, distortion, and chromatic aberrations are well-known problems that must be dealt with in any sensor/imaging application.
• Certain applications require very high image resolution - often with tolerances of inches.
• an actual digital imaging device may be located anywhere from several feet to miles from its target, resulting in a very large scale factor. Providing images with very large scale factors, that also have resolution tolerances of inches, poses a challenge to even the most robust imaging system.
• Ortho-imaging is an approach that has been used in an attempt to address this problem.
• ortho-imaging renders a composite image of a target by compiling varying sub-images of the target.
• a digital imaging device that has a finite range and resolution records images of fixed subsections of a target area sequentially. Those images are then aligned according to some sequence to render a composite of a target area.
• the present invention relates to remote data collection and processing system using a variety of sensors.
• the system may include computer console units that control vehicle and system operations in real-time.
• the system may also include global positioning systems that are linked to and communicate with the computer consoles.
• cameras and/or camera array assemblies can be employed for producing an image of a target viewed through an aperture.
• the camera array assemblies are communicatively connected to the computer consoles.
• the camera array assembly has a mount housing, a first imaging sensor centrally coupled to the housing having a first focal axis passing through the aperture.
• the camera array assembly also has a second imaging sensor coupled to the housing and offset from the first imaging sensor along an axis, that has a second focal axis passing through the aperture and intersecting the first focal axis within an intersection area.
• the camera array assembly has a third imaging sensor, coupled to the housing and offset from the first imaging sensor along the axis, opposite the second imaging sensor, that has a third focal axis passing through the aperture and intersecting the first focal axis within the intersection area. Any number of one-to-n cameras may be used in this manner, where "n" can be any odd or even number.
• the system may also include an Attitude Measurement Unit (AMU) such as inertial, optical, or similar measurement units communicatively connected to the computer consoles and the camera array assemblies.
• AMU Attitude Measurement Unit
• the AMU may determine the yaw, pitch, and/or roll of the aircraft at any instant in time and successive DGPS positions may be used to measure the vehicle heading with relation to geodesic north.
• the AMU data is integrated with the precision DGPS data to produce a robust, real-time AMU system.
• the system may further include a mosaicing module housed within the computer consoles.
• the mosaicing module includes a first component for performing initial 100 processing on an input image.
• the mosaicing module also includes a second component for determining geographical boundaries of an input image with the second component being cooperatively engaged with the first component.
• the mosaicing module further includes a third component for mapping an input image into the composite image with accurate geographical position.
• the third component being cooperatively engaged with 105 the first and second components.
• a fourth component is also included in the mosaicing module for balancing color of the input images mapped into the composite image.
• the fourth component can be cooperatively engaged with the first, second and third components.
• the mosaicing module can include a fifth component for blending borders between adjacent input images mapped into the composite image.
• the 110 fifth component being cooperatively engaged with the first, second, third and fourth components.
• a sixth component an optional forward oblique and/or optional rear oblique camera array system may be implemented that collects oblique image data and merges the image data with attitude and positional measurements in order to create a three
• 3D point cloud i.e., 3D point cloud
• DEM digital elevation model
• the 3D point cloud or DEM is a representation of the ground surface including man made structures.
• the DEM may be created using stereographic techniques from ortho and/or oblique imagery, or, alternatively, provided by LIDAR or an existing DEM.
• the DEM or 3D point cloud may be created from any overlapping images from a single camera
• the present invention may employ a certain degree of lateral 130 oversampling to improve output quality and/or co-mounted, co-registered oversampling to overcome physical pixel resolution limits.
• FIG. 1 illustrates a vehicle based data collection and processing system of the present invention
• FIG. 1A illustrates a portion of the vehicle based data collection and processing 140 system of FIG. 1;
• FIG. IB illustrates a portion of the vehicle based data collection and processing system of FIG. 1 ;
• FIG. 2 illustrates a vehicle based data collection and processing system of FIG. 1 with the camera array assembly of the present invention shown in more detail;
• FIG. 3A illustrates camera array assembly configured in across track, cross-eyed fashion in accordance with certain aspects of the present invention
• FIG. 3B illustrates a camera array assembly configured in a long track, crosseyed fashion in accordance with certain aspects of the present invention
• FIG. 3C-1 illustrates a camera array assembly configured in along track, cross- 150 eyed fashion in accordance with certain aspects of the present invention
• FIG. 3C-2 illustrates a sequence of images obtained from a camera array assembly configured in along track, cross-eyed fashion in accordance with certain aspects of the present invention
• FIG. 3D illustrates a camera array assembly configured in across track, cross- 155 eyed and along track, cross-eyed fashion in accordance with certain aspects of the present invention
• FIG. 3E illustrates a camera array assembly configured in a long track, crosseyed and across track, cross-eyed fashion in accordance with certain aspects of the present invention
• FIG. 4A illustrates one embodiment of an imaging pattern retrieved by the camera array assembly of FIGS. 1 and 3A;
• FIG. 4B illustrates one embodiment of an imaging pattern retrieved by the camera system of FIGS. 1 and 3B
• FIG. 4C-1 illustrates one embodiment of an imaging pattern retrieved by the 165 camera system of FIGS. 1 and 3C-1;
• FIG. 4C-2 illustrates one embodiment of an imaging pattern retrieved by the camera system of FIGS. 1 and 3C-2;
• FIG. 4D illustrates one embodiment of an imaging pattern retrieved by the camera system of FIGS. 1 and 3D;
• FIG. 4E illustrates one embodiment of an imaging pattern retrieved by the camera system of FIGS. 1 and 3E;
• FIG. 5 depicts an imaging pattern illustrating certain aspects of the present invention
• FIG. 6 illustrates an image strip in accordance with the present invention
• FIG. 7 illustrates another embodiment of an image strip in accordance with the present invention.
• FIG. 8 illustrates one embodiment of an imaging process in accordance with the present invention
• FIG. 9 illustrates diagrammatic ally how photos taken with the camera array
• FIG. 10 is a block diagram of the processing logic according to certain embodiments of the present invention.
• FIG. 11 illustrates lateral oversampling looking down from a vehicle according to certain embodiments of the present invention
• FIG. 12 illustrates lateral oversampling looking down from a vehicle according to certain embodiments of the present invention
• FIG. 13 illustrates flight line oversampling looking down from a vehicle according to certain embodiments of the present invention
• FIG. 14 illustrates flight line oversampling looking down from a vehicle
• FIG. 15 illustrates progressive magnification looking down from a vehicle according to certain embodiments of the present invention
• FIG. 16 illustrates progressive magnification looking down from a vehicle according to certain embodiments of the present invention
• FIG. 17 illustrates progressive magnification looking down from a vehicle according to certain embodiments of the present invention
• FIG. 18 is a schematic of the system architecture according to certain embodiments of the present invention.
• FIG. 19 illustrates lateral co-mounted, co-registered oversampling in a sidelap 200 sub-pixel area for a single camera array looking down from a vehicle according to certain embodiments of the present invention
• FIG. 20 illustrates lateral co-mounted, co-registered oversampling in a sidelap sub-pixel area for two overlapping camera arrays looking down from a vehicle according to certain embodiments of the present invention
• FIG. 21 illustrates fore and lateral co-mounted, co-registered oversampling in sidelap sub-pixel areas for two stereo camera arrays looking down from a vehicle according to certain embodiments of the present invention
• FIG. 22A illustrates a rear right side perspective view of the camera array of FIG.
• FIG. 22B illustrates a front right side perspective view of the camera array of FIG.
• FIG. 23 illustrates a bottom view of concave or retinal camera array assembly configured in across track, cross-eyed and long track, cross-eyed fashion in accordance with certain aspects of the present invention
• FIG. 24 illustrates one embodiment of an oblique camera array assembly in accordance with certain aspects of the present invention
• FIG. 25 illustrates an image strip in accordance with the present invention
• FIG. 26A illustrates one embodiment of a camera array assembly configured in along track, cross-eyed fashion in accordance with certain aspects of the present 220 invention.
• FIG. 26B illustrates a bottom view of the camera array of FIG. 26A.
• FIGS. 1, 1A & IB A vehicle based data collection and processing system 100 of the present invention is shown in FIGS. 1, 1A & IB. Additional aspects and embodiments of the 230 present invention are shown in FIGS. 2 & 18.
• System 100 includes one or more computer consoles 102.
• the computer consoles contain one or more computers 104 for controlling both vehicle and system operations. Examples of the functions of the computer console are the controlling digital color sensor systems that can be associated with the data collection and processing system, providing the display data to a pilot,
• a communications link 235 coordinating the satellite generated GPS pulse-per-second (PPS) event trigger (which may be 20 or more pulses per second), data logging, sensor control and adjustment, checking and alarming for error events, recording and indexing photos, storing and processing data, flight planning capability that automates the navigation of the vehicle, data, and providing a real-time display of pertinent information.
• PPS GPS pulse-per-second
• control computer console interface between the control computer console and the vehicle autopilot control provides the ability to actually control the flight path of the vehicle in real-time. This results in a more precise control of the vehicle's path than is possible by a human being. All of these functions can be accomplished by the use of various computer programs that are synchronized to the GPS PPS signals and take into account the various electrical
• the computer is embedded within the sensor.
• One or more differential global positioning systems 106 are incorporated into the system 100.
• the global positioning systems 106 are used to navigate and determine precise flight paths during vehicle and system operations. To accomplish this, the global
• 250 positioning systems 106 are communicatively linked to the computer console 102 such that the information from the global positioning systems 106 can be acquired and processed without flight interruption.
• Zero or more GPS units may be located at known survey points in order to provide a record of each sub-seconds' GPS satellite-based errors in order to be able to back correct the accuracy of the system 100. GPS and/or
• ground based positioning services may be used that eliminate the need for ground control points altogether. This technique results in greatly improved, sub-second by sub-second positional accuracy of the data capture vehicle.
• One or more AMUs 108 that provide real-time yaw, pitch, and roll information that is used to accurately determine the attitude of the vehicle at the instant of data
• the present attitude measurement unit (e.g., Applanix POS AV), uses three high performance fiber optic gyros, one gyro each for yaw, pitch, and roll measurement. AMUs from other manufacturers, and AMUs that use other inertial measurement devices can be used as well. Additionally, an AMU may be employed to determine the instantaneous attitude of
• the AMU Connected to the AMU can be one or more multi-frequency DGPS receivers 110.
• the multi-frequency DGPS receivers 110 can be integrated with the AMU's yaw, pitch, and roll attitude data in order to more accurately determine the location of the remote sensor platform in three dimensional space. Additionally, the direction of geodesic North may be integrated with the AMU's yaw, pitch, and roll attitude data in order to more accurately determine the location of the remote sensor platform in three dimensional space. Additionally, the direction of geodesic North may
• 270 be determined by the vector created by successive DGPS positions, recorded in a synchronized manner with the GPS PPS signals.
• One or more camera array assemblies 112 for producing an image of a target viewed through an aperture are also communicatively connected to the one or more computer consoles 102.
• the camera array assemblies 112 which will be described in 275 greater detail below, provide the data collection and processing system with the ability to capture high resolution, high precision progressive scan or line scan, color digital photography.
• the system may also include DC power and conditioning equipment 114 to condition DC power and to invert DC power to AC power in order to provide electrical
• the system may further include a navigational display 116, which graphically renders the position of the vehicle versus the flight plan for use by the pilot (either onboard or remote) of the vehicle to enable precision flight paths in horizontal and vertical planes.
• the system may also include an EMU module comprised of LIDAR, SAR 118 or a forward and rear oblique camera array for capturing three
• the EMU module 118 can include a laser unit 120, an EMU control unit 122, and an EMU control computer 124. Temperature controlling devices, such as solid state cooling modules, can also be deployed as needed in order to provide the proper thermal environment for the system.
• the system also includes a mosaicing module, not depicted, housed with the 290 computer console 102.
• the mosaicing module which will be described in further detail below, provides the system the ability to gather data acquired by the global positioning system 106, the AMU 108, and the camera system 112 and process that data into useable orthomaps.
• the system 100 also can include a self-locking flight path technique that 295 provides the ability to micro-correct the positional accuracy of adjacent flight paths in order to realize precision that exceeds the native precision of the AMU and DGPS sensors alone.
• a complete flight planning methodology is used to micro plan all aspects of missions.
• the inputs are the various mission parameters (latitude/longitude, resolution,
• vehicle path checks for alarm conditions and corrective actions, notifies the pilot and/or crew of overall system status, and provides for fail-safe operations and controls. Safe operations parameters may be constantly monitored and reported. Whereas the current system uses a manned crew, the system is designed to perform equally well in an unmanned vehicle.
• FIG. 2 shows another depiction of the present invention.
• the camera array assembly 112 is shown in more detail. As is shown, the camera array assembly 112 allows for images to be acquired from the rear oblique, the forward oblique and the nadir positions.
• FIGS. 3A-3E describe in more detail examples of camera array assemblies of 315 the present invention.
• FIGS. 3A-3E provide examples of camera array assemblies 300 airborne over target 302 (e.g., terrain).
• target 302 e.g., terrain
• the camera array assembly 300 comprises a housing 304 within which imaging sensors 306, 308, 310, 312 and 314 are disposed along a 320 concave curvilinear axis 316.
• the housing may be a mount unit.
• Assembly 300 is adapatably mountable to a vehicle that moves with respect to a terrain along a path.
• the radius of curvature of axis 316 may vary or be altered dramatically, providing the ability to effect very subtle or very drastic degrees of concavity in axis 316.
• axis 316 may be completely linear - having no 325 curvature at all.
• the imaging sensors 306, 308, 310, 312 and 314 couple to the housing
• Attachment members 318 may comprise a number of fixed or dynamic, permanent or temporary, connective apparatus.
• the attachment members 318 may comprise simple welds, removable clamping devices, or electro-mechanically controlled universal joints.
• the system 100 may have a real-time, onboard navigation system to provide a visual, bio-feedback display to the vehicle pilot, or remote display in the case of operations in an unmanned vehicle.
• the pilot is able to adjust the position of the vehicle in real-time in order to provide a more accurate flight path.
• the pilot may be onboard the vehicle or remotely located and using the flight display to control the vehicle
• the system 100 may also use highly fault-tolerant methods that have been developed to provide a software inter-leaved disk storage methodology that allows one or two hard drives to fail and still not lose target data that is stored on the drives.
• This software inter-leaved disk storage methodology provides superior fault-tolerance and
• the system 100 may also incorporate a methodology that has been developed that allows for a short calibration step just before mission data capture.
• the calibration methodology step adjusts the camera settings, mainly exposure time, based on sampling the ambient light intensity and setting near optimal values just before reaching the region
• a moving average algorithm is then used to make second-by- second camera adjustments in order to deliver improved, consistent photo results. This improves the color processing of the orthomaps. Additionally, the calibration may be used to check or to establish the exact spatial position of each sensor device (cameras, DPG, AMU, EMU, etc.). In this manner, changes that may happen in the spatial location of these devices
• system 100 may incorporate a methodology that has been developed that allows for calibrating the precision position and attitude of each sensor device (cameras, DPG, AMU, EMU, etc.) on the vehicle by flying over an area that contains multiple known, visible, highly accurate geographic positions.
• a program takes
• the imaging sensors may be arranged in across track, cross-eyed fashion.
• housing 304 comprises a simple enclosure inside of which imaging sensors 306, 308, 310, 312 and
• FIG. 3A depicts a 5-camera array, the system works equally well when utilizing any number of imaging sensors from 1 to any number. Sensors 306 through 314 couple, via the attachment members 318, either collectively to a single
• the housing 304 may itself comprise only a supporting cross member of concave curvature (e.g., a mount unit) to which the imaging sensors 306 through 314 couple, via members 318.
• the housing 304 may comprise a hybrid combination of enclosure and
• the housing 304 further comprises an aperture 320 formed in its surface, between the imaging sensors and target 302.
• the aperture 320 may comprise only a void, or it may comprise a protective screen or window to maintain environmental integrity within the housing 304.
• a protective transparent plate is used for any sensor, special coatings
• the aperture 320 may comprise a lens or other optical device to enhance or alter the nature of the images recorded by the sensors.
• the aperture 320 is formed with a size and shape sufficient to provide the imaging sensors 306 through 314 proper lines of sight to a target region 322 on terrain 302.
• the imaging sensors 306 through 314 are disposed within or along housing 304 such that the focal axes of all sensors converge and intersect each other within an intersection area bounded by the aperture 320. Depending upon the type of image data being collected, the specific imaging sensors used, and other optics or equipment employed, it may be necessary or desirable to offset the intersection area or point of
• the imaging sensors 306 through 314 are separated from each other at angular intervals.
• the exact angle of displacement between the imaging sensors may vary widely depending upon the number of imaging sensors utilized and on the type of imaging data being collected.
• the angular displacement between the imaging sensors may also be unequal, if required, so as to provide a desired
• the focal axes of all imaging sensors may intersect at exactly the same point, or may intersect at a plurality of points, all within close proximity to each other and within the intersection area defined by the aperture 320.
• the imaging sensor 310 is centrally disposed within the housing 304 along axis 316.
• the imaging sensor 310 has a focal axis 324, directed orthogonally from the housing 304 to align the line of sight of the imaging sensor with the image area 326 of the region 322.
• the imaging sensor 308 is disposed within the housing 304 along the axis 316, adjacent to the imaging sensor 310.
• the imaging sensor 312 is disposed within the housing 304 adjacent to the imaging sensor 310, on the opposite side of the axis 316 as the imaging sensor 308. The imaging sensor 312 is aligned such that its line of sight
• the imaging sensor 306 is disposed within the housing 304 along the axis 316, adjacent to the sensor 308. The imaging sensor 306 is aligned such that its line of sight coincides with the image area 336 of region 322, and such that its focal axis 338
• the imaging sensor 314 is disposed within housing 304 adjacent to sensor 312, on the opposite side of axis 316 as sensor 306.
• the imaging sensor 314 is aligned such that its line of sight coincides with image area 340 of region 322, and such that its focal axis 344 converges with and intersects the other focal axes within the area bounded by
• the imaging sensors 306 through 314 may comprise a number of digital imaging devices including, for example, individual area scan cameras, line scan cameras, infrared sensors, hyperspectral and/or seismic sensors. Each sensor may comprise an individual imaging device, or may itself comprise an imaging array.
• sensors 306 through 314 may all be of a homogenous nature, or may comprise a combination of varied imaging devices.
• the imaging sensors 306 through 314 are hereafter referred to as cameras 306 through 314, respectively.
• lens distortion is typically a source of imaging problems. Each individual lens must be carefully calibrated to determine
• Cameras 306 through 314 are alternately disposed within housing 304 along axis 316 such that each camera's focal axis converges upon aperture 320, crosses focal
• the camera array assembly 300 is configured such that adjoining borders of image areas 326, 328, 332, 336 and 340 overlap slightly. In an embodiment, the adjoining borders of image areas 340 and 332, 332 and 326, 326 and 328, and 328
• the adjoining borders overlap between about 10% and about 80%. In another embodiment, the adjoining borders overlap between about 20% and about 60%.
• assembly 300 provides the ability to produce images having customizable FOVs, of
• assembly 300 may be deployed to produce stereoscopic images.
• any number of mount units, containing any number of imaging sensors having various shapes and sizes, may be combined to provide imaging data on any desired target region.
• the imaging sensors may be arranged in a long-track, cross-eyed fashion.
• housing 304 comprises a simple enclosure inside of which imaging sensors 306, 308, 310, 312 and 314 are disposed.
• the housing 304 may be replaced by a mount unit (not shown) inside of which imaging sensors 306, 308, 310, 312 and 314 are
• FIG. 3B depicts a 5 -camera array
• the system works equally well when utilizing any number of camera sensors from 1 to any number.
• Sensors 306 through 314 couple, via the attachment members 318, either collectively to a single transverse cross member, or individually to lateral cross members disposed between opposing walls of the housing 304.
• the housing 304 may
• the housing 304 itself comprise only a supporting cross member of concave curvature (e.g., a mount unit) to which the imaging sensors 306 through 314 couple, via members 318.
• the housing 304 may comprise a hybrid combination of enclosure and supporting cross member.
• the housing 304 further comprises an aperture 320 formed in its surface, between the imaging sensors and target 302.
• the aperture 320 may comprise only a void, or it may comprise a protective screen or window to maintain environmental integrity within the housing 304.
• the aperture 320 may comprise a lens or other optical device to enhance or alter the nature of 465 the images recorded by the sensors.
• the aperture 320 is formed with a size and shape sufficient to provide the imaging sensors 306 through 314 proper lines of sight to a target region 322 on terrain 302.
• the imaging sensors 306 through 314 are disposed within or along housing 304 such that the focal axes of all sensors converge within an intersection area bounded by
• the aperture 320 may be 470 the aperture 320.
• the specific imaging sensors used, and other optics or equipment employed it may be necessary or desirable to offset the intersection area or point of convergence above or below the aperture 320.
• the imaging sensors 306 through 314 are separated from each other at angular intervals. The exact angle of displacement between the imaging sensors may
• the angular displacement between the imaging sensors may also be unequal, if required, so as to provide a desired image offset or alignment.
• the focal axes of all imaging sensors may intersect at exactly the same point,
• 480 may intersect at a plurality of points, all within close proximity to each other and within the intersection area defined by the aperture 320.
• the imaging sensor 310 is centrally disposed within the housing 304 along axis 316.
• the imaging sensor 310 has a focal axis 324, directed orthogonally from the housing 304 to align the line of sight of the imaging sensor with
• the imaging sensor 308 is disposed within the housing 304 along the axis 316, adjacent to the imaging sensor 310.
• the imaging sensor 308 is aligned such that its line of sight coincides with the image area 328 of the region 322, and such that its focal axis 330 converges with and intersects the axis 324 within the area bounded by the aperture 320.
• the imaging sensor 312 is disposed within the
• the imaging sensor 312 is aligned such that its line of sight coincides with the image area 332 of the region 322, and such that its focal axis 334 converges with and intersects axes 324 and 330 within the area bounded by the aperture 320.
• the imaging sensor 306 is disposed within the housing 304 along the axis 316,
• the imaging sensor 306 is aligned such that its line of sight coincides with the image area 336 of region 322, and such that its focal axis 338 converges with and intersects the other focal axes within the area bounded by aperture 320.
• the imaging sensor 314 is disposed within housing 304 adjacent to sensor 312, on the opposite side of axis 316 as sensor 306. The imaging sensor 314 is aligned such that
• Cameras 306 through 314 are alternately disposed within housing 304 along axis 316 such that each camera's focal axis converges upon aperture 320, crosses focal
• the camera array assembly 300 is configured such that adjoining borders of image areas 326, 328 and 332 overlap slightly. In an embodiment, the adjoining borders of image areas 326 and 328 and/or 326 and 332 overlap between about
• the adjoining borders overlap between about 30% and about 95%. In another embodiment, the adjoining borders overlap between about 50% and about 90%.
• the adjoining borders of image areas 328 and 336, and 332 and 340 may or may not overlap slightly. In an embodiment, the adjoining borders of image areas 328
• the adjoining borders overlap between about 30% to about 95%. In another embodiment, the adjoining borders overlap between about 50% and about 90%.
• the imaging sensors may be arranged in a long-track, cross-eyed fashion as in camera array 2600.
• mount unit 2604 comprises a simple structure inside of which imaging sensors 2606, 2608, 2610 and 2612 are disposed. Whereas FIG. 26A depicts a 4-camera array, the system works equally well when utilizing any number of imaging sensors from 1 to any number. Sensors 2606 through 2612 couple, via the attachment members 2618, either collectively to a single transverse cross member, or individually to 525 lateral cross members disposed between opposing walls of the mount unit 2604.
• the mount unit 2604 further comprises an aperture 2620 formed in its surface, between the imaging sensors and a target (not shown).
• the aperture 2620 may comprise only a void, or it may comprise a protective screen or window to maintain 530 environmental integrity within the mount unit 2604.
• a protective transparent plate is used for any sensor, special coatings may be applied to the plate to improve the quality of the sensor data.
• the aperture 2620 may comprise a lens or other optical device to enhance or alter the nature of the images recorded by the sensors.
• the aperture 2620 is formed with a size and shape sufficient to provide the 535 imaging sensors 2606 through 2612 proper lines of sight to a target region on terrain (not shown).
• the imaging sensors 2606 through 2614 are disposed within or along a concave curvilinear array axis 2616 in mount unit 2604 such that the focal axes of all sensors converge and intersect each other within an
• intersection area bounded by the aperture 2620 540 intersection area bounded by the aperture 2620.
• the imaging sensors 2606 through 2612 are separated from each other at angular intervals. The exact angle of displacement
• the angular displacement between the imaging sensors may also be unequal, if required, so as to provide a desired image offset or alignment.
• imaging sensors may intersect at exactly the same point, or may intersect at a plurality of points, all within close proximity to each other and within the intersection area defined by the aperture 2620.
• the imaging sensors may be arranged in a long-track, cross-eyed fashion for mapping infrastructure.
• housing 304 comprises a simple enclosure inside of which imaging sensors 306, 308, 310, 312 and 314 are disposed.
• the housing 304 may be replaced by a mount unit (not shown) inside of which imaging sensors 306, 308, 310, 312 and 314 are disposed.
• FIG. 3B depicts a 5 -camera array, the system works equally well when utilizing any number of camera sensors from
• sensors 306 through 314 couple, via the attachment members 318, either collectively to a single transverse cross member, or individually to lateral cross members disposed between opposing walls of the housing 304.
• the housing 304 may itself comprise only a supporting cross member of concave curvature (e.g., a mount unit) to which the imaging sensors 306
• the housing 304 may comprise a hybrid combination of enclosure and supporting cross member.
• the housing 304 further comprises an aperture 320 formed in its surface, between the imaging sensors and target 302.
• the aperture 320 may comprise only a void, or it may comprise a protective screen or window to maintain
• the aperture 320 may comprise a lens or other optical device to enhance or alter the nature of the images recorded by the sensors.
• the aperture 320 is formed with a size and shape sufficient to provide the imaging sensors
• the imaging sensors 306 through 314 are disposed within or along housing 304 such that the focal axes of all sensors converge within an intersection area bounded by the aperture 320.
• the imaging sensors 306 through 314 are separated from each other at angular intervals.
• the exact angle of displacement between the imaging sensors may vary widely depending upon the number of imaging sensors utilized and on the type of imaging data being collected. The angular
• 585 displacement between the imaging sensors may also be unequal, if required, so as to provide a desired image offset or alignment.
• the focal axes of all imaging sensors may intersect at exactly the same point, or may intersect at a plurality of points, all within close proximity to each other and within the intersection area defined
• the imaging sensor 310 is centrally disposed within the housing 304 along axis 316.
• the imaging sensor 310 has a focal axis 324, directed orthogonally from the housing 304 to align the line of sight of the imaging sensor with the image area 326 of the region 322.
• the imaging sensor 308 is disposed within the
• the imaging sensor 312 is disposed within the housing 304 adjacent to the imaging sensor 310, on the opposite side of the axis 316 as
• the imaging sensor 312 is aligned such that its line of sight coincides with the image area 332 of the region 322, and such that its focal axis 334 converges with and intersects axes 324 and 330 within the area bounded by the aperture 320.
• the imaging sensor 306 is disposed within the housing 304 along the axis 316, adjacent to the sensor 308. Dissimilar to the camera assembly of FIG. 3B, the imaging
• 605 sensor 306 is aligned such that its line of sight coincides with the forward oblique image area 336 of region 322, and such that its focal axis 338 converges with and intersects the other focal axes within the area bounded by aperture 320.
• imaging sensor 306 captures the forward oblique image area 336 including a rear view of a second tower 346.
• the imaging sensor 314 is disposed within housing 304
• imaging sensor 314 is aligned such that its line of sight coincides with rear oblique image area 340 of region 322, and such that its focal axis 344 converges with and intersects the other focal axes within the area bounded by aperture 320. As depicted in FIGS. 3C & 4C, imaging sensor 314 captures the forward oblique image area 340 including a front view
• cameras 306 through 314 are alternately disposed within housing 304 along axis 316 such that each camera's focal axis converges upon aperture 320, crosses focal axis 324, and aligns its field of view with a target area opposite its respective position in the array resulting in a "cross-eyed",
• the camera array assembly 300 is configured such that adjoining borders of image areas 326, 328 and 332 overlap slightly. In an embodiment, the adjoining borders of image areas 326 and 328 and/or 326 and 332 overlap between about 1% and about 99% of the image area. In another embodiment, the adjoining borders overlap between about 30% and about 95%. 625 In another embodiment, the adjoining borders overlap between about 50% and about 90%.
• the adjoining borders of image areas 328 and 336, and 332 and 340 may or may not overlap slightly. In an embodiment, the adjoining borders of image areas 328 and 336, and 332 and 340 overlap between about 0% and about 100%. In another
• the adjoining borders overlap between about 30% to about 95%. In another embodiment, the adjoining borders overlap between about 50% and about 90%.
• FIG. 3C-2 an exemplary sequence of images obtained using a camera array assembly configured in a along track, cross-eyed fashion is depicted. Although the camera array assembly of FIG. 3C-1 is shown, other along track camera
• FIG. 3C-2 illustrates how long track sensors cover infrastructure such as transmission towers, insulators/conductors, transformers and other linearly aligned corridor objects by collecting a sequence of overlapping images that cover front and back sides of long track objects.
• the imaging sensors may be 640 arranged in a cross track, cross-eyed and long-track, cross-eyed fashion. See e.g., FIGS.
• FIG. 22A depicts a concave or retinal camera array assembly 2200 from a rear right side perspective view. Whereas FIG. 22A depicts a 15-camera array, the system works equally well when utilizing any number of imaging sensors from 1 to any number.
• Assembly 2200 is similar in composition, construction and operation to 645 assembly 300.
• Assembly 2200 comprises a first imaging array 2202, a second imaging array 2204 and a third imaging array 2206.
• Array 2204 is configured as a primary sensor array, disposed within assembly 2200 such that the focal axis 2208 of its primary imaging sensor 2210 is directed downwardly from assembly 2200, orthogonal to a target area 2212 along a terrain 2214.
• Assembly 2200 is adaptably mountable to a vehicle that 650 moves, with respect to terrain 2214, along a flight path 2216.
• Array 2202 is offset, with respect to flight path 2216, in front of mount unit 2204 by angular offset 2218.
• array 2206 is offset, with respect to flight path 2216, behind array 2204 by angular offset 2220.
• Angular offset 2218 is selected such that focal axis 2222 of 655 primary imaging sensor 2224 disposed within mount unit 2202 is directed downward toward target area 2212 forming angle 2232 with the target surface.
• Angular offset 2220 is selected such that the focal axis 2228 of primary imaging sensor 2230 disposed within mount unit 2206 is directed downward toward target area 2212 forming angle 2226 with the target surface.
• angular offsets 2218 and 2220 are equal, but they may be
• the focal axes of the other individual imaging sensors disposed within mount units 2202, 2204 and 2206 form similar angular relationships to the target area 2212 and one another, subject to their relative positions along the mount units.
• the imaging sensors may be arranged in
• FIG. 22B depicts a concave or retinal camera array assembly 2200 from a front right side perspective view.
• Array 2204 is configured as a primary sensor array, disposed within assembly 2200 such that the focal axis 2208 of its primary imaging sensor 2210 is directed downwardly from assembly 2200, orthogonal to a target
• Mount unit 2202 is offset, with respect to perpendicular
• mount unit 2206 is offset, with respect to perpendicular 2234 of flight path 2216, to the right (viewing the array from the rear) of mount unit 2204 by angular offset 2220.
• Angular offset 2218 is selected such that focal
• axis 2222 of primary imaging sensor 2224 disposed within mount unit 2202 is directed downward toward target area 2212 forming angle 2232 with the target surface.
• Angular offset 2220 is selected such that the focal axis 2228 of primary imaging sensor 2230 disposed within mount unit 2206 is directed downward toward target area 2212 forming angle 2226 with the target surface.
• angular offsets 2218 and 2220 are equal,
• the focal axes of the other individual imaging sensors disposed within mount units 2202, 2204 and 2206 form similar angular relationships to the target area 2212 and one another, subject to their relative positions along the mount units.
• the imaging sensors may be any type of the imaging sensors.
• FIG. 23 depicts a concave or retinal camera array assembly 2300 from a bottom view. Whereas FIG. 23 depicts a 25 -camera array, the system works equally well when utilizing any number of imaging sensors from 1 to any number.
• Assembly 2300 comprises a primary compound concave curvilinear mount unit 2302 and a plurality of compound curvilinear mount units 2304 that are formed of a size and curvatures sufficient to offset and arch over or contact mount unit 2302 at various angular intervals. Any number of mount units 2304 may be employed, and may be so numerous as to form a dome structure for mounting sensors.
• assembly 2300 may comprise two mount units in an orthogonal (i.e., 90°) relationship with one another.
• Another assembly, having three mount units, may be configured such that the angular displacement between the mount units is 60°.
• a primary imaging sensor 2306 is centrally disposed along the concave side of mount unit 2302, with its focal axis directed orthogonally downward from assembly 2300.
• a number of imaging sensors 2308 are disposed, in accordance with the teachings of the present invention, along the concave sides of mount units 2302 and 2304 in a "cross-eyed" fashion.
• the cross-eyed imaging sensors 2308 are alternatively disposed along mount units 2302 and 2304 such that the focal axis of each imaging sensor converges upon the focal axis of imaging sensor 2306 at an intersection area (not shown), and aligns its field of view with a target area opposite its respective position in the array.
• assembly 2300 provides the ability to produce images having customizable FOVs, of a generally circular nature. Depending on the mount units and imaging sensors utilized, assembly 2300 may be deployed to produce stereoscopic images. In alternative embodiments, any number of mount units, containing any number of imaging sensors having various shapes and sizes, may be combined to provide imaging data on any desired target region.
• the attachment members 318 are of a permanent and fixed nature (e.g., welds)
• the spatial relationship between the aperture 320, the cameras, and their lines of sight remain fixed as will the spatial relationship between image areas 326, 328, 332, 336 and 340.
• Such a configuration may be desirable in, for example, a satellite surveillance application where the camera array assembly 300 will remain at an essentially fixed distance from region 322. The position and alignment of the cameras is set such that areas 326, 328, 332, 336 and 340 provide full imaging coverage of region 322.
• attachment members 318 are of a temporary or adjustable nature, however, it may be desirable to selectively adjust, either manually or by remote automation, the position or alignment of the cameras so as to shift, narrow or widen areas 326, 328, 332, 336 and 340 - thereby enhancing or altering the quality of images collected by the camera array assembly 300.
• the rigid mount unit may or may not be affixed to a rigid 725 mount plate.
• the mount unit is any rigid structure to which at least one imaging sensor may be affixed.
• the mount unit may be a housing, which encloses the imaging sensor, but may be any rigid structure including a brace, tripod, or the like.
• an imaging sensor means any device capable of receiving and processing active or passive radiometric energy, i.e., light, sound, heat, gravity, and the like, from a 730 target area.
• imaging sensors may include any number of digital cameras, including those that utilize a red-blue-green filter, a bushbroom filter, or a hyperspectral filter, LIDAR sensors, infrared sensors, heat-sensing sensors, gravitometers and the like.
• Imagining sensors do not include attitude measuring sensors such as gyroscopes, GPS devices, and the like devices, which serve to orient the vehicle with the aid of satellite 735 data and/or inertial data.
• the multiple sensors are different.
• single, i.e., at least one, rigid mount unit may be affixed to the same rigid mount plate.
• multiple, i.e., at least two, rigid mount units may be affixed to the same rigid mount plate.
• the mount unit preferably has an aperture through which light and/or energy may pass.
• the mount plate is preferably planer, but may be non-planer.
• the mount plate preferably has aperture(s) in alignment with the aperture(s) of the
• a rigid structure is one that flexes less than about 100 th of a degree, preferably less than about 1,000 th of a degree, more preferably less than about 10,000 th of a degree while in use.
• the rigid structure is one that flexes less than about 100 th of a degree, preferably less than about 1,000 th of a degree, more preferably less than about
• Camera 310 is designated as the principal camera.
• camera 310 serves as a plane of reference.
• the orientations of the other cameras 306, 308, 312 and 314 are measured relative to the plane of reference.
• the relative orientations of each camera are measured in terms of the yaw, pitch and roll angles required to rotate the image plane of the camera to become parallel to the plane of reference.
• the order of rotations is preferably yaw, pitch, and roll.
• the imaging sensors affixed to the mount unit(s) may not be aligned in the same plane. Instead, the angle of their mount relative to the mount angle of a first sensor affixed to the first mount unit, preferably the principle nadir camera of the first mount unit, may be offset. Accordingly, the imaging sensors may be co-registered to calibrate the physical mount angle offset of each imaging sensor relative to each other. In an
• multiple, i.e., at least two, rigid mount units are affixed to the same rigid mount plate and are co-registered.
• the cameras 306 through 314 are affixed to a rigid mount unit and co-registered.
• the geometric centerpoint of the AMU preferably a gyroscope, is determined using GPS and inertial data. The physical position of the first sensor affixed to the first mount unit, preferably
• the principle nadir camera of the first mount unit is calculated relative to a reference point, preferably the geometric centerpoint of the AMU. Likewise, the physical position of all remaining sensors within all mount units are calculated-directly or indirectly- relative to the same reference point.
• the boresight angle of a sensor is defined as the angle from the geometric
• the boresight angle of the first sensor may be determined using the ground target points.
• the boresight angles of subsequent sensors are preferably calculated with reference to the boresight angle of the first sensor.
• the sensors are preferably calibrated using known ground targets, which are preferably photo-
• the imaging sensor within the second mount unit may be any imaging sensor, and is preferably a LIDAR.
• the second imaging sensor is a digital camera
• the boresight angle of the sensor(s) affixed to the second mount unit are calculated with reference to the boresight angle of the first sensor.
• the physical offset of the imaging sensor(s) within the second mount unit may be calibrated with reference to the boresight angle of the first sensor within the first mount unit.
• FIGS. 4A-4E images of areas 336, 328, 326, 332 and 340 taken by cameras 306 through 314, respectively, are illustrated from an overhead view. Again, because of the "cross-eyed" arrangement, the image of area 336 is taken by camera 306, the image of area 340 is taken by camera 314, and so on. In one embodiment of the present invention, images other than those taken by the center camera
• FIG. 800 310 take on a trapezoidal shape after perspective transformation. See e.g., FIG. 4A.
• Cameras 306 through 314 form an array along axis 316 that is, in most applications, pointed down vertically.
• cameras 308, 310 and 312 form an ortho array along axis 316 that is pointed down vertically
• cameras 306 and 314 form an oblique array also along axis 316. See e.g., FIGS. 4B & 4C.
• infrastructure e.g., infrastructure
• a second array of cameras configured similar the array of cameras 306 through 314, is aligned with respect to the first array of cameras to have an oblique view providing a
• the angle of declination from horizontal of the heads-up camera array assembly may vary due to mission objectives and parameters but angles of 25-45 degrees are typical.
• Other alternative embodiments, varying the mounting of camera arrays, are similarly comprehended by the present invention. See e.g., FIGS. 4D & 4E. In all such embodiments, the relative positions and attitudes of the cameras are
• an external mechanism e.g., a robot, a motorcycle, or a bicycle, or a motorcycle.
• GPS timing signal is used to trigger the cameras simultaneously thereby capturing an array of input images.
• a mosaicing module then renders the individual input images 820 from such an array into an ortho-rectified compound image (or "mosaic"), without any visible seams between the adjacent images.
• the mosaicing module performs a set of tasks comprising: determining the geographical boundaries and dimensions of each input image; projecting each input image onto the mosaic with accurate geographical positioning; balancing the color of the images in the mosaic; and blending adjacent input
• the mosaicing module performs only a single transformation to an original input image during mosaicing. That transformation can be represented by a 4 x 4 matrix.
• pixels in the mosaic may not be mapped to by any pixels in the input images (i.e., input pixels). Warped lines could potentially result as artifacts in the mosaic.
• each input and output pixel is further divided into an n x /n grid of sub-pixels. Transformation is performed from sub-pixels to sub-pixels. The final value of an output pixel is the average value of its sub-pixels for which there is a corresponding input sub-pixel. Larger n and m values produce mosaics of higher resolution, but do require extra processing time.
• the mosaicing module may utilize the following information: the spatial position (e.g., x, y, z coordinates) of each camera's focal point at the time an input image is captured; the attitude (i.e., yaw, pitch, roll) of each camera's image plane relative to the target region's ground plane at the time an input image was captured; each camera's fields of view (i.e., along track and cross
• the attitude can be provided by the AMUs associated with the system.
• Digital terrain models (DTMs) or Digital surface models (DSMs) can be created from information obtained using a LIDAR module 118.
• LIDAR is similar to the more familiar radar, and can be thought of as laser radar. In radar, radio waves are transmitted into the atmosphere that scatters some of the
• LIDAR also transmits and receives electromagnetic radiation, but at a higher frequency since it operates in the ultraviolet, visible and infrared region of the electromagnetic spectrum.
• LIDAR transmits light out to a target area.
• the transmitted light interacts with and is changed by the target area. Some of this light is reflected / scattered back to the LIDAR instrument where it can be 855 analyzed.
• the change in the properties of the light enables some property of the target area to be determined.
• the time for the light to travel out to the target area and back to LIDAR device is used to determine the range to the target.
• DTM and DSM data sets can also be captured from the camera array assembly.
• Traditional means of obtaining elevation data may also be used such as stereographic 860 techniques.
• Range finders are the simplest LIDAR and is used to measure the distance from the LIDAR device to a solid or hard target.
• DIAL LIDAR is used to measure chemical concentrations (such as ozone, water
• a DIAL LIDAR uses two different laser wavelengths that are selected so that one of the wavelengths is absorbed by the molecule of interest while the other wavelength is not. The difference in intensity of the two return signals can be used to deduce the concentration of the molecule being investigated.
• Doppler LIDAR is used to measure the velocity of a target.
• the wavelength of the light reflected/scattered off the target will be changed slightly. This is known as a Doppler-shift and therefore Doppler LIDAR. If the target is moving away from the LIDAR, the return light will have a longer wavelength (sometimes referred to as a red shift), if moving towards the LIDAR the return light will be at a shorter wavelength (blue
• the target can be either a hard target or an atmospheric target (e.g. microscopic dust and aerosol particles that are carried by the wind.
• a camera's focal point is preferably used as a perspective transformation center. Its position in space may be determined, for example, by a multi-frequency carrier phase post-processed GPS system mounted on the host craft. The offsets, in three
• a camera's focal point is preferably carefully measured against the center of the GPS antenna. These offsets may be combined with the position of the GPS antenna, and the orientation of the host craft, to determine the exact position of the camera's focal point.
• the position of the GPS antenna is preferably determined by processing of collected GPS data against similar ground-based GPS antennas deployed at
• One or more AMUs are preferably mounted onboard for attitude determination.
• the attitude of the AMU reference plane relative to the target region's ground plane is preferably measured and recorded at short intervals, with accuracy better than one-hundredth of one degree.
• the attitude of the AMU 890 reference plane may be defined as the series of rotations that can be performed on the axes of this plane to make it parallel to the ground plane. The term “align" could also be used to describe this operation.
• the attitude of center camera 310 (i.e. its image plane), relative to the AMU, is preferably precisely calibrated.
• center camera 310 is preferably also be carefully calibrated. This dependent calibration is more efficient than directly calibrating each camera.
• the camera array assembly 300 is remounted, only center camera 310 needs to be recalibrated. Effectively, a series of two transformations is applied to an input image from center camera 310. First, the center camera's image plane is aligned to the AMU plane. Then, the AMU plane is
• the position of the focal point of center camera 310 may be determined as 905 described above.
• the x and y components of this position preferably determine the position of the mosaic's nadir point 400 on the ground.
• Field of view (FOV) angles of each camera are known, thus the dimensions of each input image may be determined by the z component of that camera's focal point.
• An average elevation of the ground is preferably determined by computing the average elevation of points in the DTMs of the 910 area, and then each input image is projected to an imaginary horizontal plane at this elevation. Relief displacement is then preferably applied using the DTMs of the area.
• the DTMs can be obtained from many sources including: the USGS 30- or 10-meter DTMs available for most of the US; commercial DTMs; or DTMs obtained by a LIDAR or SAR EMU device mounted on the host craft that captures data concurrently with the 915 cameras.
• the resulting compound image also needs to have radiometric consistency throughout, and no visible seams at the joints between two adjacent images.
• the present invention provides a number of techniques for achieving this goal.
• a characteristic of a conventional camera is the exposure time (i.e., the time the shutter is open to collect light onto the image plane). The longer the exposure time, the lighter the resultant image becomes. Exposure time must adapt to changes in ambient lighting caused by conditions such as: cloud coverage; the angle and position of the sun relative to the camera; and so forth. Optimal exposure time may also depend on
• Exposure time is adjusted to keep the average intensity of an image within a certain desired range. For example, in 24-bit color images each Red, Green and Blue component can have intensity values from 0 to 255. In most instances, however, it is
• an exposure control module controls exposure time for each of the cameras or imaging sensors. It examines each input image and calculates average image intensity. Based on a moving average (i.e., average intensity of the last X number of images), the exposure control module determines whether to increase or
• the module 935 decrease exposure time.
• the module can use a longer running average to effect a slower reaction to changes in lighting conditions, with less susceptibility to unusually dark or light images (e.g., asphalt roads or water).
• the exposure control module controls exposure time for each camera separately.
• 945 maximum exposure for example is 1 millisecond.
• Certain applications of this invention involve aerial photography or surveillance. It is
• aerial images of the ground usually contain plants and vegetation - which have more consistent reflectivity than water bodies or man-made structures such as roads and buildings.
• images of plants and vegetation are usually green-dominant (i.e., the green component is the greatest of the red, green and blue values). Therefore, intensity correlation can be made more accurate by focusing on the green-dominant 955 pixels.
• the exposure control module computes the average intensity of an image by selecting only green-dominant pixels. For example, if an image has 1 million pixels and 300,000 are green-dominant, only those 300,000 green-dominant pixels are included in the calculation of average intensity. This results in an imaging process that is less
• intensity value is over 127 (i.e., over-exposed), exposure time is reduced so that less light is captured. Similarly, when intensity value is under 127 (i.e., under-exposed), exposure time is increased so that more light is captured.
• the exposure control module reduces intensity differences between input
• the mosaicing module of the present invention addresses this with an anti-
• 500, 502, 504, 506 and 508 converge from image plane 509 and cross through focal point 510 as they range across imaging target area 512 (e.g., ground terrain).
• 500 through 508 may comprise individual resolution columns of a single camera or sensor, or may represent the focal axes of a number of independent cameras or sensors.
• column 504 serves as the axis and point 513 at which column
• the exposure control module applies an anti-vignetting function multiplying the original intensity of an input pixel with a column-dependent anti-vignetting factor. Because the receiving surface is represented as a plane with a coordinate system, each column will have a number of
• the off-axis angle 514 is: zero for center column 504; larger for columns 502 and 506;
• the overall field of view angle 516 (FOVx angle) is depicted between columns 504 and 508.
• the function f(x) can be approximated by a number of line segments between columns. For a point falling within a line segment between any given columns cl and 1000 c2, an adjustment factor is computed as follows:
• Each set of input images needs to be stitched into a mosaic image. Even
• the exposure control module regulates the amount of light each camera or sensor receives
• the resulting input images may still differ in intensity.
• the present invention provides an intensity-balancing module that compares overlapping area between adjacent input images, to further balance the relative intensities. Because adjoining input images are taken simultaneously, the overlapping areas should, in theory, have identical intensity
• intensity values are usually not the same. Some such factors causing intensity difference could include, for example, the exposure control module being biased by unusually bright or dark objects present in the field of view of only a particular camera, or the boresight angles of cameras being different (i.e., cameras that are more slanted receive less light than those more vertical).
• a correlation vector (fR, fG, FB) is determined using, for example, the following process. Let V be a 3 x 1 vector representing the values (R, G and B) of a pixel:
• a correlation matrix C may be derived as:
• FR Avglr/Avgln
• Avglr Red average intensity of overlapped region in 1030 reference image
• Avgln Red average intensity of overlapped region in new image
• FG and FB are similarly derived.
• the correlation matrix scales pixel values of the secondary image so that the average intensity of the overlapping area of the secondary image becomes identical to the average intensity of the overlapping area of the reference image.
• the second image can 1035 be balanced to the reference image by multiplying its pixel values by the correlation matrix.
• a center image is considered the reference image.
• the reference image is first copied to the compound image (or mosaic). Overlapping areas between the reference
• BCM balancing correlation matrix
• I(center) Average intensity of overlapping area in center image
• Balancing factor I(center) / I(adjoining).
• the balancing factor for each color channel (i.e., red, green and blue) is independently computed. These three values form the BCM.
• the now-balanced adjoining image is copied to the mosaic. Smooth transitioning at the border of the 1050 copied image is providing by "feathering" with a mask. This mask has the same dimension as the adjoining image and comprises a number of elements. Each element in the mask indicates the weight of the corresponding adjoining image pixel in the mosaic. The weight is zero for pixels at the boundary (i.e. the output value is taken from the reference image), and increases gradually in the direction of the adjoining image until it
• the mosaic 1055 becomes unity - after a chosen blending width has been reached. Beyond the blending area, the mosaic will be entirely determined by the pixels of the adjoining image. Similarly, the overlaps between all the other constituent input images are analyzed and processed to compute the correlation vectors and to balance the intensities of the images.
• a correlation matrix is determined using, for example, the following process
• FIG. 6 depicts a strip 600 being formed in accordance with the present invention.
• V be a vector that represents the R, G and B values of a pixel:
• h be the transition width of region 608, and y be the along-track 606 distance from the boundary 610 of the overlapped region to a point A, whose pixel values are represented by V.
• V The balanced value of V, called V is:
• the mosaic can be divided into a number of segments corresponding to the position of the original input images that make up the mosaic. The process described above is applied to each segment separately to provide better local color consistency.
• pixels at the border of two segments may create 1090 vertical seams (assuming north-south flight lines).
• balancing factors for pixels in this area have to be "transitioned” from that of one segment to the other. This is explained now with reference to FIG. 7.
• FIG. 7 depicts a strip 700 being formed in accordance with the present invention.
• a base mosaic 702 and a new segment 704 overlap in area 706.
• segment 708 overlaps in area 710.
• Segments 704 and 708 overlap in area 712, and areas 706, 710 and 712 all overlap and coincide at area 714.
• point 716 serves as an origin for y-axis 718 and x-axis 720. Movement along y-axis 718 represents movement along the flight path of the imaging system. Point 716 is located at the lower left of area 714.
• the dimensions of a strip are determined by the minimum and maximum x and y values of the constituent mosaics.
• An output strip is initialized to a background color.
• a first mosaic is transferred to the strip.
• the next mosaic (along the flight path) is processed next. Intensity values of the overlapping areas of the new mosaic and the first mosaic are correlated, separately for each color
• the new mosaic is divided into a number of segments corresponding to the original input images that made up the mosaic.
• a mask matrix comprising a number of mask elements, is created for the new mosaic.
• a mask element contains the correlation matrix for a corresponding pixel in the new mosaic. All elements in the mask are initialized to unity.
• the size of the mask can be limited to just the transition area of the
• the correlation matrix is calculated for the center segment.
• the mask area corresponding to the center segment is processed.
• the values of the elements at the edge of the overlap area are set to the correlation vector.
• gradually moving away from the first mosaic along the strip the components of the correlation matrix are either increased or decreased (whether they are less or more than unity, respectively) until they 1115 become unity at a predetermined transition distance.
• the area of the mask corresponding to a segment adjoining the center segment is then processed similarly.
• the area 714 formed by the first mosaic and the center and adjoining segments of the new image requires special treatment. Because the correlation matrix for the adjoining segment may not be identical to that of the center segment, a seam may appear at the border of the two 1120 segments in the overlap area 714 with the first mosaic.
• the corner is influenced by the correlation matrices from both segments.
• its correlation matrix is the distance-weighted average of the two segments, evaluated as follows:
• VI is the balanced RGB vector based on segment 704;
• V2 is the balanced RGB vector based on segment 708;
• V is the combined (final) balanced RGB vector
• x-axis is the line going through bottom of overlapped region
• y-axis is the line going through the left side of the overlapped region between segments 704 and 708;
• 1135 h is the transition width
• d is the width of the overlapped region between segments 704 and 708.
• the mask areas corresponding to other adjoining segments are computed similarly.
• a color fidelity (i.e., white -balance) filter is applied. This multiplies R and B components with a determinable factor to
• the factor 1140 enhance color fidelity.
• the factor may be determined by calibrating the cameras and lenses.
• the color fidelity filter ensures that the colors in an image retain their fidelity, as perceived directly by the human eye.
• the Red Within the image capture apparatus, the Red,
• Green and Blue light receiving elements may have different sensitivities to the color they are supposed to capture.
• a "while-balance" process is applied - where image of a white
• pixels in the image of that white object should have equivalent R, G and B values.
• the average color values for each R, G and B may be avgR, avgG and avgB, respectively.
• the R, G and B values of the pixels are
• 1150 R values are multiplied by the ratio avgG / avgR;
• a strip In most applications, a strip usually covers a large area of non-water surface.
• the present invention provides an intensity normalization module that normalizes the average intensity of each strip so that the mean and standard deviation are of a desired value. For example, a mean of 127 is the norm in photogrammetry. A standard deviation of 51 helps to spread the intensity value over an
• Each strip may have been taken in different lighting conditions and, therefore, may have different imaging data profiles (i.e., mean intensity and standard deviation).
• This module normalizes the strips, such that all have the same mean and standard deviation. This enables the strips to be stitched together without visible seams.
• This intensity normalization comprises a computation of the mean intensity for each channel R, G and B, and for all channels. The overall standard deviation is then computed. Each R, G and B value of each pixel is transformed to the new mean and standard deviation:
• new value new mean + (old value - old mean) * (new std/old std).
• tiled mosaics for an area of interest.
• Finished tiles can correspond to the USGS quads or quarter-quads.
• Stitching strips into mosaics is similar to stitching mosaics together to generate strips, with strips now taking the role of the mosaics.
• problems may arise if the line crosses elevated structures such as buildings, bridges, etc.
• a terrain- 1180 guided mosaicing process may be implemented to guide the placement of a seam line.
• LIDAR or DEM data collected with, or analyzed from, image data may be processed to determine the configuration and shaping of images as they are mosaiced together.
• a seam line may not be a straight line - instead comprising a seam line that shifts back and forth to snake through elevated
• Process 800 begins with a series 802 of one, or more, raw collected images. Images 802 are then processed through a white-balancing process 804, transforming them into a series of
• Series 802 is then processed through anti- vignetting function 806 before progressing to the orthorectification process 808.
• orthorectification may rely on position and attitude data 810 from the imaging sensor system or platform, and on DTM data 812.
• DTM data 812 may be developed from position data 810 and from, for example, USGS DTM data 814 or LIDAR data 816.
• series 802 is converted by mosaicing module 820 into compound image 822.
• Module 820 performs the mosaicing and feathering processes during this conversion.
• one or more compound images 822 are further combined in step 824, by mosaicing with a gradient and feathering, into image strip 826.
• Image strips are
• step 830 The mosaicing performed in step 830 may comprise a terrain-guided mosaicing, relying on DTM data 812 or LIDAR data 816.
• FIG. 9 illustrates diagrammatically how photos taken with the camera array
• This embodiment shows a photo patter illustration looking down from a vehicle, using data ortho-rectified from five cameras.
• FIG. 10 is a block diagram of the processing logic according to certain embodiments of the present invention. As shown in block diagram 1000, the processing
• 1210 logic accepts one or more inputs, which may include elevation measurements 1002, attitude measurements 1004 and/or photo and sensor imagery 1006. Certain inputs may be passed through an initial processing step prior to analysis, as is shown in block 1008, wherein the attitude measurements are combined with data from ground control points. Elevation measurements 1002 and attitude measurements 1004 may be combined to 1215 generate processed elevation data 1010. Processed elevation data 1010 may then be used to generate elevation DEM 1014 and DTM 1016. Similarly, attitude measurements 1006 may be combined with photo and sensor imagery 1006 to generate georeferenced images 1012, which then undergo image processing 1018, which may include color balancing and gradient filtering.
• image processing 1018 which may include color balancing and gradient filtering.
• DTM 1016 or a USGS DEM 1022 is combined with processed images 1018 to generate ortho-rectified imagery 1024.
• Ortho-rectified imagery 1024 then feeds into self-locking flight lines 1026. Balancing projection mosaicing 1028 then follows, to generate final photo output 1030.
• the present invention may capture ortho and/or oblique image data using a
• 3D point cloud may be a representation of the ground surface including man-made structures.
• DEM digital elevation model
• the 3D point cloud or DEM may be calculated from any overlapping image data from a single camera overlapping in time or overlapping image data from any two cameras overlapping in space and/or time.
• the sequence of overlapping images may be ortho-rectified using standard photogrammetry techniques to produce an orthomap in
• each pixel has an unique latitude and longitude coordinate as discussed above and a unique elevation coordinate.
• overlapping ortho and/or oblique images are needed to determine a stereoscopic parallax and create a stereo/three dimensional view as depicted in FIG. 24.
• the overlapping images may be
• FIG. 24 depicts a sequence of overlapping oblique images obtained from an oblique camera array at two different times. Although the camera array assembly of FIG. 24 is shown, other ortho and/or oblique camera array assemblies may be used. The adjoining borders of image area 2402 and 2404 should overlap slightly. In
• the adjoining borders of image areas 2402 and 2404 overlap between about 1% and about 100%. In another embodiment, the adjoining borders of image areas 2402 and 2404 overlap between about 20% to about 70%. In another embodiment, the adjoining borders of image areas 2402 and 2404 overlap between about 50% and about 70%. In another embodiment, a sidelap area 2406 overlaps between about 20% and
• the elevation of an object may be calculated using standard stereographic techniques from overlapping ortho and/or oblique images or, alternatively, it may be obtained directly from LIDAR or a pre-existing DEM as described below.
• the principal point of each image e.g., principal point 2502 of image 2402 is located using direct
• Radial displacement is due differences in the relative distance of objects from the principal point. All objects that are away from the principal point will exhibit radial
• a stereoscopic parallax is caused by capturing images of the same object from different points of view along the flight path.
• the elevation of an object may be calculated using stereoscopic parallax:
• h is the object's elevation of height
• H' is the flight altitude
• dP is the differential parallax
• 1280 P is the average image base length.
• the elevation of the object may be calculated using overlapping oblique images:
• H' flight altitude which may be obtained by multiplying the representative fraction by the focal length of the camera
• d is the length of an object from base to top
• r is the distance from the principal point to top of object.
• FIG. 11 is an illustration of a lateral oversampling pattern 1100 looking down from a vehicle according to certain embodiments of the present invention showing minimal lateral oversampling.
• the central nadir region 1102 assigned to the center camera overlaps only slightly with the left nadir region 1104 and
• FIG. 12 is an illustration of a lateral oversampling pattern 1200 looking down from a vehicle according to certain embodiments of the present invention showing a greater degree of lateral oversampling.
• the central nadir region 1202 shows a high degree of overlap with left nadir region 1204 and right nadir region 1206.
• FIG. 13 is an illustration of a flight line oversampling pattern 1300 looking down from a vehicle according to certain embodiments of the present invention showing a certain degree of flight line oversampling but minimal lateral oversampling.
• Central nadir regions 1302 and 1304 are central nadir regions 1302 and 1304 .
• nadir regions 1306 and 1308 are overlapped to one another along the flight line, but do not overlap laterally with left nadir regions 1306 and 1308 or with right nadir regions 1310 and 1312.
• FIG. 14 is an illustration of flight line oversampling looking down from a vehicle according to certain embodiments of the present invention showing significant flight line oversampling as well as significant lateral oversampling. It can be seen that
• each of the central nadir regions 1402 through 1406 are significantly overlapped with one another as well as with left nadir regions 1408 through 1412 and right nadir regions 1414 through 1418.
• Left nadir regions 1408 through 1412 are overlapped with one another, as are right nadir regions 1414 through 1418. Accordingly, each point on the surface is sampled at least twice, and in some cases as many as four times. This technique uses the fact that in the area of an image that is covered twice, or more, by different camera sensors, a doubling of the image resolution is possible in both the lateral (across path) and flight line (along path) directions for an overall quadrupling of the resolution.
• FIG. 15 is an illustration of a progressive magnification pattern 1500 looking down from a vehicle according to certain embodiments of the present invention.
• Central nadir region 1502 is bounded on its left and right edges by inner left nadir region 1504 and inner right nadir region 1506, respectively.
• Inner left nadir region 1504 is bounded on its left edge by outer left nadir region 1508, while inner right nadir region 1506 is bounded on its right edge by outer right nadir region 1510. Note that these regions exhibit a minimal degree of overlap and oversampling from one to another.
• FTG. 16 is an illustration of a progressive magnification pattern 1600 looking down from a vehicle according to certain embodiments of the present invention.
• Central nadir region 1602 is bounded on its left and right edges by inner left nadir region 1604 and inner right nadir region 1606, respectively.
• Inner left nadir region 1604 is bounded on its left edge by outer left nadir region 1608, while inner right nadir region 1606 is bounded on its right edge by outer right nadir region 1610. Note that, as above, these regions exhibit a minimal degree of overlap and oversampling from one to another.
• Within each of the nadir regions 1604 through 1610 there is a central image region 1614 through 1620 shown shaded in grey.
• FTG. 17 is an illustration of a progressive magnification pattern 1700 looking down from a vehicle according to certain embodiments of the present invention.
• a left inner nadir region 1702 and a right inner nadir region 1704 overlap in the center.
• a left intermediate nadir region 1706 and a right intermediate nadir region 1708 are disposed partly outside of regions 1702 and 1704, respectively, 1345 each sharing an overlapping area with the respective adjacent area by approximately 50%.
• An outer left nadir region 1710 and an outer right nadir region 1712 are disposed partly outside of regions 1706 and 1708, respectively, each sharing an overlapping area with the respective adjacent area by approximately 50%.
• a central image region 1714 is disposed in the center of pattern 1700, comprised of the central portions of nadir regions
• FIG. 18 depicts a schematic of the architecture of a system 1800 according to certain embodiments of the present invention.
• System 1800 may include one or more GPS satellites 1802 and one or more SATCOM satellites 1804.
• One or more GPS location systems 1806 may also be included, operably connected to one or more modules
• a DGPS 1810 may communicate with one or more SATCOM satellites 1804 via a wireless communications link 1826.
• One or more SATCOM satellites 1804 may, in turn,
• One or more data capture system applications 1812 may interface with an autopilot 1816, an SSD and/or a RealTime StitchG system 1820, which may also interact with one another.
• SSD 1814 may be operably connected to RealTime DEM 1818.
• RealTime DEM 1818 and RealTime StitchG 1820 may be connected to a storage
• 1365 device such as disk array 1824.
• the present invention may employ a certain degree of co-mounted, co- registered oversampling to overcome physical pixel resolution limits. These co-mounted, co-registered oversampling techniques work equally well with across track camera arrays or along track camera arrays or any combination thereof.
• FIG. 19 is an
• FIG. 1370 illustration of a lateral co-mounted, co-registered oversampling configuration 1900 for a single camera array 112 looking down from a vehicle according to certain embodiments of the present invention showing minimal lateral oversampling.
• the cameras overlap a few degrees in the vertical sidelap area 1904 and 1908.
• FIG. 19 depicts a 3- camera array, these subpixel calibration techniques work equally well when utilizing any
• the camera sensors may be co- registered to calibrate the physical mount angle offset of each sensor relative to each other and/or to the nadir camera. This provides an initial, "close” calibration. These initial calibration parameters may be entered into an onboard computer system 104 in the
• rectangles labeled A, B, and C represent image areas 1902, 1906 and 1910 from a 3-camera array C-B-A (not shown). Images of areas 1902, 1906 and 1910 taken by cameras A through C (not shown), respectively, are illustrated from an overhead view. Again, similar to FIGS. 3 & 4, because of the "cross-
• the hatched areas labeled A/B and B/C sidelaps represent image 1390 overlap areas 1904 and 1908, respectively.
• the left image overlap area 1904 is where right camera A overlaps with the center/nadir camera B
• the right image overlap area 1908 is where the left camera C overlaps with the center/nadir camera B.
• the camera sensor grid bisects each pixel in the overlap areas 1904 and 1908, which effectively quadruples the image resolution in these areas 1395 1904 and 1908 via the mechanism of co-mounted, co-registered oversampling.
• this quadrupling of alignment precision between adjacent cameras 1400 improves the systems 100 alignment precision for all sensors affixed to a rigid mount plate.
• the cameras and sensors are affixed to a rigid mount unit, which is affixed to the rigid mount plate, as discussed above.
• the angular alignment of adjacent cameras affixed to the rigid mount unit is improved, the angular alignment of the other sensors is also enhanced.
• This enhancement of alignment precision for the 1405 other sensors affixed to the rigid mount plate also improves the image resolution for those sensors.
• FIG. 20 A lateral co-mounted, co-registered oversampling configuration 2000 for two overlapping camera arrays 112 is illustrated in FIG. 20. These sub-pixel calibration techniques work equally well with across track camera arrays, along track camera arrays
• FIG. 20 is an illustration of a lateral co- mounted, co-registered oversampling configuration 2000 for two overlapping camera arrays 112 looking down from a vehicle according to certain embodiments of the present invention showing maximum lateral oversampling.
• the adjacent cameras overlap a few degrees in the vertical sidelap areas 2006, 2008, 2014 and 2016, and the corresponding
• FIG. 20 depicts two 3-camera arrays
• these subpixel calibration techniques work equally well when utilizing two overlapping camera arrays with any number of camera sensors from 2 to any number of cameras being calibrated.
• the camera sensors may be co-
• each sensor 1420 registered to calibrate the physical mount angle offset of each sensor relative to each other and/or to the nadir camera.
• multiple, i.e., at least two, rigid mount units are affixed to a rigid mount plate and are co-registered. This provides an initial, "close” calibration. These initial calibration parameters may be entered into an onboard computer system 104 in the system 100, and updated during flight.
• FIG. 20 the rectangles labeled A, B, and C represent image areas 2002, 2010, 2018, and 2004, 2012, 2020 from two overlapping 3-camera arrays C- B-A (not shown), respectively. Images of areas 2002, 2010, 2018, and 2004, 2012, 2020 taken by cameras A through C (not shown) and overlapping cameras A' through C' (not shown), respectively, are illustrated from an overhead view. Again, similar to FIGS. 3 &
• the image of area 2002 is taken by right camera A
• the image of area 2010 is taken by center/nadir camera B
• the image of area 2018 is taken by left camera C.
• the image of area 2004 is taken by right camera A'
• the image of area 2012 is taken by center camera B'
• the image of area 2020 is taken by left camera C'.
• the hatched areas labeled A/B and B/C sidelaps represent two overlapping image overlap areas 2006, 2008 and 2014, 2016, respectively.
• the left image overlap areas 2006, 2008 is where right camera A overlaps with the center/nadir
• the right image overlap areas 2014 and 2016 is where the left camera C overlaps with the center/nadir camera B, and where the left camera C' overlaps with the center camera
• the camera sensor grid bisects each pixel in the overlap areas 2006, 2008 and 2014, 2016, which effectively 1445 quadruples the image resolution in these areas 2006, 2008 and 2014, 2016 via the mechanism of co-mounted, co-registered oversampling.
• the overlapping camera 1455 sensor grids bisects each pixel in the sidelap areas 2006 and 2008, which effectively quadruples the image resolution in these areas 2006 and 2008 via the mechanism of co- mounted, co-registered oversampling.
• the overlapping camera sensor grids bisects each pixel in the sidelap areas 2014 and 2016, which effectively quadruples the image resolution in these areas 2014 1460 and 2016.
• This 64 times improvement of alignment precision between adjacent and 1465 corresponding cameras enhances the systems 100 alignment precision for all sensors affixed to a rigid mount plate.
• Cameras A through C and, optionally, other sensors are affixed to a first rigid mount unit and cameras A' through C and, optionally, other sensors are affixed to a second rigid mount unit, which are each affixed to a rigid mount plate.
• the angular alignment of adjacent and/or corresponding 1470 cameras affixed to the first and/or second rigid mount units is improved, the angular alignment of the other sensors is also enhanced.
• This enhancement of alignment precision for the other sensors affixed to the rigid mount plate also improves the image resolution for those sensors.
• the overlapping grid detail labeled "OVERLAPPING GRID 4X” represents overlapping areas 2022 and 2024 in right images areas 2018 and 2020, respectively.
• one camera array is monochrome, and another camera array is red-green-blue. Even though each array covers different color bands, 1485 simple image processing techniques are used so that all color bands realize the benefit of this increased resolution. Another advantage provided by these techniques is that, in the case where one camera array is red-green-blue and the other, overlapping camera array is an infrared or near infrared (or some other bandwidth), which results in a superior multi- spectral image.
• FIG. 21 is an illustration of a fore and lateral co-mounted, co-registered oversampling configuration 2100 for two camera arrays 112 looking down from a vehicle according to certain embodiments of the present invention.
• FIG. 21 is an illustration of a fore and lateral co-mounted, co-registered oversampling configuration 2100 for two overlapping camera arrays 112 looking down from a vehicle
• FIG. 21 depicts two 3-camera arrays, these subpixel calibration techniques work equally well
• the camera sensors may be co- registered to calibrate the physical mount angle offset of each sensor relative to each other and/or to the nadir camera.
• multiple, i.e., at least two, rigid 1510 mount units are affixed to a rigid mount plate and are co-registered. This provides an initial, "close” calibration. These initial calibration parameters may be entered into an onboard computer system 104 in the system 100, and updated during flight.
• rectangles labeled A, B, and C represent image areas 2102, 2106 and 2110 from a 3-camera array C-B-A (not shown), and the rectangles
• the vertical hatched areas represent four image overlap areas 2104, 2108, 2124 and 2128.
• the rear, left image overlap area 2104 is where rear, right camera A overlaps with the center/nadir camera B
• the rear, right image overlap area 1530 2108 is where rear, left camera C overlaps with the center/nadir camera B.
• the forward, left image overlap area 2124 is where forward, right camera D overlaps with the center/nadir camera E
• the forward, right image overlap area 2128 is where forward, left camera F overlaps with the center camera E.
• the overlapping grid detail labeled "SIDELAP 1535 AREA 4: 1" represents overlaping sidelap overlap areas 2104, 2108 and 2124, 2128.
• the camera sensor grid bisects each pixel in the overlap areas 2104, 2108, 2124 and 2128, which effectively quadruples the image resolution in these areas 2104, 2108, 2124 and 2128 via the mechanism of co- mounted, co-registered oversampling.
• This quadrupling of the image resolution quadruples the alignment precision between adjacent cameras, as discussed above. [00138]
• This quadrupling of alignment precision between adjacent cameras improves the systems 100 alignment precision for all sensors affixed to a rigid mount plate.
• This enhancement of alignment precision for the other sensors affixed to the rigid mount plate also improves the image resolution for those sensors.
• the horizontal hatched areas represent three image overlap areas 2112, 2116 and 2120.
• the forward, left image overlap area 2112 is where rear, right camera A overlaps with the forward, right camera D, forward, center image overlap area
• 1555 2116 is where rear, center/nadir camera B overlaps with the forward, center camera E, and the rear, right image overlap area 2120 is where rear, left camera C overlaps with forward, left camera F.
• the camera sensor grid bisects each pixel in the overlap areas 2112, 2116 and 2120, which effectively quadruples the image resolution in these areas 2112, 2116 and 2120 via the mechanism of co-mounted, co- registered oversampling.
• This quadrupling of alignment precision between corresponding cameras improves the systems 100 alignment precision for all sensors affixed to a rigid mount plate.
• Cameras A through C and, optionally, other sensors are affixed to a first rigid mount unit and cameras D through F and, optionally, other sensors are affixed to a
• the overlapping camera sensor grids bisects each pixel in the intersecting areas 2114 and 2118, which effectively quadruples the image resolution in these areas 2114 and 2118 via the mechanism of co-mounted, co-registered oversampling.
• first rigid mount unit 1590 affixed to a first rigid mount unit and cameras D through E and, optionally, other sensors are affixed to a second rigid mount unit, which are each affixed to a rigid mount plate.
• second rigid mount unit which are each affixed to a rigid mount plate.
• one camera array is monochrome, and another camera array is red-green-blue. Even though each array covers different color bands, simple image processing techniques are used so that all color bands realize the benefit of 1600 this increased resolution. Another advantage provided by these techniques is that, in the case where one camera array is red-green-blue and the other, overlapping camera array is an infrared or near infrared (or some other bandwidth), which results in a superior multi- spectral image.
• This example is used to make the point that there are physical limits for pixel resolution in glass as well as pixel density limits for an imaging sensor.
• 1620 glass can effectively be overcome.
• a single camera array results in 1 times (or no) oversampling benefits.
• two overlapping camera arrays results in 4 times overall improvement in both image resolution and overall geospatial horizontal and vertical accuracy.
• three overlapping camera arrays results in 16 times overall improvement
• four overlapping camera arrays results in 64 times overall improvement
• N is the number of overlapping camera arrays.
• subpixel calibration techniques may be combined with the self- 1635 locking flight path techniques, as disclosed in U.S. Publication No. 2004/0054488A1, now U.S. Patent No. 7,212, 938B2, the disclosure of which is hereby incorporated by reference in full.
• the present invention may also employ flight line oversampling as 1640 well to further improve the image resolution, as shown in FIGS. 13-17.
• flight line oversampling techniques work equally well with across track camera arrays, along track camera arrays or any combination thereof.
• the flight lines overlap each other in an image region because each flight line is parallel to one another. These overlapping image regions may be used to calibrate the sensors by along-track and
• the self-locking flight path may comprise any pattern that produces at least three substantially parallel travel lines out of a group of three or more travel lines. Further, at least one of the travel lines should be in an opposing direction to the other substantially parallel travel lines.
• the travel pattern may comprise any pattern that produces at least three substantially parallel travel lines out of a group of three or more travel lines. Further, at least one of the travel lines should be in an opposing direction to the other substantially parallel travel lines.
• the travel pattern may comprise any pattern that produces at least three substantially parallel travel lines out of a group of three or more travel lines. Further, at least one of the travel lines should be in an opposing direction to the other substantially parallel travel lines.
• the travel pattern may comprise any pattern that produces at least three substantially parallel travel lines out of a group of three or more travel lines. Further, at least one of the travel lines should be in an opposing direction to the other substantially parallel travel lines.
• the travel pattern may comprise any pattern that produces at least three substantially parallel travel lines out of a group of three or more travel lines. Further, at least one of
• 1650 comprises at least one pair of travel lines in a matching direction and at least one pair of travel lines in an opposing direction.
• the self-locking flight path technique includes an algorithm to significantly reduce these positional errors.
• these positional improvements may be realized by using a pattern matching technique to automatically match a pixel pattern area obtained from a flight line (e.g., North/South) with the same pixel pattern area obtained from an adjacent flight line (e.g., North/South) with the same pixel pattern area obtained from an adjacent flight line (e.g., North/South) with the same pixel pattern area obtained from an adjacent flight line (e.g., North/South) with the same pixel pattern area obtained from an adjacent
• a flight line e.g., North/South
• the latitude/longitude coordinates from one or more GPS location systems may be used to accelerate this pattern matching process.
• subpixel calibration and self-locking flight path techniques may be combined with stereographic techniques because stereographic techniques rely
• subpixel calibration, self-locking flight path and stereographic techniques provide a greatly improved Digital Elevation Model, which results in superior image.
• these subpixel calibration and self-locking flight path techniques may be used to provide a dynamic, RealTime calibration of the system 100.
• these subpixel calibration and self-locking flight path techniques may be used to provide a dynamic, RealTime calibration of the system 100.
• these techniques provide the ability to rapidly "roll on” one or more camera array assemblies 112 onto the system 100, to immediately begin collecting image data of a target area and to quickly produce high-quality images because the individual sensors have been initially calibrated in the rigid mount unit(s) affixed to the rigid mount plate, as discussed above.
• the camera sensors are co-registered to calibrate the
• each sensor 1680 physical mount angle offset of each sensor relative to each other and/or to the nadir camera.
• multiple, i.e., at least two, rigid mount units are affixed to a rigid mount plate and are co-registered. This provides an initial, "close” calibration. These initial calibration parameters may be entered into an onboard computer system 104 in the system 100, and updated during flight using oversampling techniques, as
• the system 100 comprises a RealTime, self-calibrating system to update the calibration parameters.
• the onboard computer 104 software comprises a RealTime software "daemon" (i.e., a background closed- loop monitoring software) to constantly monitor and update the calibration parameters using
• a RealTime software "daemon" i.e., a background closed- loop monitoring software
• the RealTime daemon combines subpixel calibration, self-locking flight path and stereographic techniques to improve the stereographic image resolution and overall geospatial horizontal and vertical accuracy.
• stereographic techniques are used to match known elevation data to the
• the system 100 comprises a RealTime GPS data system to provide GPS input data. Calibration accuracy is driven by input data from electronic 1700 devices such as a GPS and an IMU, and by calibration software which is augmented by industry standard GPS and IMU software systems. Accordingly, a key component of this RealTime, self-calibrating system is a RealTime GPS input data via a potentially low bandwidth communication channel such as satellite phone, cell phone, RF modem, or similar device. Potential sources for the RealTime GPS input data include project 1705 controlled ad-hoc stations, fixed broadcast GPS locations (or similar) or inertial navigation via an onboard IMU.
• the present invention may employ anti-vibration and isothermal methods to further reduce
• FIGS. 26A and 26B illustrate an embodiment of multistage isolation of the cameras for increased accuracy.
• FIGS. 26A and 26B depict an anti-vibration member and a thermal sleeve, other anti- vibration and isothermal methods may be used. Further, although FIGS. 26 A and 26B
• 1715 depict a camera array configured in along track, cross-eyed fashion with ortho and oblique imaging sensors, these anti-vibration and isothermal techniques work equally as well with other camera arrays with ortho imaging sensors, oblique imaging sensor or any combination thereof.
• 1720 imaging sensors are arranged in along track, cross-eyed fashion. As depicted in FIG.
• mount unit 2604 comprises a simple structure inside of which imaging sensors 2606, 2608, 2610 and 2612 are disposed.
• the imaging sensors 2606 through 2614 are disposed within or along a concave curvilinear array axis 2616 in mount unit 2604 such that the focal axes of all sensors converge and intersect each other within an intersection
• oblique imaging sensor 2606 has lens 2628
• ortho imaging sensor 2608 has lens 2630
• ortho imaging sensor 2610 has lens 2632
• oblique imaging sensors has lens 2634. Vibration of the imaging sensor and lens assembly can cause vibration inaccuracies due to alignment variations of the individual
• FIGS. 26A and 26B depict anti-vibration/isothermal sleeves 2622 and 2624 for oblique imaging
• the anti-vibration/isothermal sleeves 2622 and 2624 may be identical or different depending on the specific requirements of the application.
• the anti-vibration/isothermal sleeves may be made of any material capable of vibrationally dampening and/or thermally isolating the lens.
• each lens may be secured to an
• FIG. 26B depicts anti-vibration attachment members 2636 and 2638 for oblique imaging sensor 2606 and ortho imaging sensor 2608, respectively.
• the anti-vibration attachment 2636 and 2638 may be identical or different depending on the specific requirements of the application.
• the anti- vibration attachment member may be made of any material capable of vibrationally
• this enhanced metric accuracy creates a virtual frame.
• modules and processes of the present invention may be combined together in a single functional instance (e.g., one software program), or may comprise operatively associated separate functional devices (e.g., multiple networked processor/memory blocks). All such implementations are comprehended by the present invention.

Landscapes

• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Radar, Positioning & Navigation (AREA)
• Remote Sensing (AREA)
• Studio Devices (AREA)
• Image Processing (AREA)
• Instructional Devices (AREA)
• Image Analysis (AREA)
• Traffic Control Systems (AREA)

Applications Claiming Priority (2)

Publications (2)

Family

ID=50150296

Family Applications (1)

Country Status (5)

Families Citing this family (16)

Family Cites Families (9)

• 2013
  • 2013-07-26 JP JP2015528490A patent/JP6282275B2/ja active Active
  • 2013-07-26 WO PCT/US2013/052278 patent/WO2014031284A1/en active Application Filing
  • 2013-07-26 CN CN201380053255.8A patent/CN104704424B/zh active Active
  • 2013-07-26 EP EP13831711.0A patent/EP2888628A4/de not_active Withdrawn
  • 2013-07-26 CA CA2880229A patent/CA2880229A1/en not_active Abandoned

Also Published As

Similar Documents

Legal Events