JP6282275B2 - Infrastructure mapping system and method - Google Patents

Description

<Reference to related applications>
This application claims priority to US patent application Ser. No. 13 / 590,735 filed Aug. 21, 2012 (currently issued as US Pat. No. 8,483,960). The US patent application is a continuation-in-part of US patent application Ser. No. 12 / 798,899 filed on Apr. 13, 2010, which was filed on Oct. 11, 2006. Claims priority to US patent application Ser. No. 11 / 581,235 (currently issued as US Pat. No. 7,725,258). The US patent application claims priority to US patent application Ser. No. 10 / 664,737 filed Sep. 18, 2003 (currently issued as US Pat. No. 7,127,348). The US patent application claims priority to US Provisional Patent Application No. 60 / 412,504 “Vehicle Based Data Collection and Processing System” filed on September 20, 2002. US Patent Application No. 13 / 590,735, filed on August 21, 2013, is also a continuation-in-part of US Patent Application No. 12 / 462,533, filed on August 5, 2009 This US patent application is a divisional application of US patent application Ser. No. 10 / 229,696 (currently issued as US Pat. No. 7,893,957) filed on August 28, 2002.

<Technical Field of the Present Invention>
The present invention relates generally to the field of remote imaging technology, and more particularly to a system for rendering a high resolution, high accuracy, low distortion digital image over a very wide field of view.

  Remote sensing and remote imaging are a wide range of technologies including several diverse and very important practical applications such as geological mapping, geological analysis and weather forecasting. Aviation-based and satellite-based photography and imaging are particularly useful remote imaging technologies, which recently have digital images that include parameters such as spectrum, space, elevation, vehicle or platform position and orientation. The dependence on data collection and processing is extremely high. Currently spatial data-characterizing renovation and location of real estate, public roads and highways, environmental disasters and environmental conditions, public infrastructure (eg telephone lines, pipelines) and geological features-are collected and processed in digital form And is communicated and conveniently provided for various applications (eg, dynamic GPS mapping) as high-precision mapping and monitoring data. Elevation data may be used to improve the spatial and location accuracy of the entire system, which can be obtained from an existing Digital Elevation Model (DEM) dataset or active radiation It can be collected using spectral measurement data of a measurement measuring Doppler based device or passive stereographic calculation.

  The main challenges facing remote sensing and remote imaging applications are spatial resolution and spectral fidelity. Photographic problems such as spherical aberration, astigmatism, field curvature, distortion, and chromatic aberration are well known problems that must be addressed in any sensor / imaging application. Depending on the application, very high image resolution is required and sometimes the tolerance is in inches. Depending on the system used (e.g. automobile, aircraft, satellite, spacecraft or platform), the actual digital imaging device may be located a few feet to a few miles away from its target, resulting in As a result, the scale factor becomes very large. Problems with even robust imaging systems arise when the magnification of the image is so great that the resolution tolerance is in inches. As described above, in the conventional system, the resolution quality and the size of the target area to be imaged must be produced in a trade-off relationship. If the system is designed to provide high resolution digital images, the field of view (FOV) of the imaging device is usually small. When a system is designed to produce a large FOV, typically the larger the FOV that the system provides, the lower the resolution of the spectral and spatial data, which usually increases distortion.

  Ortho-imaging is one strategy that has been used to combat this problem. In general, in ortho imaging, a composite image of a target is rendered by compiling a sub-image in which the target changes. Typically, in aeronautical imaging applications, digital imaging devices with finite range and resolution continuously record images of fixed subsections of the target area. These images are then aligned according to several sequences and a composite image of the target area is rendered.

  Such drawing processing is usually very time consuming and labor intensive. In many cases, these processes require repetitive processing, and particularly when thousands of sub-images are rendered, the image quality and resolution are significantly reduced. When image data can be processed automatically, it is often repeatedly transformed and sampled, reducing color fidelity and image sharpness with each series of operations. When an automatic correction system or an automatic adjustment system is used, such a system is susceptible to image anomalies (e.g., abnormally bright or dark targets), overcorrecting or insufficiently correcting image data, Reliability of image data interpretation is low. When manual drawing of an image is necessary or desired, the time and labor costs required are enormous.

  Therefore, there is a need for a measurement camera system that provides measurement accuracy, stability, and repeated imaging. In particular, for very large FOVs, including infrastructure, and associated data sets, orthoscopy that provides efficient and versatile imaging while maintaining image quality, accuracy, position accuracy and sharpness of measurements. There is a need for a system that draws images, and optionally diagonal images. In addition, automation algorithms are widely applied at the planning, collection, navigation, and processing stages of all relevant operations.

  The present invention relates to a remote data collection processing system using various sensors. The system may include a computer console unit that controls the operation of the moving body and the operation of the system in real time. The system may also include a global positioning system linked to and in communication with the computer console. Also, a camera and / or camera array assembly can be used that produces an image of the target that is visible through the aperture. The camera array assembly is communicatively connected to the computer console. The camera array assembly has a mounting housing, and a first image sensor is mounted in the center of the mounting housing, the first image sensor having a first focal axis passing through the aperture. Yes. The camera array assembly also includes a second image sensor that is attached to the housing and is offset from the first image sensor along an axis. The second image sensor has a second focal axis that passes through the aperture and intersects the first focal axis in the intersection region. The camera array assembly includes a third image sensor that is attached to the housing and is offset from the first image sensor along the axis and opposite the second image sensor. It is in. The third image sensor has a third focal axis that passes through the aperture and intersects the first focal axis in the intersecting region. Any number of cameras from 1 to n may be used in this way. Here, “n” may be any odd or even number.

  The system may also include an Attitude Measurement Unit (AMU), such as an inertial measurement unit, optical measurement unit, or similar measurement unit, communicatively connected to the computer console and camera array assembly. . The AMU allows the aircraft yaw, pitch and / or roll to be determined in time at any point in time, and using a continuous DGPS position, the mobile device for geodesic north orientation The heading may be measured. AMU data is integrated with precise DGPS data, resulting in a robust real-time AMU system. The system may further include a mosaicing module stored within the computer console. The mosaicing module includes a first component that performs initial processing on the input image. The mosaicing module also includes a second component that is operatively connected to the first component to determine a geographic boundary of the input image. The mosaicing module further comprises a third component that maps the input image to the composite image at an accurate geographical location. The third component is connected in a cooperative manner with the first component and the second component. The mosaicing module also includes a fourth component that adjusts the color balance of the input image mapped to the composite image. The fourth component may be operatively connected to the first component, the second component, and the third component. Furthermore, the mosaicing module may include a fifth component that blends the boundaries between adjacent input images mapped to the composite image. The fifth component can be operatively connected to the first component, the second component, the third component, and the fourth component.

  A sixth component, an optional forward oblique and / or optional rear oblique camera array system, may be provided, the system providing oblique image data. Collect and merge the posture and position measurements and image data to generate a three-dimensional image (ie, a 3D point cloud) or a digital elevation model (DEM). A 3D point cloud or DEM is an indication of the ground surface including an artificial structure. The DEM may be generated from ortho and / or oblique images using stereographic techniques, or alternatively may be derived from LIDAR or existing DEM. A DEM or 3D point cloud can be generated from any image over time obtained from one camera, or from multiple images over two cameras overlapping in space and / or time It's okay. This generation may be performed in real time within the mobile or may be post-processed thereafter. This sixth component works with other components. All components may be attached to a rigid platform mount to provide sensor data co-registration. Vibrations, turbulence, temperature gradients, and other forces can act on the moving body or platform, causing errors in the alignment relationship between the sensors. The use of a common rigid platform mount and / or thermal sleeve for the sensor provides significant advantages over other systems that do not use this co-registration configuration.

  Furthermore, the present invention improves the output quality by using lateral oversampling and / or the physical limit of pixel resolution by using co-mounted co-registered oversampling. You may overcome.

For a better understanding of the present invention and for illustrating how the invention may be implemented, reference will now be made in detail to the accompanying drawings. In addition, the corresponding number attached | subjected to different drawing is attached | subjected to the corresponding element.
FIG. 1 shows a data collection processing system using a moving body of the present invention. FIG. 1A shows a part of a data collection processing system using the mobile body of FIG. FIG. 1B shows a part of a data collection processing system using the moving body of FIG. FIG. 2 shows the camera array assembly in detail for the data collection processing system using the moving body of FIG. 1 of the present invention. FIG. 3A illustrates a camera array assembly configured in an across track, cross-eyed fashion, according to some aspects of the present invention. FIG. 3B illustrates a camera array assembly configured in a long track, cross-eyed fashion in accordance with some aspects of the present invention. FIG. 3C-1 illustrates a camera array assembly configured in an along track, cross-eyed fashion, according to some aspects of the present invention. FIG. 3C-2 illustrates a series of images obtained from a camera array assembly constructed in an along track perspective view, according to some aspects of the present invention. FIG. 3D illustrates a camera array assembly configured with an across track perspective and an along track perspective in accordance with some aspects of the present invention. FIG. 3E illustrates a camera array assembly configured with a long track perspective and an across track perspective in accordance with some aspects of the present invention. FIG. 4A illustrates one embodiment of an image pattern read by the camera array assembly of FIGS. 1 and 3A. FIG. 4B illustrates one embodiment of an image pattern read by the camera system of FIGS. 1 and 3B. FIG. 4C-1 shows an embodiment of an image pattern read by the camera system of FIGS. 1 and 3C-1. FIG. 4C-2 shows an embodiment of an image pattern read by the camera system of FIGS. 1 and 3C-2. FIG. 4D shows one embodiment of an image pattern read by the camera system of FIGS. 1 and 3D. FIG. 4D shows one embodiment of an image pattern read by the camera system of FIGS. 1 and 3E. FIG. 5 shows an image pattern representing an aspect of the present invention. FIG. 6 shows an image strip according to the invention. FIG. 7 shows another embodiment of an image strip according to the present invention. FIG. 8 illustrates one embodiment of an imaging process according to the present invention. FIG. 9 schematically shows how the photos taken with the camera array assembly are adjusted to produce individual frames. FIG. 10 is a block diagram of processing logic in accordance with certain embodiments of the present invention. FIG. 11 illustrates lateral oversampling looking down from a moving body, according to a particular embodiment of the present invention. FIG. 12 illustrates lateral oversampling looking down from a moving body, according to a particular embodiment of the present invention. FIG. 13 illustrates oversampling of a flight line, looking down from a mobile, according to a particular embodiment of the present invention. FIG. 14 illustrates oversampling of a flight line, looking down from a mobile object, according to a particular embodiment of the present invention. Figure 15 is based on a specific embodiment of the present invention and illustrates a progressive enlargement looking down from a mobile. FIG. 16 is based on a specific embodiment of the present invention and illustrates a progressive enlargement looking down from a mobile. FIG. 17 is based on a specific embodiment of the present invention and illustrates a progressive enlargement looking down from a mobile. FIG. 18 is a schematic diagram of a system configuration according to a specific embodiment of the present invention. FIG. 19 illustrates comounting in the side wrap sub-pixel region, co-register side direction oversampling, looking down from a mobile, for a single camera array, according to a specific embodiment of the present invention. FIG. 20 illustrates co-mounting in the side wrap sub-pixel region, co-register side direction oversampling oversampling, looking down from a moving body, for two overlapping camera arrays, according to a specific embodiment of the present invention. Show. FIG. 21 is based on a specific embodiment of the present invention, for a two-row overlapping stereo camera array, co-mounted in the side wrap sub-pixel region, co-register forward and fore and side, looking down from the moving body. Directional oversampling is illustrated. FIG. 22A shows a perspective view of the camera array of FIG. 3D viewed from the right rear side. FIG. 22B shows a perspective view of the camera array of FIG. 3E viewed from the front right side. FIG. 23 is a bottom view of a concave or retinal camera array assembly configured in an across track perspective and a long track perspective according to some aspects of the present invention. FIG. 24 illustrates one embodiment of an oblique camera array assembly constructed in accordance with some aspects of the present invention. FIG. 25 shows an image strip according to the invention. FIG. 26A illustrates one embodiment of a camera array assembly constructed according to some aspects of the present invention and configured in an along track perspective view. FIG. 26B shows a bottom view of the camera array assembly of FIG. 26A.

 The implementation and use of various embodiments of the present invention are described in detail below, but it is understood that the present invention provides many applicable inventive concepts and can be embodied in a wide variety of specific situations. Should be. The specific embodiments described herein are merely illustrative of specific ways to make and use the invention and do not limit the scope of the invention.

  A data collection processing system (100) using a mobile object of the present invention is shown in FIGS. 1, 1A and 1B. Further aspects and embodiments of the invention are shown in FIGS. The system (100) includes one or more computer consoles (102). The computer console includes one or more computers (104) that control both the maneuvering of the mobile and the operation of the system. Examples of functions of the computer console include control of a digital color sensor system capable of cooperating with a data collection processing system, supply of display data to a pilot, adjustment of a GPS-generated pulse per second (PPS) event trigger generated by a satellite ( Data logging, sensor control and adjustment, error event checking and alerting, photo recording and indexing, data storage for flight planning capabilities to automate mobile navigation And provide real-time display of processing and related information. The communication interface between the control computer console and the mobile autopilot controller provides the ability to actually control the flight path of the mobile in real time. Thereby, the control of the path of the moving body is more accurate than the control that can be performed by humans. All these functions can be achieved using various computer programs that are synchronized with the GPS PPS signal and take into account the various electrical latencies of the measuring device. In an embodiment, the computer is embedded in the sensor.

  One or more differential global positioning systems (106) are incorporated into the system (100). The global positioning system (106) is used to navigate and determine the exact flight path during mobile maneuvering and system operation. To accomplish this, the Global Positioning System (106) can communicate with the computer console (102) so that information from the Global Positioning System (106) can be obtained and processed without interrupting the flight. It is linked so that it can communicate. Zero or more GPS units are placed at known survey points to record each sub-second GPS satellite-based error to back correct the accuracy of the system (100). can do. GPS and / or ground based positioning services may be used to eliminate ground control points. With this technique, the position accuracy of the data collection moving body is greatly improved in units of 1 second or less.

  One or more AMUs (108) are also communicatively linked to the computer console (102) to provide real-time yaw information, real-time pitch information, and real-time roll information, which information is mobile Used to accurately determine body posture immediately after data collection. This attitude measurer (AMU) (eg, Applanix POS AV) uses three high performance fiber optic gyros, one gyro for each measurement of yaw angle, pitch angle, and roll angle. AMUs using other manufacturers' AMUs and other inertial measurement devices can be used as well. In addition, the AMU is used to determine the instantaneous attitude of the moving body, and the system provides fault-tolerance against statistical errors during AMU reading. One or more multi-frequency DGPS receivers 110 may be connected to the AMU. The multi-frequency DGPS receiver 110 can be integrated with the AMU yaw attitude data, pitch attitude data, and roll attitude data to more accurately determine the position of the remote sensor platform in the three-dimensional space. Further, the north direction of the geodesic line is determined by a vector generated by continuous DGPS positioning, and can be recorded in synchronization with the GPS PPS signal.

  One or more camera array assemblies (112) that generate images of the target viewed through the aperture are also communicatively connected to the one or more computer consoles (102). The camera array assembly (112), described in more detail below, provides a data acquisition and processing system with the capability to capture high resolution, high precision progressive scan or line scan color digital photography. To do.

  The system can also include a DC power conditioner (114) that regulates DC power, converts DC power to AC power, and provides power to the system. The system may further include a navigation display (116) that is relative to the flight plan used by the mobile operator (in-flight or at a remote location). Is graphically rendered to allow precise flight paths in horizontal and vertical planes. The system may also include an EMU module comprising a LIDAR, SAR (118) or front-back tilt camera array for capturing three-dimensional elevation / relief data. The EMU module (118) may include a laser unit (120), an EMU control unit (122), and an EMU control computer (124). Temperature control devices such as solid state cooling modules can also be deployed as needed to provide a suitable thermal environment for the system.

  The system also includes a mosaicing module (not shown) stored in the computer console (102). The mosaicing module, described in more detail below, collects data acquired by the Global Positioning System (106), AMU (108), and Camera System (112), and can use the orthomap. Gives the system the ability to process

  The system (100) can also include a self-locking flight path technique. This technology finely modifies the position accuracy of adjacent flight paths and provides the performance to achieve accuracy that exceeds the native accuracy of only the AMU sensor and the DGPS sensor.

  A complete flight planning methodology is used to plan all aspects of the mission in detail. Inputs are various mission parameters (latitude / longitude, resolution, color, accuracy, etc.). The output is a detailed online digital map and data files that are stored onboard the data collection mobile and are used for real-time navigation and alerting. The ability to interface flight plan data directly to the autopilot is an additional built-in capability. Use computer programs to automatically control flight paths, attitude adjustments, graphic displays, and moving maps of mobile paths, check warning conditions and corrective actions, and provide pilots and / or crew with overall system status By notifying, the danger of steering and control is prevented. Safe maneuver parameters should be constantly monitored and reported. While the current system uses manned crew, the system is designed to work equally well with unmanned mobile vehicles.

  FIG. 2 shows another view of the present invention. FIG. 2 shows the camera array assembly (112) in more detail. As shown, the camera array assembly 112 can acquire images from diagonally rear, diagonally front, and nadir positions.

  3A through 3E show more detailed examples of the camera array assembly of the present invention. In particular, FIGS. 3A-3E illustrate an example of an airborne camera array assembly (300) over a target (302) (eg, the ground surface). For purposes of illustration, the relative size of the assembly (300) and the relative distance between the assembly (300) and the ground surface (302) are not drawn to scale in FIGS. 3A-3E. The camera array assembly (300) includes a housing (304) in which the image sensors (306) (308) (310) (312) (314) are arranged on a concave curved axis (316). Are arranged along. In a preferred embodiment, the housing may be a mounting unit. The assembly (300) can be adjustably mounted on a moving body that moves relative to the ground surface along a path. The radius of curvature of the shaft (316) can vary significantly or can vary greatly, and the concave shape of the shaft (316) can be very small or very large. Alternatively, the shaft (316) may be perfectly straight and not bent at all. The image sensors (306), (308), (310), (312), and (314) are directly or indirectly attached to the housing (304) by attachment members (318). The attachment member (318) is fixed or movable and may include a number of permanent or temporary coupling devices. For example, the attachment member (318) may be a simple weld, a removable clamping device, or an electromechanically controlled universal joint.

  Still further, the system (100) has a real-time navigation system on board to provide a visual biofeedback display to the mobile operator or a remote display in the case of unmanned mobile maneuvers. . The pilot can adjust the position of the mobile in real time to provide a more accurate flight path. The pilot can control the moving body via the communication link using the flight display even when he is on the moving body or at a remote location.

  The system (100) may also use a fault tolerant method that will fail if the target data stored on the drive fails for one or two hard drives. Developed to provide a software inter-leaved disk storage methodology that does not lose target data. This software interleaved disk storage method approach provides superior fault tolerance and portability compared to other hardware methodologies such as RAID-5.

  The system (100) can also incorporate techniques that have been developed so that the calibration step can be performed in a short time just prior to acquisition of mission data. This calibration method step adjusts the camera settings, mainly the exposure time, by sampling the intensity of ambient light and setting it near the optimum value immediately before reaching the region of interest. A camera adjustment in seconds is then performed using a moving average algorithm, resulting in an improved consistent photo. This improves the color processing of the orthomap. Furthermore, calibration can be used to check or determine the exact spatial position of each sensor device (camera, DPG, AMU, EMU, etc.). In this way, possible changes in the spatial location of these devices are evaluated to maintain overall system precision metrics.

  In addition, by flying over an area including a plurality of known visible high-precision geographical positions, the accurate position and orientation of each sensor device (camera, DPG, AMU, EMU, etc.) on the moving body can be determined. Techniques developed so that they can be calibrated may be incorporated into the system (100). The program takes this data as input, outputs precise position data, and then uses this position data to accurately process the orthomap.

  In an embodiment of the camera array assembly (300), the image sensor may be arranged in an across track perspective view. As shown in FIG. 3A, the housing (304) is a simple-shaped housing, in which the image sensors (306) (308) (310) (312) (314) are arranged. In a preferred embodiment, the housing (304) may be replaced with a mounting unit (not shown) in which the image sensors (306) (308) (310) (312) (314) are disposed. Although FIG. 3A shows an array of five cameras, the system works equally well from 1 to any number of image sensors. The sensors (306) to (314) may be attached together by a mounting member (318) to one transverse cross member disposed between opposing walls of the housing (304), or a plurality of Individually attached to lateral cross members. In another embodiment, the housing (304) itself constitutes a supporting cross member (e.g., a mounting unit) bent into a concave shape, and the image sensor (306) to (314) is a member of the cross member. It may be attached via (318). In still other embodiments, the housing (304) can be a composite combination of a housing and a supporting cross member. In the housing (304), an aperture (320) is further formed on the surface between the image sensor and the target (302). Depending on the specific model of the mother aircraft, the aperture (320) may be only a gap or a protective screen or window to maintain environmental integrity within the housing (304). Can also be included. If a transparent protective plate is used for any sensor, a special coating may be applied to the plate to improve the quality of the sensor data. If desired, the aperture (320) may comprise a lens or other optical device to enhance or alter the quality of the image recorded by the sensor. The aperture (320) is formed in a size and shape that can give the image sensors (306) to (314) an appropriate line of sight to the target area (322) on the ground surface (302).

  The image sensors (306) to (314) are either inside the housing (304) or the housing (so that the focal axes of all sensors converge and intersect each other within the intersection region defined by the aperture (320). 304). Depending on the type of image data collected, the specific image sensor used, and other optical elements or devices used, the crossing region or focusing point may be offset above or below the aperture (320). It may be necessary or it may be desirable. The image sensors (306) to (314) are separated from each other by angular intervals. The exact displacement angle between image sensors may vary greatly depending on the number of image sensors used and the type of image data collected. Angular displacement between image sensors may also be made non-uniform to provide the desired image offset or image alignment, if desired. Depending on the number of image sensors used and the particular shape of the array, the focal axes of all image sensors may intersect exactly at the same point, or all points may be very close to each other and the aperture You may cross at a plurality of points as in the crossing region defined by (320).

  As shown in FIG. 3A, the image sensor (310) is centrally disposed along the axis (316) in the housing (304). The image sensor (310) has a focal axis (324) that is orthogonal to the housing (304) so that the line of sight of the sensor is aligned with the image area (326) of the area (322). Yes. The image sensor (308) is disposed in the housing (304) along the axis (316) next to the image sensor (310). The image sensor (308) is arranged so that its line of sight coincides with the image area (328) of the area (322), and its focal axis (330) is focused and within the area defined by the aperture (320). Is adjusted to intersect the axis (324). The image sensor (312) is disposed along the axis (316) in the housing (304) next to the image sensor (310) and opposite the sensor (308). The arrangement of the image sensor (312) is such that its line of sight coincides with the image area (332) of the area (322), and its focal axis (334) coincides with the area defined by the aperture (320). It is adjusted to intersect the axis (324) (330) within the region. The image sensor (306) is disposed in the housing (304) along the axis (316) next to the sensor (308). The image sensor (306) has a line of sight that matches the image area (336) of the area (322), and its focal axis (338) matches the area defined by the aperture (320). It has been adjusted to intersect with the other focal axis. The image sensor (314) is disposed along the axis (316) in the housing (304), next to the sensor (312) and opposite the sensor (306). The arrangement of the image sensor (314) is such that its line of sight coincides with the image area (340) of the area (322), and its focal axis (344) coincides with the area defined by the aperture (320). It is adjusted to intersect other focal axes within the region.

  Image sensors (306)-(314) may include several digital imaging devices including, for example, individual area scan cameras, line scan cameras, infrared sensors, high spectrum sensors, and / or seismic sensors. Each sensor may include a separate imaging device, or the sensor itself may include an imaging array. The image sensors (306)-(314) may all be the same or a combination of various imaging devices. In the following description, for convenience, the image sensors (306) to (314) are described as cameras (306) to (314), respectively.

  In large film cameras or large digital cameras, the typical cause of imaging problems is lens distortion. Each individual lens must be carefully calibrated to determine the correct distortion rate. In one embodiment of the present invention, a small digital camera having a lens angle width of less than 17 degrees is used. This relieves significant distortion efficiently and inexpensively.

  The focal axis of each camera is focused at the aperture (320), intersects the focal axis (324), and aligns its field of view with the target area opposite the individual positions in the array, so that the camera and imaging target The cameras (306) to (314) are arranged alternately along the axis (316) in the housing (304) such that a “retarded” retinal relationship is established between them. The camera array assembly (300) is configured such that adjacent boundaries of the image areas (326) (328) (332) (336) (340) slightly overlap. In an embodiment, the adjacent boundary between the image region (340) and the image region (332), the adjacent boundary between the image region (332) and the image region (326), the adjacent region between the image region (326) and the image region (328). The border between the image region (328) and the image region (336) overlaps between about 1% and about 99% of the image region. In another embodiment, adjacent boundaries overlap between about 10% and about 80%. In another embodiment, adjacent boundaries overlap between about 20% and about 60%.

  Depending on the shape and size of the image sensors (306) (308) (310) (312) (314), the assembly (300) provides the ability to generate an image with a generally circular customizable FOV. Depending on the mounting unit and image sensor used, the assembly (300) may be deployed to generate a stereoscopic image. In alternative embodiments, any number of mounting units including any number of image sensors having various shapes and sizes may be combined to provide image data at a desired target area.

  In another embodiment of the camera assembly (300), the image sensor may be arranged in a long track perspective view. As shown in FIG. 3B, the housing (304) has a simple enclosure in which the image sensors (306) (308) (310) (312) (314) are arranged. In a preferred embodiment, the housing (304) may be replaced with a mounting unit (not shown) in which the image sensors (306) (308) 310) (312) (314) are disposed. Although FIG. 3B shows an array of five cameras, the system works equally well if the number of camera sensors used is between 1 and any number. The sensors (306)-(314) can be attached to the mounting member (318) in a single transverse crosspiece or individually in a plurality of side crosspieces disposed between opposing walls of the housing (304). ). In an alternative embodiment, the housing (304) itself may be only a concavely curved support cross member (eg, a mounting unit), and via the member (318), the image sensors (306) through ( 314) are connected. In other embodiments, the housing (304) may be a hybrid combination of an enclosure and a support cross. The housing (304) further includes an aperture (320) formed on the surface between the image sensor and the target (302). Depending on the specific type of host aircraft, the aperture (320) may be empty only or may include a protective screen or window to maintain environmental integrity within the housing (304). Also good. If a protective transparent plate is used for any sensor, a special coating may be applied to the plate to improve the sensor data quality. Optionally, the aperture (320) may comprise a lens or other optical element to improve or change the nature of the image recorded by the sensor. The aperture (320) is formed to have a size and shape sufficient to give the image sensors (306)-(314) a proper line of sight to the target area (322) on the ground surface (302). Yes.

  The image sensors (304) to (314) are arranged so that the focal axes of all the sensors are gathered in an intersecting region having the aperture (320) as a boundary. Depending on the type of image data collected, the image sensor used, and other optical elements or devices used, it is necessary that the crossing region or focusing point be offset above or below the aperture (320). May be or may be desirable. The image sensors (304) to (314) are separated from each other by angular intervals. The exact displacement angle between the image sensors may vary greatly depending on the number of image sensors used and the type of image data collected. The displacement angle between the image sensors may also be made non-uniform to provide the desired image offset or image alignment, if desired. Depending on the number of image sensors used and the configuration of the array, the focal axes of all image sensors may intersect exactly at the same point, or they are all close to each other and the aperture ( 320) may intersect at a plurality of points within the intersecting region defined by 320).

  As shown in FIG. 3B, the image sensor (310) is centrally disposed along the axis (316) within the housing (304). The image sensor (310) has a focal axis (324) that is oriented orthogonally from the housing (304) to direct the image sensor's line of sight to the image region (326) of region (322). Aligned. The image sensor (308) is disposed along the axis (316) in the housing (304) next to the image sensor (310). The image sensor (308) is arranged so that its line of sight coincides with the image area (328) of the area (322), and its focal axis (330) is aligned within the area bounded by the aperture (320). It is aligned so that it intersects with (324). The image sensor (312) is disposed next to the image sensor (310) in the housing (304) on the opposite side of the shaft (316) from the image sensor (308). The image sensor (312) is arranged so that its line of sight coincides with the image area (332) of the area (322), and its focal axis (334) is aligned within the area bounded by the aperture (320). (324) Aligned to intersect with (330). The image sensor (306) is disposed along the axis (316) in the housing (304) next to the image sensor (308). The image sensor (306) is configured so that its line of sight coincides with the image area (336) of the area (322), and the focal axis (338) is other in the area bounded by the aperture (320). Are aligned so as to intersect with the focal axis. The image sensor (314) is disposed next to the image sensor (312) in the housing (304) on the opposite side of the axis (316) from the image sensor (306). The image sensor (314) is configured so that the line of sight coincides with the image area (340) of the area (322), and the focal axis (344) is within the area bounded by the aperture (320). Are aligned so as to intersect with the focal axis.

  The focal axis of each camera is focused at the aperture (320), intersects the focal axis (324), and aligns its field of view with the target area opposite the individual positions in the array, so that the camera and imaging target The cameras (306) to (314) are arranged alternately along the axis (316) in the housing (304) so that a “perspective” retinal relationship is established between them. The camera array assembly (300) is configured such that adjacent boundaries of the image areas (326), (328), and (332) slightly overlap. In some embodiments, the adjacent boundary between the image region (326) and the image region (328) and / or the adjacent boundary between the image region (326) and the image region (332) is about 1% of the image region and about Overlap between 99%. In another embodiment, adjacent borders overlap between about 30% and about 95%. In another embodiment, adjacent boundaries overlap between about 50% and about 60%.

  The border between the image region (328) and the image region (336) and the border between the image region (332) and the image region (340) may or may not overlap slightly. In some embodiments, the adjacent boundary between image region (328) and image region (336), and the adjacent boundary between image region (332) and image region (340) is between about 0% and about 100%. Overlap. In another embodiment, their adjacent boundaries overlap between about 30% and about 95%. In another embodiment, their adjacent boundaries overlap between about 50% and about 90%.

  In another embodiment similar to the camera array of FIG. 3B, the image sensors may be arranged in a long track perspective on the camera array (2600). As shown in FIG. 26A, the mounting unit (2604) has a simple structure in which the image sensors (2606) (2608) (2610) (2612) are arranged. Although FIG. 26A depicts an array of four cameras, the system works equally well if the number of cameras used is between 1 and any number. The sensors (2606) to (2612) can be attached to one transverse cross member arranged between opposing walls of the housing (2604) by the attachment member (2618) or to a plurality of side cross members. Can be attached individually.

  Similar to the camera assembly of FIG. 3B, the mounting unit (2604) further comprises an aperture (2620) formed on the surface between the image sensor and the target (not shown). Depending on the particular type of host aircraft, the aperture (2620) may be empty only or may include a protective screen or window to maintain environmental integrity within the mounting unit (2604). May be. If a protective transparent plate is used for any sensor, a special coating may be applied to the plate to improve the sensor data quality. Optionally, the aperture (2620) may comprise a lens or other optical element to improve or change the nature of the image recorded by the sensor. The aperture (2620) is formed to have a size and shape sufficient to give the image sensors (2606) to (2612) an appropriate line of sight to a target area on the ground surface (not shown). .

  As discussed in FIG. 3B, the image sensors (2606) through (2614) are mounted on the mounting unit (2) so that the focal axes of all the sensors converge and intersect each other within the intersection region defined by the aperture (2620). 2604) in or along the array axis (2616) forming a concave curve. Depending on the type of image data collected, the image sensor used, and other optical elements or devices used, it may be necessary to offset the intersection region or focus point above or below the aperture (2620). Sometimes desirable. The image sensors (2606) to (2612) are separated from each other by angular intervals. The exact displacement angle between image sensors may vary greatly depending on the number of image sensors used and the type of image data collected. The displacement angle between the image sensors may also be made non-uniform to provide the desired image offset or image alignment, if desired. Depending on the number of image sensors used and the shape of the array, the focal axes of all image sensors may intersect exactly at the same point, or all the points are very close together and the aperture You may cross at a plurality of points as in the crossing region defined by (2620).

 In another embodiment of the camera assembly (300), the image sensor may be arranged in a long track perspective to create an infrastructure map. As shown in FIG. 3C, the housing (304) has a simple enclosure in which the image sensors (306) (308) (310) (312) (314) are arranged. In a preferred embodiment, the housing (304) may be replaced with a mounting unit (not shown) in which the image sensors (306) (308) (310) (312) (314) are disposed. Although FIG. 3B depicts an array of five cameras, the system works equally well if the number of camera sensors used is between 3 and any number. As described above, the sensors (306) to (314) can be combined into a single transverse cross member or individually into a plurality of lateral cross members disposed between opposing walls of the housing (304). And are connected via a mounting member (318). In an alternative embodiment, the housing (304) itself may be only a concavely curved support cross member (eg, a mounting unit), and via the member (318), the image sensors (306) through ( 314) are connected. In other embodiments, the housing (304) may be a hybrid combination of an enclosure and a support cross. The housing (304) further includes an aperture (320) formed on the surface between the image sensor and the target (302). Depending on the specific type of host aircraft, the aperture (320) may be empty only or may include a protective screen or window to maintain environmental integrity within the housing (304). Also good. If a protective transparent plate is used for any sensor, a special coating may be applied to the plate to improve the sensor data quality. Optionally, the aperture (320) may comprise a lens or other optical element to improve or change the nature of the image recorded by the sensor. The aperture (320) is formed to have a size and shape sufficient to give the image sensors (306)-(314) a proper line of sight to the target area (322) on the ground surface (302). Yes.

  Similar to the camera assembly of FIG. 3B, the image sensors (306)-(314) are arranged on the housing (304) so that the focal axes of all sensors are focused in the intersection region defined by the aperture (320). Located inside or along the housing (304). Depending on the type of image data collected, the image sensor used, and other optical elements or devices used, it may be necessary to offset the intersection region or focusing point above or below the aperture (320). It may also be desirable. The image sensors (306) to (314) are separated from each other by angular intervals. The exact displacement angle between image sensors may vary greatly depending on the number of image sensors used and the type of image data collected. Angular displacement between image sensors may also be made non-uniform to provide the desired image offset or image alignment, if desired. Depending on the number of image sensors used and the particular shape of the array, the focal axes of all image sensors may intersect at exactly the same point, or all points are very close to each other. , May intersect at a plurality of points such as in the intersection region defined by the aperture (320).

  As shown in FIG. 3C, the image sensor (310) is centrally disposed along the axis (316) in the housing (304). The image sensor (310) has a focal axis (324) that is orthogonal to the housing (304) so that the line of sight of the sensor is aligned with the image area (326) of the area (322). Yes. The image sensor (308) is disposed in the housing (304) along the axis (316) next to the image sensor (310). The arrangement of the image sensor (308) is such that its line of sight coincides with the image area (328) of the area (322), and its focal axis (330) coincides with the area defined by the aperture (320). , Adjusted to intersect the axis (324) within that region. The image sensor (312) is disposed in the housing (304) next to the image sensor (310), opposite the axis (316) from the sensor (308). The image sensor (312) is arranged so that its line of sight coincides with the image area (332) of the area (322) and its focal axis (334) coincides with the area defined by the aperture (320). Is aligned to intersect the axis (324) (330). The image sensor (306) is disposed in the housing (304) along the axis (316) next to the sensor (308). Unlike the camera assembly of FIG. 3B, the image sensor (306) has its line of sight coincident with the image area (336) of the area (322) and its focal axis (338) is defined by the aperture (320). Is aligned so as to intersect with another focal axis within the region. As shown in FIGS. 3C and 4C, the image sensor (306) captures a front oblique image region (366) including a rear image of the second tower (346). The image sensor (314) is disposed in the housing (304) next to the sensor (312) on the opposite side of the shaft (316) from the sensor (306). The image sensor (314) is arranged so that its line of sight coincides with the image area (340) of the area (322), and its focal axis (344) coincides with the area defined by the aperture (320). In order to cross the other focal axis. As shown in FIGS. 3C and 4C, a rear oblique image region (340) including a front image of the first tower (346) is captured.

  Similar to the camera assembly of FIG. 3B, the focal axis of each camera is focused at the aperture (320), intersects the focal axis (324), and its field of view is a target area opposite the individual positions in the array. The cameras (306) to (314) alternate along the axis (316) in the housing (304) so that a `` perspective '' retinal relationship is established between the camera and the imaging target. Placed in. The camera array assembly (300) is configured such that adjacent boundaries of the image areas (326), (328), and (332) slightly overlap. In some embodiments, the adjacent boundary between the image region (326) and the image region (328) and / or the adjacent boundary between the image region (326) and the image region (332) is about 1% of the image region. It overlaps between about 99%. In another embodiment, adjacent borders overlap between about 30% and about 95%. In another embodiment, adjacent borders overlap between about 50% and about 90%.

  The border between the image region (328) and the image region (336) and the border between the image region (332) and the image region (340) may or may not overlap slightly. In some embodiments, the adjacent boundary between image region (328) and image region (336), and the adjacent boundary between image region (332) and image region (340) is between about 0% and about 100%. Overlap. In another embodiment, their adjacent boundaries overlap between about 30% and about 95%. In another embodiment, their adjacent boundaries overlap between about 50% and about 90%.

  Referring to FIG. 3C-2, an exemplary sequence obtained using a camera array assembly configured with an along track perspective view is shown. Although the camera array assembly of FIG. 3C-1 is shown, other along track camera array assemblies may be used. FIG. 3C-2 shows that a long track sensor collects a series of overlapping images covering the front and rear sides of a long track object, resulting in a transmission tower, insulator / conductor, transformer, and other linear shapes. It shows how a long track sensor covers objects on a lined path.

  In another embodiment of the camera array, the image sensor may be arranged in a cross-track perspective and long-track manner. See Figures 3D and 22A. FIG. 22A shows a concave or retinal camera array assembly (2200) as a rear right perspective. Although FIG. 22A shows an array of 15 cameras, the system works equally well with any number of image sensors used, from 1 to any number. The assembly (2200) is similar to the assembly (300) in configuration, structure and operation. The assembly (2200) includes a first imaging array (2202), a second imaging array (2204), and a third imaging array (2206). The array (2204) is configured as a main sensor array, the focal axis (2008) of the first image sensor (2210) is directed downward from the assembly (2200), and is a target along the ground surface (2214). Arranged in the assembly (2200) to be orthogonal to the region (2212). The assembly (2200) is adjustably mounted on a moving body that moves relative to the ground surface (2124) along the flight path (2216). The arrays (2202) (2204) (2206) are arranged in the assembly (2200) as a subassembly of the image sensor. The array (2202) is offset by an angular offset (2218) with respect to the flight path (2216) on the front side of the mounting unit (2204). Similarly, the array (2206) is offset by an angular offset (2220) with respect to the flight path (2216) on the rear side of the mounting unit (2204). The angle offset (2218) is down towards the target area (2212) so that the focal axis (2222) of the main image sensor (2224) in the mounting unit (2202) makes an angle (2232) with the target surface. Selected to face. The angle offset (2220) is down towards the target area (2212) so that the focal axis (2228) of the main image sensor (2230) in the mounting unit (2206) makes an angle (2226) with the target surface. Selected to face. The angular offset (2218) and the angular offset (2220) are preferably the same, but they may be different to provide the desired imaging effect. The focal axes of the other individual image sensors arranged in the mounting units (2202) (2204) (2206) are the same in the target area (2212) and relative to each other for their relative position along the mounting unit. Such an angular relationship may be formed.

  In some embodiments of the camera array, the image sensor may be arranged in alternative along track perspective and across track perspective. See FIGS. 3E and 22B. FIG. 22B shows the concave or retinal camera array assembly (2200) as a front right perspective. The array (2204) is configured as a main sensor array, with the focal axis (2008) of the main image sensor (2210) pointing downward from the assembly (2200) and a target area along the ground surface (2214). It is arranged in the assembly (2200) so as to be orthogonal to (2212). The mounting unit (2202) is offset by an angular offset (2218) to the left of the mounting unit (2204) (as viewed from the back) with respect to the normal (2234) to the flight path (or track) (2216) . Similarly, the mounting unit (2206) is offset by an angular offset (2220) to the right of the mounting unit (2204) (as viewed from the back) with respect to the normal (2234) to the flight path (2216). The angle offset (2218) is down towards the target area (2212) so that the focal axis (2222) of the main image sensor (2224) in the mounting unit (2202) makes an angle (2232) with the target surface. Selected to face. The angle offset (2220) is down towards the target area (2212) so that the focal axis (2228) of the main image sensor (2230) in the mounting unit (2206) makes an angle (2226) with the target surface. Selected to face. The angular offset (2218) and the angular offset (2220) are preferably the same, but they may be different to provide the desired imaging effect. The focal axes of the other individual image sensors arranged in the mounting units (2202) (2204) (2206) are the same in the target area (2212) and relative to each other for their relative position along the mounting unit. Such an angular relationship may be formed.

  In another embodiment of the camera array, the image sensor may be arranged in a concave or retinal configuration. FIG. 23 shows a concave or retinal camera array assembly (2300) in a bottom view. FIG. 23 shows an array of 25 cameras, but the system works equally well with one to any number of camera sensors in use. The assembly (2300) includes a first composite concave curved mounting unit (2302) and a plurality of composite curved mounting units (2304). The plurality of compound curved mounting units (2304) are offset at different angular intervals and are formed with a size and curvature sufficient to cover or contact the mounting unit (2302). Any number of mounting units (2304) may be used and many may exist to form a dome structure for mounting the sensor. The angular displacement between the mounting units (2304) varies depending on the size of the mounting unit and the desired image characteristics. For example, the assembly (2300) may comprise two mounting units that are orthogonal (ie, 90 °) to each other. Another assembly has three mounting units, and the angular displacement between the mounting units may be configured to be 60 °.

  The main image sensor (2306) is arranged in the center along the concave side of the mounting unit (2302), and its focal axis is directed downward so as to be orthogonal to the assembly (2300). The plurality of image sensors (2308) are arranged in a “perspective” manner along the concave side of the mounting units (2302) (2304) in accordance with the teachings of the present invention. In the perspective image sensor (2308), the focal axis of each image sensor coincides with the focal axis of the image sensor (2306) at the intersection region (not shown), and its field of view is opposite to the individual position of the array. They are arranged alternately along the mounting units (2302) and (2304) so as to align with a certain target area. Depending on the shape and size of the image sensor (2308), the assembly (2300) provides the ability to generate an image with a generally circular customizable FOV. Depending on the mounting unit and image sensor used, the assembly (2300) may be deployed to generate a stereoscopic image. In alternative embodiments, any number of mounting units including any number of image sensors having various shapes and sizes may be combined to provide image data at any desired target area.

  Referring again to FIGS. 3A-3E, when the attachment member (318) is permanently fixed (eg, welded), the spatial relationship between the aperture (320), the camera, and their line of sight is the image region (326). ) (328) (332) (336) (340) as well as the spatial relationship remains fixed. Such a structure is preferred for applications where the camera array assembly (300) is maintained at an essentially constant distance from the region (322), such as satellite surveillance. The position and adjustment of the camera are set so that the region (326) (328) (332) (336) (340) can image the entire range of the region (322). If the mounting member (318) is temporarily fixed or adjustable, the position of the camera or the image area (326) (328) (332) (336) (340) can be moved so that it is narrower or wider. Preferably, the array can be selectively adjusted by hand or remote automatic operation, which can improve or alter the quality of the images collected by the camera array assembly (300).

  In some embodiments, the rigid mounting unit may or may not be fixed to the rigid mounting plate. The attachment unit is any rigid structure to which at least one image sensor can be attached. The mounting unit is a housing surrounding the image sensor and may be any rigid structure including a brace, tripod, and the like. For the purposes of this disclosure, an image sensor means any device that can receive and process active or passive radiant energy, ie light, sound, heat, gravity, etc., from a target area. Specifically, the image sensor may be a digital camera, a LIDAR sensor, an infrared sensor, a thermal sensor, a gravitometer, or the like using a red-blue-green filter, a bushbroom filter, or a hyperspectral filter. May be provided. Image sensors do not include attitude measurement sensors such as gyroscopes, GPS devices, etc. that help to determine the orientation of moving objects utilizing satellite data and / or inertial data. A plurality of different sensors are preferably used.

  In another embodiment, a single, at least one component mounting unit may be secured to the same rigid mounting plate.

  In another embodiment, at least two component mounting units may be secured to the same rigid mounting plate.

  In embodiments where the image sensor is an image sensor such as a camera, LIDAR, etc., the mounting unit preferably has an aperture through which light and / or energy can pass. The mounting unit is preferably flat, but may not be flat. In embodiments where the image sensor is an image sensor such as a camera, LIDAR, etc., the mounting plate preferably has an aperture that is aligned with the aperture of the mounting unit through which light and / or energy can pass.

  A rigid structure is one that bends less than about 1 / 100th of a degree, preferably less than about 1/1000 of a degree, and more preferably less than about 1 / 10,000 of a degree during use. is there. Preferably, the rigid structure is less than about one-hundredth of a degree, preferably about 1,000 per degree, while being secured to a normal, ie non-turbulent, aircraft in flight. It bends less than a fraction, more preferably less than about 10,000th of a degree. If during normal operation, less than about 1 / 100th of a degree, preferably less than about 1 / 1,000th of a degree, more preferably less than about 1 / 10,000 of a degree, the object is They are firmly fixed to each other.

  Camera (310) is shown as the primary camera. The image plane (326) of the camera (310) serves as a plane of reference. The orientations of the other cameras (306) (308) (312) (314) are measured with respect to the reference plane. The relative orientation of each camera is measured by the yaw angle, pitch angle, and roll angle required to rotate the camera image plane to be parallel to the reference plane. The rotation is preferably in the order of yaw, pitch, and roll.

  The plurality of image sensors fixed to the mounting unit may not be adjusted in the same plane. Alternatively, it may be offset to the mounting angle of the other sensor relative to the mounting angle of the first sensor fixed to the first mounting unit, preferably the main nadir camera of the first mounting unit. Thus, multiple image sensors may be co-registered to calibrate each image sensor's physical mounting angle offset to each other. In an embodiment, a plurality, ie at least two rigid mounting units, are also fixed and co-registered on a plurality of rigid mounting plates. In an embodiment, the cameras (306)-(314) are fixed and co-registered on the rigid mounting unit. In an embodiment, the geometric center point of the AMU, preferably the gyroscope, is determined using GPS data and inertial data. The physical position of the first sensor fixed to the first mounting unit, preferably the main nadir camera of the first mounting unit, is calculated with respect to a reference point, preferably the geometric center point of the AMU. Similarly, the physical positions of all remaining sensors in all mounting units are calculated directly or indirectly with respect to the same reference point.

  The sensor's boresight angle is defined as the angle relative to the reference plane from the sensor's geometric center. The reference plane is preferably orthogonal to the target area. The aiming angle of the first sensor may be determined using a ground target point. The aiming angle of the next sensor is preferably calculated based on the aiming angle of the first sensor. The sensor is preferably calibrated using a known ground target that is preferably identifiable by photography, or alternatively, US 2004/0054488 A1 (currently US Pat. No. 7,212,938). B2) is calibrated using the self-locking flight path or any other method, the disclosure of which is hereby incorporated by reference in its entirety.

  The image sensor in the second mounting unit may be any image sensor, and is preferably a LIDAR. Alternatively, the second image sensor is a digital camera or a digital camera array. In the embodiment, the aiming angle of the sensor fixed to the second mounting unit is calculated based on the aiming angle of the first sensor. The physical offset of the image sensor in the second mounting unit may be calibrated with respect to the aiming angle of the first sensor in the first mounting unit.

  In this way, all sensors are calibrated at approximately the same epoch, using substantially the same atmospheric conditions, using the same GPS signal and the same ground target. This significantly reduces the compound error that occurs when each sensor is calibrated separately for different ground targets using different GPS signals under different atmospheric conditions.

  Referring to FIGS. 4A to 4E, the images of the regions (336), (328), (326), (332), and (340) captured by the cameras (306) to (314), respectively, are shown from the viewpoint from above. Yes. Again, because of the "cross-eyed" arrangement, the image of region (336) is taken by camera (306), the image of region (340) is taken by camera (314), etc. Become. In an embodiment of the present invention, images other than the image taken by the central camera (310) become trapezoid after perspective transformation. For example, see FIG. 4A. Cameras (306)-(314) form an array along axis (316), which is oriented vertically downward in most applications. In one embodiment, the cameras (308) (310) (312) form an ortho array along an axis (316) oriented vertically downward, and the cameras (306) (314) are also A diagonal array along the axis (316) is formed. For example, see FIGS. 4B and 4C. For example, information on infrastructure such as insulator / conductor, transformer, transmission line, power tower status, ground vegetation, trees / leaves, fences and road locations can be obtained for structures. In an alternative embodiment, a second array of cameras is configured similar to the array of cameras (306)-(314) and adjusted relative to the first array of cameras to produce a “heads-up ) ”An oblique view is obtained that gives a perspective projection. The tilt angle of the “head-up” camera array assembly from the horizontal plane may vary depending on the mission object and parameters, but an angle of 25-45 degrees is common. Other alternative embodiments include various camera array implementations, but are similarly included in the invention. For example, see FIGS. 4D and 4E. In all such embodiments, the relative position and orientation of the camera are accurately measured and calibrated to facilitate image processing according to the present invention.

  In one embodiment of the invention, an external mechanism (eg, a GPS timing signal) is used to trigger the camera at the same time, thereby capturing an array of input images. The mosaicing module then orthorectifies the individual input images from such an array without creating a visually recognizable seam between adjacent images (or “mosaic”). ) To draw. The mosaicing module performs a series of tasks that determine the geographical boundaries and dimensions of each input image, project each input image onto the mosaic with precise geographic positioning, and Adjusting the color balance of the images and fusing adjacent input images together at the seams they share. The exact order of tasks performed may vary depending on the size and nature of the input image data. In certain embodiments, the mosaicing module performs only one transformation on the original input image during mosaicing. This transformation can be represented by a 4 × 4 matrix. By combining a plurality of transformation matrices into a single matrix, processing time is reduced and the clarity of the original input image is preserved.

  While mapping the input image to the mosaic, especially if mosaicing is performed at high resolution, the pixels in the mosaic (ie, output pixels) may not be mapped by any pixel in the input image (ie, input pixels). There can be warped lines in the mosaic as artifacts. Certain embodiments of the present invention eliminate this problem with a supersampling system, where each of the input and output pixels is further divided into n × m grid sub-pixels. Subpixel to subpixel conversion is performed. The final value of the output pixel is the average value of the subpixels, and there is a corresponding input subpixel. As the n value and m value increase, a mosaic with higher resolution is generated, but additional processing time is required.

  While processing the image data, the mosaicing module can use the following information: The spatial position of the focal point of each camera when the input image is captured (eg, x, y, z coordinates), and the orientation of the image plane of each camera relative to the ground plane of the target area when the input image is captured (Ie, yaw, pitch, roll), field of view of each camera (ie, along the trajectory and across the trajectory), and a digital terrain model (DTM) of the region. The posture is given by an AMU linked to the system. A digital terrain model (DTM) or a digital surface model (DSM) can be generated from information obtained using the LIDAR module (118). LIDAR is similar to a widely known radar and can be considered a laser radar. In the radar, radio waves are transmitted into the atmosphere, and a part of the energy is scattered and returned to the radar receiver. LIDAR also transmits and receives electromagnetic radiation, but at a higher frequency band. This is because LIDAR operates in the ultraviolet, visible and infrared regions of the electromagnetic spectrum. In operation, the LIDAR sends light toward the target area. The transmitted light interacts with the target area and varies depending on the target area. Some of this light is reflected / scattered back to the LIDAR instrument where it can be analyzed. The characteristics of the target area can be determined by changing the characteristics of the light. The time until the light exits toward the target area and returns to the LIDAR device is used to determine the distance to the target.

  DTM and DSM data sets can also be captured from the camera array assembly. A stereographic technique etc. can also be used as a conventional means for obtaining altitude data.

  Currently, there are three basic types of LIDARs: Range finder, Differential Absorption LIDAR (DIAL), and Doppler LIDAR (Doppler LIDAR). The range finder LIDAR is the simplest LIDAR and is used to measure the distance from a LIDAR device to a solid or hard target. DIAL LIDAR is used to measure the concentration of chemical substances (ozone, water vapor, pollutants, etc.) in the atmosphere. DIAL LIDAR uses two different laser wavelengths, selecting one that is absorbed by the molecule of interest but not the other. Using the difference in intensity between the two return signals, the concentration of the molecule being investigated can be estimated. Doppler LIDAR is used to measure the velocity of the target. When the light transmitted from the LIDAR collides with a target moving toward or away from the LIDAR, the wavelength of the light reflected / scattered from the target changes slightly. This is known as a Doppler shift and hence Doppler LIDAR. If the target is moving away from the LIDAR, the return light has a longer wavelength (also called redshift), and if the target is moving closer to the LIDAR, the return light is a shorter wavelength. (Blue shift). The target may be a hard target or an air target (for example, fine dust or aerosol particles carried by wind).

  The focus of the camera is preferably used as the center of the perspective transformation. The position in the space can be determined, for example, by a multi-frequency carrier phase post-processed GPS system mounted on the parent aircraft. The three-dimensional offset of the camera focus is preferably carefully measured relative to the center of the GPS antenna. These offsets can be combined with the position of the GPS antenna and the orientation of the parent aircraft to determine the exact position of the camera focus. The position of the GPS antenna is preferably determined by processing the collected GPS data against a similar terrestrial GPS antenna deployed at an accurate survey point.

  One or more AMUs (eg, Applanix POS AV) are preferably installed in the aircraft for attitude determination. It is preferable to measure and record the attitude of the AMU reference plane with respect to the ground in the target area with a better accuracy than one hundredth of a degree at short intervals. The attitude of the AMU reference plane may be defined as a series of rotations made about the plane axis to make this plane parallel to the ground. The term “align” is sometimes used to describe this operation.

  The attitude of the central camera (310) (ie its image plane) relative to the AMU is preferably accurately calibrated. The orientation of each other camera relative to the central camera (310) is also preferably carefully calibrated. This dependent calibration is more efficient than directly calibrating each camera. If the camera array assembly (300) is remounted, only the central camera (310) needs recalibration. It is efficient to perform a series of two conversions on the input image from the central camera (310). First, the image plane of the central camera is adjusted with respect to the AMU plane. Next, the AMU plane is adjusted again with respect to the ground. However, these transformations are combined into a single operation by multiplying each transformation matrix. For the images from each of the other cameras, additional transformations are first performed to align with the image plane of the central camera.

  The position of the focus of the central camera (310) can be determined as described above. The x and y components at this position preferably determine the position of the nadir point (400) of the mosaic on the ground. Since the field of view (FOV) angle of each camera is known, the dimensions of each input image can be determined by the z component of the camera focus. The average elevation of the ground is determined by computing the average elevation point in the DTM of the area, and then each input image is projected onto the imaginary horizontal plane of this elevation. Next, it is preferable to apply relief displacement using the DTM of the region. DTMs can be obtained from many sources, including USGS 30-meter and USGS 10-meter DTMs available in most parts of the US, commercial DTMs, or mother machines that simultaneously capture data with cameras. DTM obtained by the LIDAR or SAR EMU device manufactured.

  In addition to being correctly positioned geographically, the resulting composite image also has a radiometric consistency throughout and visually at the junction between two adjacent images. There should be no recognizable seams. The present invention provides several techniques for achieving this goal.

  A characteristic of a conventional camera is the exposure time (that is, the time during which the shutter is opened to collect light on the image plane). The longer the exposure time, the brighter the resulting image. The exposure time must be adapted to changes in ambient light brightness caused by conditions such as cloud coverage, sun angle and position relative to the camera. The optimal exposure time may also depend on the camera's orientation with respect to the light source (e.g., a camera that is aimed towards a sunlit object is usually aimed towards a shaded object) Receive more ambient light than a camera with The exposure time is adjusted to maintain the average intensity of the image within a desired range. For example, in a 24-bit color image, the red, green and blue components can have an intensity value between 0 and 255. However, in most cases it is desirable to maintain the average intensity at an average value (ie 127).

  In the present invention, the exposure control module controls the exposure time of each camera or each image sensor. The exposure control module examines each input image and calculates an average image intensity. Based on the moving average (ie, the average intensity of the last X images), the exposure control module determines whether to increase or decrease the exposure time. This module is less sensitive to unusually dark or bright images (eg asphalt roads or water) because it can slow down the response to changes in lighting conditions using a longer running average. The exposure control module controls the exposure time of each camera separately.

  In systems where cameras are installed that do not have forward-motion compensation mechanisms, the maximum exposure time must be set. When the exposure time is set to a value larger than the maximum value, blurring due to motion may occur. For example, assume that the camera is mounted on an airplane that travels at 170 miles / hour (ie, about 3 inches / ms). Assume that the desired pixel resolution is 6 inches. The forward movement during image capture should be limited to half the pixel size, in this case 3 inches. For this reason, the maximum exposure in this example is 1 millisecond.

  When controlling the quality of imaging, changes in light intensity are caused by changes in ambient light, whether it is an unusually bright or dark object (e.g. a reflecting water body, metallic It is useful to be able to determine whether it is caused by the presence of roofs, asphalt, etc.). Applications of the present invention include shooting or monitoring in the air. Aerial images of the ground are usually observed to contain plants and vegetation, which have a more constant reflectivity than water bodies or artificial structures such as roads and buildings. Of course, images of plants and vegetation are usually dominated by green (ie, the green component is the largest among the red, green and blue values). Therefore, the intensity correlation can be made more accurate by focusing on the green dominant pixels.

  The exposure control module computes the average intensity of the image by selecting only those pixels where green is dominant. For example, if the image has 1 million pixels and green is dominant for 300,000 pixels, only those 300,000 pixels that are dominant in green are included in the average intensity calculation. As a result, the imaging process is less sensitive to biasing caused by man-made buildings and water bodies where green is usually not dominant for pixels. As mentioned above, it is desirable to maintain an intensity value of about 127. If the intensity value exceeds 127 (ie, overexposure), the exposure time is shortened and less light is captured. Similarly, if the intensity value is below 127 (ie, underexposure), the exposure time is lengthened and more light is captured. For example, consider a system that flies over a very high target terrain area with many white roofs. The average intensity of the captured image will tend to be high. In most conventional systems, the exposure time will be shortened to compensate. However, in the above example, it is not appropriate to shorten the exposure time. The reason is that the average intensity of the image is biased by the brightness of the roof. Shortening the exposure time will result in an image where the ground is darker than it actually is. In contrast, if only green dominant pixels are processed according to the present invention, pixels representing an overly bright roof will bias the average intensity and the exposure time will not change.

  For this reason, the exposure control module reduces the intensity difference between the input images. Nevertheless, further processing is performed to improve the tonal balance. There are several factors that receive non-uniform light from the image plane (eg, lens physics, atmospheric conditions, spatial / positional relationship of the imaging device). The central part of the camera or sensor receives more light than the edge.

  The mosaicing module of the present invention addresses this with an anti-vignetting function, which will be described with reference to FIG. Several focal columns (500) (502) (504) (506) (508) are focused from the image plane (509) and intersect at the focal point (510) to target the area (512) to be imaged (E.g., the ground). Columns (500)-(508) may be individual resolution columns of a single camera or sensor, or may represent the focal axes of several independent cameras or sensors. For reference purposes, the column (504) functions as an axis, and the point (513) where the column (504) intersects the image plane (509) functions as the main position. The exposure control module applies the anti-vignetting function and multiplies the original intensity of the input pixel by the column-dependent anti-vignetting factor. Since the light receiving surface is represented as a plane having a coordinate system, each column has several resolution rows (not shown). This relationship can be expressed as follows for column x and pixel y in row y:

<Adjusted intensity> = <Original intensity> * f (x);
Where f (x) is a function of the form:
f (x) = cos (off-axis angle) ** 4
The off-axis angle (514) is zero for the central column (504), larger for the columns (502) and (506), and even larger for the columns (500) and (508). The overall viewing angle (516) (FOV x angle) is shown between column (504) and column (508).

The function f (x) can be approximated between columns by several line segments. For any point within the line segment between column c1 and column c2, the adjustment factor is computed as follows:
<Adjustment coefficient of c> = f (c1) + [f (c2) −f (c1) * (c−c1) / (c2−c1)];
However, f (c1) and f (c2) are f function values of off-axis angles of the column c1 and the column c2, respectively.

  Each set of input images needs to be stitched into the mosaic image. Even if the exposure control module adjusts the amount of light received by each camera or sensor, the resulting input image may remain different in intensity. The present invention provides an intensity balance module that compares overlapping areas between adjacent input images and performs further balance adjustment of relative intensity. Since adjacent input images are obtained simultaneously, the overlap region should theoretically have the same intensity in both input images. However, due to various factors, the intensity values are usually not the same. Some factors that cause intensity differences include, for example, that the exposure control module is biased by unusually bright or dark objects present in the field of view of only a particular camera, and the camera's boresight angle is different ( That is, a camera with a large inclination has a smaller amount of received light than a more vertical camera.

To balance two adjacent images, one is selected as a reference image and the other is a secondary image. For example, the correlation vector (fR, fG, FB) is determined using the following process. Let V be a 3 × 1 vector representing pixel values (R, G and B):
The correlation matrix C can be derived as follows:
However, FR = AvgIr / AvgIn, AvgIr is the average red intensity of the overlapping area in the reference image, AvgIn is the average red intensity of the overlapping area in the new image, and FG and FB are similarly derived.

  The correlation matrix scales the pixel values of the secondary image so that the average intensity of the overlapping area of the secondary image is the same as the average intensity of the overlapping area of the reference image. The second image can be balanced with the reference image by multiplying the pixel value by the correlation matrix.

For this reason, in one embodiment of the balance processing according to the present invention, the central image is the reference image. The reference image is first copied to the composite image (or mosaic). The overlap area between the reference image and the adjacent image (for example, the image immediately to the left) is associated with each other, and a balance correlation matrix (BCM) is calculated. The BCM is multiplied by a vector representing the pixels of the adjacent image in order to match the intensity of the overlap region in both images. One embodiment of this relationship can be expressed as:
I (center) = average intensity of the overlapping area of the center image,
I (adjacent) = average intensity of overlapping parts of adjacent images,
Balance coefficient = I (center) / I (adjacent).

  The balance factor for each color channel (ie, red, green and blue) is calculated separately. These three values form the BCM. The adjacent image subjected to the balance process is copied to the mosaic. A smooth transition at the boundary of the copied image is made by “feathering” with a mask. This mask has the same size as the adjacent image and includes several elements. Each element in the mask indicates the weight of the corresponding neighboring image pixel in the mosaic. The weight is 0 for the border pixels (ie, the output value is taken from the reference image), and in the direction of the adjacent image until it reaches 1 (unity) after reaching the chosen blending width. Increase gradually. Beyond the mixed region, the mosaic is completely determined by the pixels of the adjacent image. Similarly, the overlapping portion between all other constituent input images is analyzed and processed, the correlation vector is calculated, and the image intensity is balanced.

For example, the correlation matrix is determined using the following process (see FIG. 6). FIG. 6 shows a strip (600) produced in accordance with the present invention. A basic mosaic (602) and a new mosaic (604) are added along the path (or trajectory) (606) and overlap each other in the region (308). Let V be a vector representing the R, G, and B values of the pixel:
The pixel value is represented by V, where h is the transition width of the region (608) and y is the distance along the trajectory (606) from the boundary (610) of the overlapping region to the point A. . Let C be the correlation matrix:
V 'the balanced V value:
If 0 <y <h, V ′ = [y / h.I + (1−y / h) .C] × V;
If y> = h, V ′ = V;
Where I is the identity matrix:
It should be noted that the “feathering” technique is also used in combination with a gradient to reduce the visually perceived seam.

  When the mosaic is long, the intensity difference between the overlapping portions may change from one end of the mosaic to the other. It may not be possible to compute a single correlation vector so that there is no visually recognized seam. The mosaic can be divided into several segments corresponding to the positions of the original input images that make up the mosaic. The above process is applied to each segment separately so that the colors become more locally consistent.

  Under such a precise algorithm, pixels at the boundary of two segments (assuming north-south flight paths) can produce vertical seams. In order to avoid this problem, the balance factor of the pixels in this region must be “translated” from the balance factor of one segment to the balance factor of the other segment. This will be described with reference to FIG.

  FIG. 7 shows a strip (700) produced in accordance with the present invention. The basic mosaic (702) and the new segment (704) overlap in the region (706). The mosaic (702) and another new segment (708) overlap in the region (710). Segment (704) and segment (708) overlap in region (712), and regions (706), (710), and (712) all overlap in region (714) and match. For convenience of explanation, the point (716) is the origin of the y-axis (718) and the x-axis (720). Movement along the y-axis (718) represents movement along the flight path of the imaging system. The point (716) is located at the lower left of the region (714).

According to the invention, the dimension of the strip is determined by the minimum value x and the maximum value y of the constituent mosaic. The output strip is initialized to the background color. The first mosaic is transferred to the strip. Next, the adjacent mosaic (along the flight path) is processed. The intensity value of the overlapping area of the new mosaic and the first mosaic has an individual correlation for each color channel. The new mosaic is divided into several segments corresponding to the original input image that made up the mosaic. A mask matrix containing several mask elements is generated for the new mosaic. The mask element contains a correlation matrix for the corresponding pixel in the new mosaic. All elements in the mask are initialized to 1. The size of the mask can be limited only to the transition area of the new mosaic. The mask area corresponding to the central segment is processed. The value of the element at the end of the overlapping area is set in the correlation vector. Next, gradually moving away from the first mosaic along the strip, the component of the correlation matrix is increased or decreased (smaller or larger than 1 respectively) until it becomes 1 at a predetermined transition distance. Next, the area of the mask corresponding to the segment adjacent to the central segment is processed in the same manner. However, the region (714) formed by the first mosaic and the central and adjacent segments of the new image require special processing. Since the correlation matrix for the adjacent segment is not the same as the correlation matrix of the central segment, a seam may appear at the boundary between the two segments in the overlapping region (714) with the first mosaic. Therefore, the corner is affected by the correlation matrix of both segments. For a mask cell A at a distance x to the boundary with the central segment and a distance y to the overlapping edge, the correlation matrix is a distance-weighted average of the two segments, as follows: Rated by:
For the pixel A (x, y) having the distance x to the boundary of the central segment in the region (714), the balance-adjusted value is calculated as a distance weighted average of the values calculated using the two segments. Is;
V1 is an RGB vector that is balanced based on segment (704);
V2 is an RGB vector that is balanced based on segment (708);
V ′ is the sum of the balanced (final) RGB vectors V ′ = ((d−x) / d) .V1 + (x / d) .V2
Is;
The x-axis is a line that passes through the bottom of the overlap region,
The y-axis is a line that passes through the left side of the overlapping area of segment (704) and segment (708),
h is the transition width,
d is the width of the overlapping region of the segment (704) and the segment (708).
Mask areas corresponding to other adjacent segments are similarly calculated.

  Further, in the present invention, a color fidelity (ie, white balance) filter is applied. This filter multiplies the R and B components by a determinable factor to increase color fidelity. This factor can be determined by calibrating the camera and lens. The color fidelity filter ensures that the colors of the image retain their fidelity as perceived directly by the human eye. In an image capture device, red, green, and blue light receiving elements may differ in sensitivity to the color that is to be captured. A “white balance” process is applied when an image of a white object is captured. In theory, the pixels of the white object image should have the same R, G, and B values. In practice, however, the average color values for R, G, and B may be avgR, avgG, and avgB, respectively, due to sensitivity differences and other factors. To equalize the color components, the R, G, and B values of the pixels are multiplied by the following ratio:

R value is multiplied by the ratio avgG / avgR;
The B value is multiplied by the ratio avgG / avgB.
The final result is set so that the RGB components of the image of the white object are equal.

  In most applications, the strip usually covers a large non-water surface area. Thus, the average strength of the strip is less likely to be distorted by anomalies such as highly reflective surfaces. The present invention provides an intensity normalization module that normalizes the average intensity of each strip so that the average and standard deviation are the desired values. For example, the average value 127 is the standard for photogrammetry. The standard deviation 51 is useful for extending the intensity value over an optimum range for visually recognizing image features. Each strip may have a different image data profile (ie, average intensity and standard deviation) since it may be taken in different lighting conditions. This module normalizes the strips so that they all have the same mean and standard deviation. Thus, the strips are stitched together so that the seam is not visible.

This intensity normalization includes the calculation of the average intensity for each of the R, G and B channels and all channels. Thereafter, the overall standard deviation is calculated. Each R, G, and B value for each pixel is converted to a new mean and standard deviation:
New value = new average + (old value-old average) * (new standard deviation / old standard deviation)

  Next, a plurality of adjacent strips are combined to generate a tiled mosaic of the target area. Completed tiles can accommodate USGS quads or quarter-quads. Stitching strips into a mosaic is similar to stitching mosaics together to create a strip, where the strip plays the role of a mosaic. A seam line between two strips can cause problems if the line crosses an elevated structure such as a building or bridge. This is a classic problem in photogrammetry and arises from the parallax caused by the same object seen from two different viewpoints. During building imaging, for example, one strip represents a view from one side of the building, while another strip represents a view from the other side of the building. After the images are stitched together, the resulting mosaic may look like a tent tepee. In order to deal with this problem, a terrain-guided mosaicing process is performed to guide the placement of seam lines. For example, LIDAR data and DEM data collected together with image data, or LIDAR data and DEM data analyzed from image data may be processed, and when they are mosaiced together, the configuration and shape of the image may be determined. For this reason, in some mosaic images, the seam line is not a straight line but includes a seam line that shifts back and forth so as to meander the elevated structure.

  Referring to FIG. 8, there is shown one embodiment of an imaging process (800) according to the invention described above. The process (800) begins with a series of one or more collected raw images (802). The image (802) is then processed by a white balance adjustment process (804) and converted into a series of intermediate images. Next, the series of images (802) is processed by the anti-vignetting function (806) before proceeding to the orthorectification process (808). As described above, ortho correction may depend on position and orientation data (810) from the image sensor system or platform and DTM data (812). The DTM data (812) may be created from position data (810), for example, USGS DTM data (814) or LIDAR data (816). The series of images (802) is then orthocorrected, followed by color balance adjustment (818). After the color balance adjustment, the series of images (802) is converted into a composite image (822) by the mosaicing module (820). Module (820) performs mosaicing and feathering during this conversion. Here, the one or more composite images (822) are further combined in step (824) into image strips (826) by gradient mosaicking and feathering. The image strip is processed by intensity normalization (828). The normalized strip (828) is then re-mosiced in step (830) by mosaicing and feathering including gradients to render the final tile mosaic (832). The mosaicing performed in step (830) may include terrain-guided mosaicing based on DTM data (812) or LIDAR data (816).

  FIG. 9 schematically shows how photographs taken with the camera array assembly can be adjusted to produce individual frames. In the present embodiment, ortho-corrected data from five cameras is used, and a photo patter diagram viewed from a moving body is shown.

  FIG. 10 is a block diagram of processing logic in accordance with certain embodiments of the present invention. As shown in block diagram (1000), processing logic receives one or more inputs. One or more inputs may include elevation measurements (1002), attitude measurements (1004), and / or photo and sensor imagery (1006). As shown in block (1008), prior to analysis, certain inputs may be passed to an initial processing step where attitude measurements are combined with data from ground control points. The altitude measurement value (1002) and the posture measurement value (1004) are combined to generate processed altitude data (1010). The processed elevation data (1010) is then used to generate elevation DEM (1014) and DTM (1016). Similarly, posture measurements (1006) are combined with photo and sensor images (1006) to generate a georeferenced image (1012), which is then subjected to image processing (1018). Is done. Image processing (1018) may include color balancing and gradient filtering.

  Depending on the data set used (1020), either DTM (1016) or USGS DEM (1022) is combined with the processed image (1018) to produce an orthocorrected image (1024). The orthocorrected image (1024) is then sent to the self-locking flight line (1026). This is followed by balancing projection mosaicing (1028) to produce the final photo output (1030).

  The present invention may capture ortho and / or oblique image data using a camera array as shown in FIGS. 3A-3E, 22A-22B, and 23, ie, a three-dimensional image (ie Image data may be merged using posture and position measurements to generate a 3D point cloud) or digital elevation model (DEM). As described above, the D3 point cloud or DEM is a representation of the ground surface including an artificial structure. The D3 point cloud or DEM may be generated from ortho and / or oblique images using stereographic techniques, or alternatively may come directly from the LIDAR or existing DEM. A DEM or 3D point cloud can be generated from any image over time obtained from one camera, or from multiple images over two cameras overlapping in space and / or time It's okay. An orthomap may be generated by orthorectifying a series of overlapping images using common photogrammetry techniques, where each pixel has its own latitude and lightness as described above. It has coordinates and unique elevation coordinates.

  In order to calculate the elevation of an object, the overlapping ortho and / or oblique images (ie, stereoscopic images) determine the stereoscopic parallax, resulting in a stereoscopic / three-dimensional view as shown in FIG. Is needed to. Overlapping images may be obtained in time overlap from one camera, or may be obtained in space and / or time overlap from two cameras. FIG. 24 shows a series of overlapping oblique images obtained from the oblique camera array at two different times. Although the camera array assembly of FIG. 24 is shown, other ortho and / or oblique camera array assemblies may be used. The border between the image area (2402) and the image area (2404) should be slightly overlapped. In some embodiments, adjacent borders of image area (2402) and image area (2404) overlap between about 1% and about 100%. In another embodiment, adjacent borders of image area (2402) and image area (2404) overlap between about 20% and about 70%. In another embodiment, adjacent borders of image area (2402) and image area (2404) overlap between about 50% and about 70%. In another embodiment, the side wrap region (2406) overlaps between about 20% and about 30% and the fore wrap region (2408) overlaps between about 50% and about 70%.

  As described below, the elevation of interest may be calculated from overlapping ortho and / or oblique images using stereographic techniques, or alternatively, may be obtained directly from LIDAR or an existing DEM. . The principal point of each image (for example, the principal point (2502) of image (2402)) determines the virtual corner fiduciary coordinates after correcting the yaw, pitch, and roll by the AMU. Or using aerial triangularization techniques (eg, upper left and lower right (UL-LR) and upper right and lower left (UR-LL)), as shown in FIG. Placed. Conjugate principal points are principal points of adjacent images and are similarly arranged. In a series of oblique images, a principal point and a conjugate principal point are arranged on a straight line. The line between the principal point and the conjugate principal point is the flight path (or track).

  Projections in the center of the image result in image displacements in which the objects are displaced or displaced from their correct positions. Relief displacement due to altitude is due to differences in the relative height of objects from a particular surface. All objects extending above or below a particular ground surface will exhibit an image displacement proportional to the height of the object. The higher the object, the greater the distortion of the photo due to altitude. High objects can show image displacement even from high altitudes.

  The radial displacement is due to the difference in the relative distance of the object from the principal point. All objects that are distant from the principal point exhibit a radial displacement proportional to the distance of the object from the principal point. The greater the distance, the greater the radial displacement.

Stereoscopic parallax occurs by capturing images of the same object from different viewpoints along the flight path. The elevation of the object may be calculated using stereoscopic parallax.
h = (H ′) * dP / (P + dP)
Here, h is the height of the altitude of the object, H ′ is the flight altitude, dP is the difference in parallax, and P is the average image base length.

Alternatively, the elevation of the object may be calculated using overlapping diagonal images.
h = d * (H ′) / r
Here, h is the height of the altitude of the object, H ′ is the flight altitude, and may be obtained by multiplying the focal length of the camera by a fractional scale, and d is the object from the base to the vertex. , R is the distance from the principal point to the vertex of the object.

  The present invention uses some degree of lateral oversampling to improve output quality. FIG. 11 is a diagram of a lateral oversampling pattern (1100) looking down from a mobile according to a particular embodiment of the present invention, showing minimal lateral oversampling. In this figure, the central nadir area (1102) assigned to the central camera overlaps only slightly with the left nadir area (1104) and the right nadir area (1106), but this overlap is minimized. ing. FIG. 12 is a diagram of a lateral oversampling pattern (1200) looking down from a mobile, according to a particular embodiment of the present invention, showing a greater degree of lateral oversampling. In this figure, the central nadir area (1202) overlaps with the left nadir area (1204) and the right nadir area (1206) to a considerable extent.

  In addition to using the lateral oversampling shown in FIGS. 11 and 12, the present invention may use flight line oversampling as well. FIG. 13 is a diagram of a flight line oversampling pattern (1300) looking down from a mobile according to a particular embodiment of the present invention, showing some flight line oversampling and minimal side oversampling. ing. The central nadir area (1302) (1304) overlaps with each other along the flight line, but the left nadir area (1306) (1308) or the right nadir area (1310) (1312) Does not overlap.

  FIG. 14 is a flight line oversampling view looking down from a mobile, according to a particular embodiment of the present invention, showing significant flight line oversampling and significant lateral oversampling. Each of the central nadir areas (1402) to (1406) overlaps each other, and further, the left nadir areas (1408) to (1412) and the right nadir areas (1414) to (1418). I understand that. The left nadir areas (1408) to (1412) overlap each other, and the same applies to the right nadir areas (1414) to (1418). Thus, each point on the surface is sampled at least twice and possibly four times. This technique doubles the image resolution in the lateral direction (direction crossing the path) and the flight line direction (direction along the path) in an image area that is covered more than once by different camera sensors. Can be doubled. In practice, the improvement in image / sensor resolution is somewhat less than twice in each dimension, approximately 40% in each dimension, or 1.4 × 1.4 = approximately twice. This is due to statistical variations in subpixel alignment / orientation. In fact, the pixel grid is rarely exactly equidistant from the overlapping pixel grid. If very precise lateral camera sensor alignment is performed at the subpixel level, a fourfold increase in image resolution could be achieved.

  FIG. 15 is a diagram of a progressive magnification pattern (1500) looking down from a mobile according to a particular embodiment of the present invention. The central nadir region (1502) is bordered by the left inner nadir region (1504) and the right inner nadir region (1506) at the left and right edges, respectively. The left inner nadir area (1504) borders the left outer nadir area (1508) at its left edge, while the right inner nadir area (1506) has its right outer edge at its right edge. It borders the bottom region (1510). It should be noted that these regions have a minimal degree of overlap and oversampling with each other.

  FIG. 16 is a diagram of a progressive enlargement pattern (1600) looking down from a mobile object, according to a particular embodiment of the present invention. The central nadir area (1602) is bounded by the left inner nadir area (1604) and the right inner nadir area (1606) at the left edge and the right edge, respectively. The left inner nadir area (1604) borders the left outer nadir area (1608) at its left edge, while the right inner nadir area (1606) is the right outer edge at its right edge. It borders the nadir area (1610). As noted above, it should be noted that these regions have a minimal degree of overlap and oversampling with each other. Inside the nadir areas (1604) to (1610) are the central image areas (1614) to (1620) shown shaded in gray, respectively.

  FIG. 17 is a diagram of a progressive enlargement pattern (1700) looking down from a mobile object, according to a particular embodiment of the present invention. In the center of the pattern (1700), the left inner nadir region (1702) and the right inner nadir region (1704) overlap at the center. The left middle nadir area (1706) and the right middle nadir area (1708) are partially arranged outside the area (1702) and the area (1704), respectively, and each area overlaps each adjacent area. About 50% share. The left outer nadir region (1710) and the right outer nadir region (1712) are partially disposed outside the region (1706) and the region (1708), respectively, and each region overlaps each adjacent region. About 50% share. The central image region (1714) is arranged at the center of the pattern (1700), and is composed of the central portions of the nadir regions (1702) to (1712).

  FIG. 18 shows a schematic diagram of the configuration of the system (1800), based on a specific embodiment of the present invention. The system (1800) may include one or more GPS satellites (1802) and one or more SATCOM satellites (1804). One or more GPS location systems (1806) may also be included, the system (1806) being operatively connected to one or more modules (1808), the module (1808) comprising LIDAR, Collect GPS and / or X, Y, Z position data and supply this information to one or more data capture system applications (1812). The DGPS (1810) communicates with one or more SATCOM satellites (1804) through a wireless communication link (1826). One or more SATCOM satellites (1804) can in turn communicate with one or more data capture system applications (1812).

  One or more data capture system applications (1812) may interface with the autopilot (1816), SSD and / or RealTime StitchG system (1820), which may also interact with each other. The SSD (1814) may be operatively connected to the RealTime DEM (1818). Finally, RealTime DEM (1818) and RealTime StitchG system (1820) may be connected to a storage device such as a disk array (1824).

  The present invention can overcome the physical limitations of pixel resolution with some use of co-mounted and co-registered oversampling. These co-mounted, co-registered oversampling techniques work equally well with across track camera arrays, along track camera arrays, or any combination thereof. FIG. 19 is a diagram of a side-coordinated, co-registered oversampling arrangement (1900) for a single camera array (112) looking down from a mobile, according to a particular embodiment of the present invention. There is minimal lateral oversampling. The cameras overlap slightly in the vertical sidelap regions (1904) (1908). FIG. 19 shows an array of three cameras, and these sub-pixel calibration techniques work equally well when using any number of camera sensors from two to any number of cameras being calibrated.

  Similar to the image sensors of FIGS. 3 and 4, the camera sensors can be co-registered to calibrate the physical offset of each sensor's mounting angle between each sensor and / or with respect to the nadir camera. This provides an initial “close” calibration. These initial calibration parameters can be input into the onboard computer system (104) of the system (100) and updated using oversampling techniques during flight.

  Referring to FIG. 19, the rectangles labeled A, B and C represent image areas (1902) (1906) (1910) from an array C-B-A (not shown) of three cameras. ing. Images of regions (1902), (1906), and (1910) photographed by cameras A to C (not shown) are shown as bird's-eye views. Again, as in FIGS. 3 and 4, because of the “cross-eyed” arrangement, the image of region (1902) is taken by right camera A, and the image of region (1906) is center / top. The image is taken by the bottom camera B, and the image of the area (1910) is taken by the left camera C. Cameras A to C form an array (not shown) that is oriented vertically downward in most applications.

  In FIG. 19, hatched areas with A / B and B / C side wraps represent image overlapping areas (1904) and (1908), respectively. The left image overlap area (1904) is an area where the right camera A overlaps the center / nadir camera B, and the right image overlap area (1908) is an area where the left camera C overlaps the center / nadir camera B. It is an area. In these side wrap regions (1904) (1908), the camera sensor grid bisects each pixel in the overlap region (1904) (1908), thereby providing a co-mounted, co-registered oversampling mechanism. Thus, the image resolution of these areas (1904) (1908) is effectively quadrupled. In practice, the image / sensor resolution is improved by a factor of 2 in each dimension, ie 2 × 2 = 4 times. By improving this image resolution by 4 times, the alignment accuracy between adjacent cameras is also improved by 4 times.

  Furthermore, by increasing the alignment accuracy between adjacent cameras by a factor of 4, the alignment accuracy of the system (100) is improved for all sensors fixed to the rigid mounting plate. As described above, the camera and the sensor are fixed to the rigid mounting unit fixed to the rigid mounting plate. Specifically, when the angle alignment of adjacent cameras fixed to the rigid mounting unit is improved, the angle alignment of other sensors is also improved. By improving the alignment accuracy of other sensors fixed to the rigid mounting plate, the image resolution of these sensors is also improved.

  An oversampling arrangement (2000) co-mounted and co-registered laterally for two overlapping camera arrays (112) is shown in FIG. These sub-pixel calibration techniques work equally well with across track camera arrays, along track camera arrays, or any combination thereof. Specifically, FIG. 20 is based on a specific embodiment of the present invention, looking down from a moving body and co-mounted and co-registered laterally for two overlapping camera arrays (112). FIG. 6 is a diagram of an oversampling arrangement (2000) showing maximum lateral oversampling. Adjacent cameras overlap slightly in the vertical side wrap area (2006) (2008) (2014) (2016), and the corresponding cameras are image areas (2002) (2010) (2018) (2004) In (2012) (2020) it is completely overlapping. Although FIG. 20 shows a two-column array of three cameras, these subpixel calibration techniques work equally well with any number of camera sensors from two to any number of cameras being calibrated. Function.

  Similar to the image sensors of FIGS. 3 and 4, the camera sensors can be co-registered to calibrate the physical offset of each sensor's mounting angle between each sensor and / or with respect to the nadir camera. In this embodiment, a plurality, ie at least two rigid mounting units, are fixed to the rigid mounting plate and co-registered. This provides an initial “precise” calibration. These initial calibration parameters can be input into the onboard computer system (104) of the system (100) and updated during flight.

  Referring to FIG. 20, the rectangles labeled A, B, and C are image regions (2002) (2010) from an array C-B-A (not shown) of three cameras in two overlapping rows. ) (2018) and (2004) (2012) (2020) respectively. Of regions (2002) (2010) (2018) and (2004) (2012) (2020) taken by cameras A to C (not shown) and overlapping cameras A ′ to C ′ (not shown), respectively. The image is shown as an overhead view. Here, as in FIGS. 3 and 4, since the arrangement is “perspective”, the image of the area (2002) is taken by the right camera A, and the image of the area (2010) is taken by the center / nadir camera B. The image in the area (2018) is taken by the left camera C. Further, the image of the area (2004) is taken by the right camera A ', the image of the area (2012) is taken by the central camera B', and the image of the area (2020) is taken by the left camera C '. Cameras A to C and overlapping cameras A 'to C' form an array (not shown) that is oriented vertically downward in most applications.

  In FIG. 20, hatched areas with A / B and B / C side wraps represent two overlapping image overlapping areas (2006) (2008) and (2014) (2016), respectively. The left image overlap area (2006) (2008) is an area where the right camera A overlaps the center / nadir camera B and the right camera A 'overlaps the center camera B'. The right image overlap area (2014) (2016) is an area where the left camera C overlaps with the center / nadir camera B and the left camera C 'overlaps with the center camera B'. In each of these side wrap regions (2006) (2008) and (2014) (2016), the grid of the camera sensor cuts off each pixel in the overlap region (2006) (2008) and (2014) (2016). Through the co-mounted and co-registered oversampling mechanism, the image resolution in these regions (2006) (2008) and (2014) (2016) is effectively quadrupled. In practice, the image / sensor resolution is improved by a factor of 2 in each dimension, ie 2 × 2 = 4 times. As described above, by improving the image resolution by 4 times, the alignment accuracy between adjacent cameras is also improved by 4 times.

  By including two overlapping camera arrays, the image resolution is effectively quadrupled for overlapping side wrap overlap regions (2006) (2008) and (2014) (2016). This surprisingly provides an overall 64 times improvement in system (100) calibration and camera alignment.

  In the overlapping side wrap area (2006) (2008), the overlapping camera sensor grid cuts each pixel in the side wrap area (2006) (2008), thereby co-mounted and co-registered. Through the oversampling mechanism, the image resolution of these regions (2006) (2008) is effectively quadrupled. Similarly, in the overlapping side wrap region (2014) (2016), the overlapping camera sensor grid cuts each pixel in the side wrap region (2014) (2016), and thus these regions ( The image resolution of 2014) (2016) is effectively quadrupled. In practice, the image / sensor resolution is again improved by a factor of 2 in each dimension, ie 2 × 2 × 2 × 2 × 2 × 2 = 64 times. By improving the overall image resolution by 64 times, the alignment accuracy between adjacent cameras is also improved by 64 times.

  By improving the alignment accuracy between adjacent and corresponding cameras by 64 times, the alignment accuracy of the system (100) is improved for all sensors fixed to the rigid mounting plate. Cameras A to C and optionally other sensors are fixed to the first rigid mounting unit, and cameras A ′ to C ′ and optionally other sensors are fixed to the second rigid mounting unit. Each mounting unit is fixed to a rigid mounting plate. Specifically, when the angular alignment of adjacent cameras and / or corresponding cameras fixed to the first and / or second rigid mounting unit is improved, the angular alignment of other sensors is also improved. By improving the alignment accuracy of other sensors fixed to the rigid mounting plate, the image resolution of these sensors is also improved.

  By including two overlapping camera arrays, the image resolution is effectively quadrupled for the entire image, not just for the overlap area of the A / B and B / C side wraps. Referring to FIG. 20, the overlapping grid details with “overlapping grid 4 ×” represent the overlapping areas (2022) and (2024) of the right image areas (2018) and (2020), respectively. In the overlapping area (2022) (2024), the overlapping camera sensor grid cuts each pixel in the overlapping area (2022) (2024), thereby co-mounted and co-registered oversampling mechanism. Accordingly, the image resolution of these regions (2022) and (2024) is effectively quadrupled. In practice, the image resolution is improved by a factor of 2 in each dimension, ie 2 × 2 = 4 times.

  In a preferred embodiment, one camera array is for monochrome and the other camera array is for RGB (red-green-blue). Even if each array covers a different color band, the advantage of improving this resolution in all color bands can be realized using simple image processing techniques. Another advantage provided by these technologies is that if one row of camera arrays is for RGB and the other overlapping camera array is for infrared or near infrared (or even some other bandwidth), it is excellent A multispectral image can be obtained.

  Accordingly, all identified improvements (ie, 4 ×) for the embodiment of FIG. 19 described above apply to the embodiment of FIG. 20, but to the calibration accuracy and overall image resolution of the system (100). On the other hand, a further great improvement (ie 64 times) can be realized with two overlapping camera arrays.

  FIG. 21 is based on a specific embodiment of the present invention and shows a two-column camera array (112) looking down from a mobile body in a front and side co-mounted, co-registered oversampling arrangement (2100). FIG. Specifically, FIG. 21 shows a two-row overlapping camera array (112) looking down from a mobile, according to a particular embodiment of the present invention, co-mounted and co-registered over and forward. FIG. 4 is a diagram of a sampling arrangement (2100) showing minimal forward and minimal lateral oversampling. Adjacent cameras overlap slightly in the vertical side wrap areas (2104) (2108) (2124) (2128), and the corresponding cameras in the horizontal side wrap areas (2112) (2116) (2120) A little overlap along. Although FIG. 21 shows a two-row array of three cameras, these sub-pixel calibration techniques have any number of camera sensors of any number of two or more cameras being calibrated and overlapped. This also works well when using two rows of camera arrays.

  Similar to the image sensors of FIGS. 3 and 4, the camera sensors can be co-registered to calibrate the physical offset of each sensor's mounting angle between each sensor and / or with respect to the nadir camera. In this embodiment, a plurality, ie at least two rigid mounting units, are fixed to the rigid mounting plate and co-registered. This provides an initial “precise” calibration. These initial calibration parameters can be input into the onboard computer system (104) of the system (100) and updated during flight.

  Referring to FIG. 21, the rectangles labeled A, B, and C respectively represent image areas (2102), (2106), and (2110) from an array C-B-A (not shown) of three cameras. The rectangles D, E, and F represent image regions (2122), (2126), and (2130) from an array FE-D (not shown) of three cameras, respectively. Images of areas (2102), (2106) and (2110) respectively photographed by cameras A to C (not shown), and areas (2122) and (2126) (2126) (each photographed by cameras D to F (not shown)) 2130) is shown as an overhead view. Here, as in FIG. 3 and FIG. 4, since the arrangement is “perspective”, the left rear image of the area (2102) is taken by the right rear camera A, and the central rear image of the area (2106) is The image is taken by the rear center / nadir camera B, and the right rear image of the area (2110) is taken by the left rear camera C. Further, the left front image of the area (2122) is taken by the right front camera D, the center front image of the area (2126) is taken by the center front camera E, and the right front image of the area (2020) is taken by the left front camera F. Taken. Cameras A to C and overlapping cameras D to F form an array (not shown) that is oriented vertically downward in most applications.

  In FIG. 21, vertical hatched areas represent four image overlapping areas (2104) (2108) (2124) (2128). In the left rear image overlapping area (2104), the right rear camera A overlaps the center / nadir camera B, and in the right rear image overlapping area (2108), the left rear camera C is center / top. This is an area overlapping the bottom camera B. In the left front image overlap area (2124), the right front camera D overlaps with the center / nadir camera E, and in the right front image overlap area (2128), the left front camera F overlaps with the center camera E. It is an area.

  Referring to FIG. 21, the details of the overlapping grid to which “side wrap region 4: 1” is attached represent overlapping side wrap overlapping regions (2104) (2108) (2124) (2128). In these side wrap overlap areas (2104) (2108) (2124) (2128), the camera sensor grid cuts each pixel in the overlap area (2104) (2108) (2124) (2128), thereby comounting Thus, the image resolution of these regions (2104) (2108) (2124) (2128) is effectively quadrupled through the co-registered oversampling mechanism. In practice, the image / sensor resolution is improved by a factor of 2 in each dimension, ie 2 × 2 = 4 times. As described above, by improving the image resolution by 4 times, the alignment accuracy between adjacent cameras is also improved by 4 times.

  By improving the alignment accuracy between adjacent cameras by a factor of four, the alignment accuracy of the system (100) is improved for all sensors fixed to the rigid mounting plate. Cameras A to C and optionally other sensors are fixed to the first rigid mounting unit, and cameras D to F and optionally other sensors are fixed to the second rigid mounting unit. Are each fixed to a rigid mounting plate. Specifically, if the angle alignment of the adjacent camera is fixed to the first or second rigid mounting unit, the angle alignment of other sensors fixed to the mounting unit is also improved. . By improving the alignment accuracy of other sensors fixed to the rigid mounting plate, the image resolution of these sensors is also improved.

  Similarly, the horizontal hatched area represents three image overlapping areas (2112) (2116) (2120). The left front image overlap area (2112) is an area where the right rear camera A overlaps the right front camera D, and the center front image overlap area (2116) is the center / nadir camera B on the rear side in the center front side. The right rear image overlap area (2120) is an area where the left rear camera C and the left front camera F overlap.

  Referring to FIG. 21, the details of the overlapping grid to which “forelap region 4: 1” is attached represent overlapping forelap overlapping regions (2112), (2116), and (2120). In these forelap overlap areas (2112), (2116), (2120), the camera sensor grid bisects each pixel in the overlap area (2112), (2116), (2120), thereby co-mounted and co-registered over-range. Through the sampling mechanism, the image resolution of these regions (2112) (2116) (2120) is effectively quadrupled. In practice, the image / sensor resolution is improved by a factor of 2 in each dimension, ie 2 × 2 = 4 times. By improving the image resolution by 4 times, the alignment accuracy between corresponding cameras is also improved by 4 times.

  By increasing the alignment accuracy between corresponding cameras by a factor of four, the alignment accuracy of the system (100) is improved for all sensors fixed to the rigid mounting plate. Cameras A to C and optionally other sensors are fixed to the first rigid mounting unit, and cameras D to F and optionally other sensors are fixed to the second rigid mounting unit. Are each fixed to a rigid mounting plate. Specifically, when the angle alignment of the corresponding camera fixed to the first or second rigid mounting unit is improved, the angle alignment of other sensors is also improved. By improving the alignment accuracy of other sensors fixed to the rigid mounting plate, the image resolution of these sensors is also improved.

  Similar to the overlapping side wrap overlap regions (2006) (2008) and (2014) (2016) of FIG. 20, the intersecting fore wrap and side wrap overlap regions (2114) (2118) of FIG. What should be done is an overall improvement of 64 times in system configuration and camera alignment. Referring to FIG. 21, the cross grid details labeled “Quad overlap region 64: 1” represent the intersecting fore wrap and side wrap overlap region (2118). In the intersecting fore-wrap and side-wrap overlap area (2114) (2118), the grid of overlapping camera sensors bisects each pixel in the intersection area (2114) (2118), thereby co-mounted and co-registered. Through the oversampling mechanism, the image resolution of these regions (2114) and (2118) is effectively quadrupled. In practice, the image / sensor resolution is again improved by a factor of 2 in each dimension, ie 2 × 2 × 2 × 2 × 2 × 2 = 64 times. By improving the overall image resolution by 64 times, the alignment accuracy between adjacent cameras is also improved by 64 times.

  By improving the alignment accuracy between adjacent and corresponding cameras by 64 times, the alignment accuracy of the system (100) is improved for all sensors fixed to the rigid mounting plate. Cameras A to C and optionally other sensors are fixed to the first rigid mounting unit, and cameras D to E and optionally other sensors are fixed to the second rigid mounting unit. Are each fixed to a rigid mounting plate. Specifically, when the angular alignment of adjacent cameras and / or corresponding cameras fixed to the first and / or second rigid mounting unit is improved, the angular alignment of other sensors is also improved. By improving the alignment accuracy of other sensors fixed to the rigid mounting plate, the image resolution of these sensors is also improved.

  In a preferred embodiment, one camera array is for monochrome and the other camera array is for RGB. Even if each array covers a different color band, the advantage of improving this resolution in all color bands can be realized using simple image processing techniques. Another advantage provided by these technologies is that if one row of camera arrays is for RGB and the other overlapping camera array is for infrared or near infrared (or some other bandwidth), Multispectral images can be obtained.

  As shown in FIGS. 19-21, these techniques can be used to overcome the resolution limitations imposed on the camera system due to the inability of optical glass to resolve “very small” objects. . As mentioned above, these techniques work equally well with across track camera arrays, along track camera arrays, or any combination thereof. In particular, it is known that the optical glass of a camera lens has physical limitations on the ability to resolve very small objects. This is often referred to as the “glass resolution limit”. For example, if a pixel of 1 millimeter is required from an altitude of 10,000 feet (3,048 meters), a ground width of about 100 feet (30.48 meters) using a very high magnification telephoto lens. It would be necessary to get swath). This can give any number of pixels with a charge-coupled device sensor (eg, 100 million pixels), but even at the resolution of the purest glass at 10,000 feet (3,048 meters) at 1 millimeter This is because image resolution of pixels cannot be made possible. This example is used to clarify the existence of image sensor pixel density limits as well as the physical limits of pixel resolution in glass.

  The calibration technique associated with the image sensor alignment of the system (100) in a rigid mounting unit mounted on a rigid mounting plate provides a unique solution to this problem, as described above. By using these techniques, the resolution limit of glass can be effectively overcome. For example, a single camera array provides a benefit of 1 × (or none) oversampling. However, the overlapping two-row camera array provides an overall increase of four times in both image resolution and overall horizontal and vertical accuracy. Furthermore, the overlapping three-row camera array provides an overall 16-fold improvement, and the overlapping four-row camera array provides an overall 64-fold improvement.

As can be inferred from these examples, the overall improvement equation is:
Overall improvement = 4 ^N
N is the number of overlapping camera arrays.

If there are 4 rows of camera arrays, there are 3 rows of overlapping camera arrays (ie, N = 3). Thus, a four-row camera array provides an overall increase of 64 times (ie, 4 ³ = 64 times) in both image resolution and global spatial and horizontal accuracy.

  In addition, these subpixel calibration techniques can be combined with the self-locking flight path technique disclosed in U.S. Patent Application Publication No. 2004 / 0054488A1 (currently U.S. Pat. No. 7,212,938B2). All of the disclosures of which are incorporated herein by reference.

In addition to the front-facing and / or side-coordinated and co-registered oversampling shown in FIGS. 19-21, the present invention also provides for further improving image resolution, as shown in FIGS. Flight line oversampling can also be used. These flight line oversampling techniques work equally well with across track camera arrays, along track camera arrays, or any combination thereof.
As shown in FIGS. 13 to 17, the flight lines overlap each other in the image area because the flight lines are parallel to each other. Using these overlapping image regions, the sensor can be calibrated by parallax along the trajectory and across the trajectory in images in adjacent flight lines by stereographic techniques.

  In an embodiment, the self-locking flight path may be any pattern that generates at least three generally parallel travel lines from a group of three or more travel lines. Furthermore, at least one travel line should be in a direction opposite to other substantially parallel travel lines. In a preferred embodiment, the progression pattern may be at least one pair of progression lines in a matching direction and at least one pair of progression lines in opposite directions.

  When using a self-locking flight path in the opposite direction, the observable position error may be doubled in some image areas. Thus, the self-locking flight path technique includes an algorithm for greatly reducing these position errors. This reduction in position error is particularly important in the outer region where the maximum position error occurs, i.e., the left and right "wing" image regions.

  In embodiments, these positional enhancements can be obtained from adjacent flight lines (eg, north / south) using pixel pattern regions obtained from flight lines (eg, north / south) using pattern matching techniques. It can be realized by automatically matching with the same pixel pattern region. In a preferred embodiment, this pattern matching process can be accelerated using latitude / longitude coordinates from one or more GPS location systems.

  Similarly, these sub-pixel calibration and self-locking flight path techniques may be combined with stereographic techniques. This is because stereographic technology relies heavily on the positional accuracy of each pixel relative to all other pixels. Specifically, these techniques provide stereographic image resolution and total geospatial horizontal and vertical accuracy, especially in the left and right "wing" image areas where maximum position errors occur. And improve. In addition, stereographic techniques are used to match known elevation data to an improved stereographic data set. Thus, a combination of sub-pixel calibration, self-locking flight path, and stereographic techniques provides a significantly improved digital elevation model and provides superior images.

  Furthermore, these sub-pixel calibration and self-locking flight path techniques can be used to provide dynamic real-time calibration of the system (100). Specifically, these techniques quickly “roll on” one or more camera array assemblies (112) onto the system (100) and immediately begin collecting image data of the target area, It provides the ability to quickly generate high quality images. This is because, as described above, the individual sensors are initially calibrated in a rigid mounting unit that is fixed to the rigid mounting plate. Specifically, the camera sensors are co-registered to calibrate the physical offset of each sensor's mounting angle between each sensor and / or with respect to the nadir camera. In one embodiment, the plurality, ie at least two rigid mounting units, are fixed to the rigid mounting plate and co-registered. This provides an initial “precise” calibration. These initial calibration parameters are input to the onboard computer system (104) of the system (100) as described above and can be updated using oversampling techniques during flight.

  In certain embodiments, the system (100) includes a real-time self-calibration system for updating calibration parameters. In particular, the onboard computer (104) software uses real-time to continuously monitor and update calibration parameters using co-mounted, co-registered oversampling and flight line oversampling techniques, as described above. Includes software “daemon” (ie, background closed-loop monitoring software). In a preferred embodiment, the real-time daemon improves the stereographic image resolution and horizontal and vertical accuracy of the entire geospace by combining subpixel calibration, self-locking flight path, and stereographic techniques. Specifically, stereographic techniques are used to match known elevation data with an improved stereographic data set. Thus, a combination of sub-pixel calibration, self-locking flight path, and stereographic techniques provides a significantly improved digital elevation model and provides superior images.

  In some embodiments, the system (100) includes a real-time GPS data system for providing GPS input data. Calibration accuracy is facilitated by input data from electronic devices such as GPS and IMU, and by calibration software augmented by industry standard GPS and IMU software systems. Thus, an important component of this real-time self-calibration system is real-time GPS input data via a potentially low-bandwidth communication channel such as a satellite phone, cell phone, RF modem, or similar device. Potential sources of real-time GPS input data include project-controlled ad hoc stations, fixed broadcast GPS locations (or the like), or inertial navigation via onboard IMUs.

  In addition to the front and / or side co-mounted and co-registered oversampling as shown in FIGS. 19-21 and the flight line oversampling shown in FIGS. 13-17, the present invention provides: Anti-vibration and isothermal techniques may be used to further reduce image collection inaccuracy within a single camera and image synthesis inaccuracy between two or more cameras. For example, FIGS. 26A and 26B illustrate an example of multi-stage separation of cameras to increase accuracy. 26A and 26B show a vibration isolation member and a thermal sleeve, but other vibration isolation and thermostat techniques may be used. Further, FIG. 26A and FIG. 26B are provided with an ortho and oblique image sensor, and show a camera array configured in an along-track perspective type. Using other camera arrays with sensors, or any combination thereof, works equally well.

  In the above embodiment of the camera array assembly (2600), the image sensor is arranged in an along track perspective view. As shown in FIG. 26A, the mounting unit (2604) has a simple structure, and the image sensors (2606) (2608) (2610) (2612) are arranged therein. The image sensors (2606) to (2614) are mounted inside the mounting unit (2604) or mounted so that the focal axes of all the sensors converge within the intersecting area defined by the aperture (2620) and intersect each other. Arranged along the unit (304).

  As shown in FIG. 26B, the oblique image sensor (2606) has a lens (2628), the ortho image sensor (2608) has a lens (2630), and the ortho image sensor (2610) has a lens ( 2632), and the oblique image sensor has a lens (2634). Vibration of the image sensor and lens assembly causes vibration inaccuracies due to individual component alignment variations. Furthermore, thermal expansion and contraction of the image sensor and lens assembly cause thermal inaccuracies due to temperature gradients. In order to minimize these vibrational and thermal inaccuracies, a vibration / thermal sleeve may be placed around each lens and / or a vibration member is used to secure each lens. May be. 26A and 26B show anti-vibration / isothermal sleeves (2622) (2624) for the oblique image sensor (2606) and the ortho image sensor (2608), respectively. The anti-vibration / isothermal sleeves (2622) (2624) may be the same or different depending on the specific specifications of the application. The anti-vibration / isothermal sleeve may be made of any material that can damp vibrations and / or thermally insulate the lens.

  Further, in order to minimize vibration inaccuracy, each lens may be fixed to a vibration isolating member (2626) as shown in FIG. 26A. FIG. 26B shows anti-vibration mounting members (2636) (2638) for the oblique image sensor (2606) and the ortho image sensor (2608), respectively. The anti-vibration mounting members (2636) and (2638) may be the same or different depending on the specific specification of the application. The anti-vibration mounting member may be made of any material that can damp vibrations and / or thermally insulate the lens.

 Using co-mounted, co-registered over-sampling techniques, sub-pixel calibration techniques, flight line over-sampling techniques, anti-vibration techniques, isothermal techniques, or any combination thereof, as if a single image Images obtained from a plurality of cameras or image sensors can be synthesized with the same accuracy as if obtained from a plurality of cameras or image sensors. In the preferred embodiment, this increased metric accuracy results in virtual frames.

  The above modules, algorithms and processes can be implemented in many techniques and configurations. Embodiments of the invention may include functional examples relating to software or hardware, or a combination thereof. Further, the modules and processes of the present invention may be combined together in a single functional example (eg, a software program), and are operably linked to separate functional devices (eg, multiple networked processors / Memory block). All such implementations are included in the present invention.

  The embodiments and examples described herein are shown to best explain the invention and its practical application, thereby enabling those skilled in the art to make and use the invention. However, one of ordinary skill in the art will appreciate that the foregoing description and examples have been presented for purposes of illustration and illustration only. The foregoing description is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations may be made in light of the above teachings without departing from the spirit and scope of the appended claims.

Claims (21)

A system for generating a map of a target area,
A global positioning receiver,
An altitude measuring unit that can be adjustably mounted on a moving object or platform;
A global positioning antenna that can be adjustably mounted on a moving object or platform;
A posture measuring unit that can be adjusted and attached to a moving object or a platform;
An image sensor system that can be adjustably mounted on a moving body or platform and that looks at a target area;
A computer, and
The image sensor system includes an attachment unit, and at least three image sensors are arranged in the attachment unit. Each of the first image sensor, the second image sensor, and the third image sensor is attached to the image sensor system. Having a focal axis through the first aperture of the unit, the first image sensor generates a first image region of the target region comprising a first data array of pixels, and the second image sensor is a second of the pixels. A second image area of the target area comprising a data array is generated, a third image sensor producing a third image area of the target area comprising a third data array of pixels, and the second image sensor And offset from the first image sensor along the first array axis to have a first image overlap region in the target region, and the third image sensor Are offset from the first image sensor along the first array axis on the opposite side of the image sensor,
The computer communicates with the altitude measurement unit, the global positioning antenna, the attitude measurement unit, the first image sensor, the second image sensor, and the third image sensor, and the altitude measurement unit, the global positioning antenna, and the attitude measurement unit. based on input from one or more, the first image sensor, a second image sensor, and at least a portion of the image region of the third image sensor, and associated with a portion of the target area,
The mounting unit includes one vibration isolation member having a second aperture corresponding to each of the first image sensor, the second image sensor, and the third image sensor, and the first image sensor, the second image sensor, and Each focal axis of the third image sensor passes through a corresponding second aperture,
Each image sensor has a lens assembly corresponding to that image sensor , each lens assembly having its front end with an anti-vibration or thermal sleeve surrounding the lens assembly barrel. The system is inserted into the second aperture corresponding to the above and attached to the vibration isolating member .   From the first image sensor, the second image sensor, and the third image sensor based on input from one or more of the three-dimensional image calculation from the overlapping image data, LIDAR data, or digital elevation model The system of claim 1, wherein at least a portion of the image elevation is associated with a portion of the target elevation. The system of claim 1, wherein the lens of each lens assembly is provided with an anti-vibration mounting member . The system according to claim 1, wherein the vibration isolation member is provided so as to straddle the first aperture. A fourth image sensor is further disposed in the mounting unit, the fourth image sensor having a focal axis through the first aperture of the mounting unit, and the fourth image sensor is the fourth data of the pixel. 2. A fourth image region of a target region comprising an array is generated, wherein the fourth image sensor is offset from the second image sensor along a first array axis on the opposite side of the third image sensor. The system described in. A fourth image sensor disposed within the mounting unit, the fourth image sensor having a focal axis through the first aperture of the mounting unit, the fourth image sensor comprising a fourth data array of pixels. generates a fourth image region of the target region comprising, fourth image sensor is offset from the first image sensor along the second array axis a long the trap perspective formula, and a fourth image sensor,
A fifth image sensor disposed within the mounting unit, the fifth image sensor having a focal axis through the first aperture of the mounting unit, the fifth image sensor comprising a fifth data array of pixels. A fifth image sensor, wherein the fifth image sensor is offset from the first image sensor along the second array axis on the opposite side of the fourth image sensor; ,
Is further provided,
The system according to claim 1, wherein the second image sensor, the third image sensor, the fourth image sensor, and the fifth image sensor are oblique image sensors.   The system of claim 1, wherein the first image sensor and the second image sensor are selected from the group consisting of a digital camera, LIDAR, infrared, thermal sensing and gravitometer.   The system of claim 5, wherein the third image sensor and the fourth image sensor are selected from the group consisting of a digital camera, LIDAR, infrared, thermal sensing, and gravimeter.   The system of claim 1, wherein the first image overlap region overlaps between about 1% and about 100% of the first image region and the second image region.   The system of claim 1, wherein the first image overlap region overlaps between about 10% and about 80% of the first image region and the second image region.   The system of claim 1, wherein the first image overlap region overlaps between about 20% and about 60% of the first image region and the second image region. The third image sensor is offset from the first image sensor along the first array axis and has a second image overlap region in the target region, and the grid of the first image sensor is in the first image overlap region. The image data of the second image sensor is divided into two parts, the grid of the first image sensor cuts the image data of the third image sensor into two parts in the second image overlapping area, and the first image overlapping area is separated from the first image area. The system of claim 1, wherein the second image area is approximately 100% and the second image overlap area is approximately 100% of the first image area and the third image area. The second image sensor is offset from the first image sensor along the first array axis along track perspective formula grid of the first image sensor, the image data of the second image sensor in the first image overlapping area 6. The system according to claim 5, wherein   From the first image sensor, the second image sensor, and the third image sensor based on input from one or more of the three-dimensional image calculation from the overlapping image data, LIDAR data, or digital elevation model The system of claim 13, wherein at least a portion of the image elevation of the is associated with a portion of the target elevation.   The system of claim 13, wherein the third image sensor is a front oblique image sensor and the fourth image sensor is a rear oblique image sensor.   The system of claim 13, wherein the first image overlap region overlaps between about 1% and about 100% of the first image region and the second image region.   The system of claim 13, wherein the first image overlap region overlaps between about 30% and about 95% of the first image region and the second image region.   The system of claim 13, wherein the first image overlap region overlaps between about 50% and about 90% of the first image region and the second image region. An image sensor system,
It is equipped with a mounting unit that can be adjusted and attached to a moving body or platform.
A first image sensor, a second image sensor, a third image sensor, and a fourth image sensor are disposed in the mounting unit, and the first image sensor, the second image sensor, the third image sensor, and the fourth image sensor. Each having a focal axis through the first aperture of the mounting unit, the first image sensor producing a first image area comprising a first data array of pixels, and the second image sensor comprising: A second image area comprising a second data array is generated, a third image sensor producing a third image area comprising a third data array of pixels, and a fourth image sensor comprising a fourth data array of pixels. generates a fourth image region, the second image sensor is offset from the first image sensor in Across track perspective formula, it has a first image overlapping area in the target area, the first image sensor Grid is to bisect the image data of the second image sensor in the first image overlapping area,
The mounting unit includes one anti-vibration member having a second aperture corresponding to each of the first image sensor, the second image sensor, the third image sensor, and the fourth image sensor. The focal axes of each of the image sensor, the third image sensor, and the fourth image sensor pass through the corresponding second aperture,
Each image sensor has a lens assembly corresponding to that image sensor, each lens assembly having its front end with an anti-vibration or thermal sleeve surrounding the lens assembly barrel. An image sensor system that is inserted into a second aperture corresponding to the above and attached to a vibration isolating member . A system for generating a map of a surface,
A global positioning receiver,
An altitude measuring unit that can be adjustably mounted on a moving object or platform;
A global positioning antenna that can be adjustably mounted on a moving object or platform;
A posture measuring unit that can be adjusted and attached to a moving object or a platform;
An imaging array that can be adjustably mounted on a moving object or platform and that looks at the surface;
A computer,
With
The imaging array
A mounting unit that can be adjustably mounted on a moving body or platform;
A first aperture formed in the mounting unit;
A first image sensor coupled to the mounting unit, having a first focal axis through the first aperture, and generating a first image area of the surface comprising a first data array of pixels;
Is coupled to the mounting unit is offset from the first image sensor A long track perspective expression along the array axis, a second focal axis that intersects the first focal axis passes through the first aperture A second image sensor for generating a second image area of the surface comprising a second data array of pixels;
Coupled to the mounting unit and offset along the array axis from the first image sensor on the opposite side of the second image sensor to produce a third image region of the surface comprising a third data array of pixels; A third image sensor which is a front oblique image sensor;
Coupled to the mounting unit and offset along the array axis from the second image sensor on the opposite side of the third image sensor to produce a fourth image region of the surface comprising a fourth data array of pixels; A fourth image sensor that is a rear oblique image sensor,
The computer communicates with the elevation measurement unit, the global positioning antenna, the attitude measurement unit, the first image sensor, and the second image sensor, and from one or more of the elevation measurement unit, the global positioning antenna, and the attitude measurement unit. based on the input, at least a portion of the image region of the first and second image sensors, associate a portion of the surface area,
The mounting unit includes one anti-vibration member having a second aperture corresponding to each of the first image sensor, the second image sensor, the third image sensor, and the fourth image sensor. The focal axes of each of the image sensor, the third image sensor, and the fourth image sensor pass through the corresponding second aperture,
Each image sensor has a lens assembly corresponding to that image sensor , each lens assembly having its front end with an anti-vibration or thermal sleeve surrounding the lens assembly barrel. The system is inserted into the second aperture corresponding to the above and attached to the vibration isolating member . At least part of the image elevations of the first image sensor and the second image sensor based on input from one or more of three-dimensional image calculation, LIDAR data, or digital elevation model from overlapping image data 21. The system of claim 20 , wherein is associated with a portion of surface elevation.