EP2423871B1 - Vorrichtung und Verfahren zur Erzeugung eines Übersichtsbildes einer Vielzahl von Bildern mittels einer Genauigkeitsinformation - Google Patents

Vorrichtung und Verfahren zur Erzeugung eines Übersichtsbildes einer Vielzahl von Bildern mittels einer Genauigkeitsinformation Download PDF

Info

Publication number
EP2423871B1
EP2423871B1 EP10174053.8A EP10174053A EP2423871B1 EP 2423871 B1 EP2423871 B1 EP 2423871B1 EP 10174053 A EP10174053 A EP 10174053A EP 2423871 B1 EP2423871 B1 EP 2423871B1
Authority
EP
European Patent Office
Prior art keywords
image
information
preprocessed
accuracy
images
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Not-in-force
Application number
EP10174053.8A
Other languages
English (en)
French (fr)
Other versions
EP2423871A1 (de
Inventor
Bernhard Rinner
Markus Quaritsch
Daniel Di Wischounig-Strucl
Saeed Yahyanejad
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Lakeside Labs GmbH
Original Assignee
Lakeside Labs GmbH
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Lakeside Labs GmbH filed Critical Lakeside Labs GmbH
Priority to EP10174053.8A priority Critical patent/EP2423871B1/de
Priority to US13/215,366 priority patent/US8797400B2/en
Publication of EP2423871A1 publication Critical patent/EP2423871A1/de
Application granted granted Critical
Publication of EP2423871B1 publication Critical patent/EP2423871B1/de
Not-in-force legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods

Definitions

  • Embodiments according to the invention relating to the field of image processing and particularly an apparatus and a method for generating an overview image of the plurality of images.
  • a challenge of low altitude imagery and mosaicking for surveillance purposes is finding an appropriate balance between seamless stitching and geo-referencing under consideration of processing time and other resources.
  • the scale difference as a result of different flying altitude resulted in several stitching errors.
  • significant stitching errors induced by scale differences among images may be visible.
  • Similar objects may have different sizes, and there may be a disparity in horizontal and vertical stitching.
  • a similar error may occur by inaccurate camera position or rotation.
  • stitching disparities may be caused by inaccurate camera angle or position.
  • a method for detecting nonflat abandoned objects by matching a reference and a target video sequence is described in "Hui Kong et al.: Detecting Abondoned Objects With a Moving Camera”.
  • the reference video is taken by a moving camera when there is no suspicious object in the scene.
  • the target video is taken by a camera following the same route and may contain extra objects. GPS information is used to roughly align the two videos and find the corresponding frame pairs.
  • an intrasequence geometric alignment based upon homographies is computed to find all possible suspicious areas, an intrasequence geometric alignment is done to remove false alarms caused by high objects, a local appearance comparison between two aligned intrasequence frames is performed to remove false alarms in flat areas, and a temporal filtering step is done to confirm the existence of suspicious objects.
  • An embodiment of the invention provides an apparatus for generating an overview image of a plurality of images.
  • Each image of the plurality of images comprises associated meta-data.
  • the apparatus comprises an image preprocessor, a storage unit and an image processor.
  • the image preprocessor is configured to preprocess a new image by assigning the new image to a position in the overview image based on a position information contained by meta-data of the new image.
  • the storage unit is configured to store a plurality of preprocessed or processed images. Each preprocessed or processed image of the plurality of preprocessed or processed image is assigned to a position in the overview image. Additionally, the storage unit is configured to provide the overview image containing the plurality of preprocessed or processed images at their assigned positions for displaying.
  • the image processor comprises an accuracy information input for receiving an accuracy information of the position information and a controllable processing engine, wherein the accuracy information of the position information are obtained by determining an accuracy of the determination of the position of an unmanned aerial vehicle using one or more environmental parameters determined by a sensor system of the unmanned aerial vehicle, wherein, if there is a strong wind, the accuracy of the determination of the position may be lower than under weak wind conditions.
  • the image processor is configured to determine an overlap region of the preprocessed new image and a stored preprocessed or stored processed image within the overview image based on the assigned position of the new image and the assigned position of the stored preprocessed or stored processed image.
  • controllable processing engine is configured to process the new image by re-adjusting the assigned position of the new image based on a comparison of features of the overlap region of the new image and the stored preprocessed or stored processed image.
  • controllable processing engine is controlled by the accuracy information of the position information received by the accuracy information input, so that a maximal re-adjustment of the assigned position of the new image is limited based on the received accuracy information of the position information.
  • Some embodiments according to the invention relate to an unmanned aerial vehicle according to claim 11.
  • the accuracy information can be taken into account, for example, for the generation of an overview image containing the taken image later on.
  • Fig. 1 shows a block diagram of an apparatus 100 for generating an overview image 124 of a plurality of images according to an embodiment of the invention.
  • Each image of the plurality of images comprises associated meta-data.
  • the apparatus 100 comprises an image preprocessor 110, a storage unit 120 and an image processor 130.
  • the image preprocessor 110 is connected to the image processor 130 and the image processor 130 is connected to the storage unit 120.
  • the image preprocessor 110 preprocesses a new image 102 by assigning the new image 102 to a position in the overview image 124 based on a position information contained by the meta-data of the new image 102.
  • the storage unit 120 stores a plurality of preprocessed or processed images 112, 122, wherein each preprocessed or processed image 112, 122 of the plurality of preprocessed or processed images comprises an assigned position in the overview image 124. Additionally, the storage unit 120 is able to provide the overview image 124 containing the plurality of preprocessed or processed images at their assigned positions for displaying.
  • the image processor 130 comprises an accuracy information input 134 for receiving an accuracy information 104 of the position information and a controllable processing engine 132.
  • the accuracy information input 134 is connected to the controllable processing engine 132 and the controllable processing engine 132 is connected to the image preprocessor 110 and the storage unit 120.
  • the image processor 130 determines an overlap region of the preprocessed new image 112 and a stored preprocessed or stored processed image 112, 122 within the overview image 124 based on the assigned position of the new image 112 and the assigned position of the stored preprocessed or stored processed image 112, 122.
  • the controllable processing engine 132 processes the preprocessed new image 112 by re-adjusting the assigned position of the preprocessed new image 112 based on a comparison of features of the overlap region of the preprocessed new image 112 and features of the overlap region of the stored preprocessed or stored processed image 112, 122.
  • controllable processing engine is controlled by the accuracy information of the position information 104 received by the accuracy information input 134, so that a maximal re-adjustment of the assigned position of the preprocessed new image 112 is limited based on the received accuracy information 104 of the position information.
  • the storage unit 120 adds the processed new image 122 with the re-adjusted assigned position to the plurality of preprocessed or processed images of the overview image 124.
  • the image-based processing of the preprocessed image 112 and the overlapping stored preprocessed or processed image 112, 122 preserves the connection of the distances of points in different images of the overview image 124 to the distances in reality at least with the accuracy of the position information contained by the meta-data.
  • a position-based alignment of images in an overview image can be refined by an image-based alignment without losing the whole information about distances between points in the overview image. So, the accuracy of preserving the distance in the overview image can be significantly increased. Further, the smoothness of transitions of images in the overview image may be increased in comparison to only position-based or orientation-based alignments.
  • the new image 102 may be provided, for example, from a storage unit (e.g. a memory card of a digital camera or a storage device of a computer) or may be transmitted online from an unmanned aerial vehicle or an airplane taking images of an area.
  • a storage unit e.g. a memory card of a digital camera or a storage device of a computer
  • Each image comprises associated meta data, which is stored together with the image or transmitted together with the image, for example.
  • the meta-data contains a position information of the image, but may also contain further data (e.g. roll, pitch, yaw of a camera, the image is taken with).
  • the position information may be, for example, a GPS position (global positioning system) or another position definition indicating a relative or an absolute position allowing to assign the new image to a position in the overview image.
  • the storage unit 120 stores a plurality of preprocessed or processed images of the overview image.
  • a preprocessed image 112 is an image comprising a position in the overview image assigned by the image preprocessor 110 before the assigned position is re-adjusted by the image processor 130. Consequently, a processed image 122 is an image after re-adjustment of the assigned position by the image processor 130.
  • a preprocessed image 112 may be stored directly after preprocessing by the storage unit 120 to enable the storage unit 120 to provide an overview image containing the new image very fast. Afterwards, the new image may be processed by the image processor 130 to refine the assigned position of the preprocessed new image 112. Alternatively, the preprocessed image 112 is directly processed by the image processor 130 without storing the preprocessed image 112.
  • the storage unit 120 is able to provide the overview image 124 to a display (which may be an optional part of the apparatus 100).
  • the image processor 130 determines an overlap region of the preprocessed new image 112 and a stored preprocessed or stored processed image 122. Although it may be possible that none of the already stored preprocessed or stored processed images 112, 122 overlap with the preprocessed new image 112, it is getting more likely when the number of images already contained by the overview image stored by the storage unit 120 increases. If a preprocessed new image 112 does not comprise an overlap region with any of the other images 112, 122 of the overview image 124 already stored by the storage unit 120, the preprocessed new image 112 may be stored directly by the storage unit 120.
  • the controllable processing engine 132 processes the preprocessed new image 112 by re-adjusting the assigned position, wherein the re-adjustment is limited based on the accuracy information 104 of the position information.
  • the accuracy information 104 of the position information may be the same for all images, may be updated periodically or depending on a change of the accuracy of the position information.
  • each image may comprise an individual accuracy information 104 of the individual position information contained by the associated meta-data. For example, if the images are taken by an unmanned aerial vehicle, the unmanned aerial vehicle may determine a position and an accuracy of the position at the time a picture is taken and transmits this information together with the image to the apparatus 100.
  • the unmanned aerial vehicle may transmit accuracy information periodically or when the accuracy changes more than a predefined threshold.
  • the accuracy of the position information is known and stored by a storage device, which provides the accuracy information of the position information to the accuracy information input 134.
  • the re-adjustment is done based on a comparison of features of the overlap region of the preprocessed new image 112 and the stored preprocessed or stored processed image 112, 122.
  • the features of the overlap region may be common feature points identified in the overlap regions of both images or areas around feature points of the overlap regions of images.
  • a re-adjustment (or a further re-adjustment) of the assigned position of the stored preprocessed or stored processed image 112, 122 may be done.
  • the re-adjustment of the assigned position of the stored preprocessed or stored processed image 112, 122 may also be limited by an accuracy information of the position information of the stored preprocessed or stored processed image 112, 122.
  • the image preprocessor 110, the storage unit 120 and the image processor 130 may be, for example, independent hardware units or part of a computer or micro controller as well as a computer program or a software product configured to run on a computer or micro controller.
  • the image preprocessor 110 may preprocess the new image 102 by considering a roll information, a pitch information and/or a yaw information to correct an orientation and/or a perspective distortion of the new image 102.
  • the new image 102 may be taken from an unmanned aerial vehicle.
  • Such an unmanned aerial vehicle is usually small and therefore susceptible for wind and other environmental influences.
  • sensors (IMU, inertial measurement unit) of the unmanned aerial vehicle may determine a roll, a pitch and a yaw of the unmanned aerial vehicle (or of the camera) at the time an image is taken and add a roll information, a pitch information and/or a yaw information to the meta-data of the image.
  • the image preprocessor 110 can use this information contained by the meta-data to compensate an orientation offset or a perspective distortion of the new image 102.
  • controllable processing engine 132 may process the preprocessed new image 112 by re-adjusting an orientation or a perspective distortion based on the comparison of the features of the overlap region.
  • the controllable process engine may be further controlled by accuracy information 104 of the roll information, the pitch information or the yaw information received by the accuracy information input 134, so that the maximal re-adjustment of the assigned position of the preprocessed new image 112 and a maximal re-adjustment of the orientation or the perspective distortion is limited based on the accuracy information 104 of the roll information, the pitch information or the yaw information.
  • the maximal re-adjustment of the assigned position may be limited by the accuracy of the determination of the position, which may also depend on the accuracy of the roll, the pitch and/or the yaw.
  • the position and orientation of the camera the image was taken with, an unmanned aerial vehicle or another vehicle taking that new image may be considered during re-adjustment of the assigned position of the new image.
  • an orientation or a perspective distortion may be re-adjusted and limited based on the accuracy information 104.
  • Fig. 3 shows an example for position and orientation errors, which can be taken into account for the re-adjustment of the position of the new image 112. It indicates the GPS error range (the real position is in this range) and the tilting error range. The sum of these two errors may give the total positioning error, which may be represented by the accuracy information of the position information. It shows an example for a maximum position error (accuracy of the position information), if the new image 102 is taken by an unmanned aerial vehicle.
  • the image preprocessor 110 may receive successively a plurality of new images 102 and preprocesses each received new image 102. Further, the storage unit 120 may store each preprocessed new image 112 and provides an updated overview image 124 after storing a predefined number of preprocessed new images 112.
  • the predefined number of images may be one, so that the storage unit 120 may provide an updated overview image after storing each preprocessed new image 112.
  • the predefined number of preprocessed images may be higher than one, so that the overview image is updated periodically after storing the predefined number of preprocessed images.
  • an overview image 124 can be provided very fast, since the position and/or orientation-based preprocessing can be done significantly faster than the image-based processing by the image processor 130.
  • the apparatus 100 may receive continuously new images from an unmanned aerial vehicle, which can be provided for displaying in an overview image directly after preprocessing. A re-adjustment of the assigned position by the image processor 130 for refinement of the transition between overlapping images may be done later on.
  • Fig. 2 shows a block diagram of an apparatus 200 for generating an overview image 124 of a plurality of images according to an embodiment of the invention.
  • the apparatus 200 is similar to the apparatus shown in Fig. 1 , but comprises additionally an optional unmanned aerial vehicle control unit 240 connected to the storage unit 120 and the image preprocessor 110 is connected to the storage unit 120.
  • the storage unit 120 may add the preprocessed new image 112 with the assigned position to the plurality of preprocessed or processed images. Further, the storage unit 120 may provide the overview image 124 containing the preprocessed new image 112 at the assigned position for displaying before the preprocessed new image 112 is processed by the image processor 130 as already mentioned above. In this way, an overview image 124 already containing the preprocessed new image 112 can be provided very fast.
  • the accuracy of the distance between two points in different images and/or the smoothness of the transitions between the preprocessed new image 112 and other overlapping images of the plurality of preprocessed or processed images 112, 122 may be increased by re-adjusting the assigned position of the preprocessed new image 112 by the image processor 130.
  • the storage unit 120 may provide the overview image 124 and later the storage unit 120 may provide a refined overview image 124 after processing the preprocessed image 112 by the image processor 130.
  • the storage unit 120 may add the processed new image 122 by replacing the stored preprocessed new image 112 by the processed new image 122.
  • the apparatus 200 may comprise an unmanned aerial vehicle control unit 240.
  • the unmanned aerial control unit 240 may generate a control signal 242 for piloting an unmanned aerial vehicle to a region corresponding to an area of the overview image 124 being not already covered by an image of the plurality of preprocessed or processed images 112, 122.
  • the unmanned aerial vehicle control unit 240 may pilot the unmanned aerial vehicle to uncovered areas of the overview image to obtain images from the uncovered areas.
  • Fig. 4 shows an example of an apparatus 200 piloting an unmanned aerial vehicle 460 with a camera 462 for taking images of the area below the unmanned aerial vehicle 460.
  • the apparatus 200 may transmit a control signal to the unmanned aerial vehicle 460 for piloting the unmanned aerial vehicle 460 to uncovered regions of the overview image 452.
  • the overview image 452 may be provided to a display 450.
  • the display 450 shows covered areas 454 and uncovered areas of the overview image 452.
  • the unmanned aerial vehicle 460 may take images from the uncovered areas and provide the image data as well as corresponding meta-data (e.g. position information, roll information, pitch information and/or yaw information) to the apparatus 200 for generating the overview image 452 of the plurality of images 454.
  • meta-data e.g. position information, roll information, pitch information and/or yaw information
  • Fig. 5 shows a flowchart of a method 500 for generating an overview image of a plurality of images according to an embodiment of the invention.
  • Each image of the plurality of images comprises associated meta-data.
  • the method 500 comprises preprocessing 510 a new image, storing 520 a plurality of preprocessed or processed images of the overview image, determining 530 an overlap region of the preprocessed new image and a stored preprocessed or stored processed image within the overview image, processing 540 the preprocessed new image, adding 550 the processed new image to the plurality of preprocessed or processed images and providing 560 the overview image containing the plurality of preprocessed or processed images.
  • the new image is preprocessed 510 by assigning the new image to a position in the overview image based on a position information being contained by the meta-data of the new image. Further, each stored preprocessed or stored processed image of the plurality of preprocessed or processed images comprising an assigned position in the overview image. The overlap region of the preprocessed new image and the stored preprocessed or stored processed image within the overview image is determined 530 based on the assigned position of the preprocessed new image and the assigned position of the stored preprocessed or stored processed image.
  • the processing 540 of the preprocessed new image is done by re-adjusting the assigned position of the preprocessed new image based on a comparison of features of the overlap region of the preprocessed new image and the stored preprocessed or stored processed image.
  • the processed new image is added 550 to the plurality of preprocessed or processed images with the re-adjusted assigned position.
  • the overview image is provided 560 containing the plurality of preprocessed or processed images at their assigned positions for displaying.
  • the method 500 may comprise adding the preprocessed new image to the plurality of preprocessed or processed images of the overview image, displaying the overview image containing a preprocessed new image at the assigned position and displaying the overview image containing the processed new image at the re-adjusted assigned position. In this way, a rough overview image may be displayed very fast and a refined overview image may be displayed later on.
  • the method 500 may comprise further steps representing the optional features of the described concept mentioned above.
  • Fig. 6 shows a block diagram of a apparatus 600 for generating an overview image 614 of a plurality of images.
  • the apparatus 600 comprises a storage unit 610 and an image processor 620.
  • the storage unit 610 is connected to the image processor 620.
  • the storage unit 610 stores a plurality of processed images of the overview image 614.
  • Each processed image 612 of the plurality of processed images comprises an assigned position in the overview image 614.
  • the storage unit 610 is able to provide the overview image 614 containing the plurality of processed images at their assigned positions for displaying.
  • the image processor 620 determines feature points of a new image 602 and compares the determined feature points of the new image 602 with feature points of a stored processed image 612 to identify common feature points and to obtain 3-dimensional positions of the common feature points. Further, the image processor 620 determines common feature points located within a predefined maximum distance of relevance to a reference plane based on the 3-dimensional positions of the common feature points to identify relevant common feature points. Additionally, the image processor 620 processes the new image 602 by assigning the new image to a position in the overview image 614 based on a comparison of an image information of each relevant common feature point of the new image 602 with an image information of each corresponding relevant common feature point of the stored processed image 612 without considering common feature points located beyond the predefined maximum distance to the reference plane. Further, the storage unit 610 adds the processed new image 612 with the assigned position to the plurality of processed images of the overview image 614.
  • the storage unit 610 may store each processed image together with the assigned position in the overview image 614.
  • the overview image 614 may be provided to display (which may be an optional part of the apparatus 600).
  • the feature points determined by the image processor may be, for example, corners or walls of houses, the contours of a car, road markings or similar features. These feature points may be determined by various algorithms as, for example, SIFT (scale-invariant feature transform) or SURF (speeded up robust features).
  • SIFT scale-invariant feature transform
  • SURF speeded up robust features
  • these feature points may be compared to feature points of a stored processed image (e.g. an overlapping image) to identify common feature points by using, for example, a correlation function (e.g. nearest neighbor). Further, for these common feature points 3-dimensional positions of the common feature points may be determined, for example, by using methods of multi-view geometry.
  • a correlation function e.g. nearest neighbor
  • the reference plane may be predefined or may be determined by the image processor 620 based on the 3-dimensional positions of the common feature points.
  • a predefined reference plane may be a horizontal plane of a 3-dimensional coordinate system (e.g. Cartesian coordinate system) with the z-coordinate equal to 0 or another predefined value.
  • the image processor 620 may determine the reference plane by fitting a horizontal plane to the 3-dimensional positions of the common feature points, so that a maximal number of common feature points are located within a predefined maximal fitting distance to the reference plane or so that the reference plane is as far away as possible from a camera position, the new image was taken from, with at least one common feature point located within the predefined maximal fitting distance.
  • the maximal fitting distance may vary between zero and, for example, a value depending on a maximal height difference of the common feature points.
  • the reference plane may be a non-horizontal plane.
  • the accuracy of preserving the distances between points in different images and/or the smoothness of transitions between images may be influenced.
  • a large maximum distance of relevance more relevant common feature points are identified, which are taken into account for assigning the position of the new image. This may increase the statistic, since more common feature points are considered, but may also increase the error obtained by considering points at different elevation levels.
  • An opposite effect is obtained by choosing a low predefined maximum distance of relevance.
  • the predefined maximum distance of relevance may also be zero, so that only common feature points are identified as relevant which are located in the reference plane.
  • the maximal fitting distance may be equal to the maximal distance of relevance.
  • common feature points may be road markings and the reference plane may be the road.
  • the reference plane may be the road.
  • only common feature points located at the height of the street may be considered for assigning the position.
  • an error obtained by considering feature points at different elevation levels e.g. feature points located at cars, houses or street lights
  • the image information of a relevant common feature point may be, for example, the position of the feature point in the image itself or the image data of an area of the image around the relevant common feature point. More accurately, the image information of a relevant common feature point in the new image 602 may be the position of the relevant common feature point in the new image 602 and the image information of a relevant common feature point in the stored image 612 may be a position of the relevant common feature point in the stored image.
  • the image processor 620 may assign the new image 602 to a position in the overview image 614, so that a distance between a position of a relevant common feature point in the new image 602 and a position of a corresponding relevant common feature point in the stored image 612 is minimized or so that a sum of distances between the position of the relevant common feature points in the new image 602 and the positions of the respective corresponding relevant common feature points in the stored image 612 is minimized.
  • the image information of a relevant common feature point in the new image 602 may be an area of the new image 602 around the relevant common feature point of a predefined size and the image information of a relevant common feature point in the stored image 612 may be an area of the stored image 612 around a relevant common feature point of the predefined size.
  • the image processor 620 may assign the new image 602 to a position in the overview image 614, so that a correlation value of an area around a relevant common feature point of the new image 602 and an area around a corresponding relevant common feature point of the stored image 612 is maximized or so that the sum of correlation values of areas around relevant common feature points of the new image 602 and areas around respective corresponding relevant common feature points of the stored image 612 is maximized.
  • a correlation value may be determined, for example, based on a given correlation function for correlating images or parts of images.
  • the image processor 620 may assign the new image 602 to a position in the overview image 614 based on the minimization of the distance or the sum of distances and re-adjusts the assignment of the new image 602 to a position in the overview image 614 based on the maximization of a correlation value or a sum of correlation values.
  • the image processor 620 may determine feature points of the new image 602 for the whole image or only parts of the new images 602 overlapping another image of the plurality of processed images. Determining feature points from the whole image may increase the efforts, but the determined feature points can be stored together with the new image after processing, so that the feature points may only be determined once.
  • the image processor 620 may determine an overlap region of the new image 602 and the stored image 612. Further, the image processor 620 may compare determined feature points of the new image 602 located in the overlap region with feature points of the stored image 612 located in the overlap region, while determined feature points outside the overlap region are neglected for the identification of common feature points.
  • the storage unit 610 and the image processor 620 may be, for example, independent hardware units or part of a computer or micro controller as well as a computer program or a software product configured to run on a computer or a micro controller.
  • Fig. 7 shows a flowchart of a method 700 for generating an overview image of a plurality of images.
  • the method 700 comprises storing 710 a plurality of processed images, determining 720 feature points of a new image, comparing 730 the determined feature points of the new image with feature points of a stored processed image, determining 740 common feature points, processing 750 the new image, adding 760 the processed new image to the plurality of processed images of the overview image and providing 770 the overview image.
  • Each stored processed image of the plurality of processed images comprises an assigned position in the overview image.
  • the determined feature points of the new image are compared 730 with feature points of a stored processed image to identify common feature points and to obtain 3-dimensional positions of the common feature points.
  • common feature points located within a predefined maximum distance of relevance to a reference plane are determined 740 based on the 3-dimensional positions of the common feature points to identify relevant common feature points.
  • the new image is processed 750 by assigning the new image to a position in the overview image based on a comparison of an image information of each relevant common feature point of the new image with an image information of each corresponding relevant common feature point of the stored processed image without considering common feature points located beyond the predefined maximum distance of relevance to the reference plane.
  • the processed new image is added 760 to the plurality of processed images of the overview image with the assigned position. Additionally, the overview image containing the plurality of processed images at their assigned positions is provided 770 for displaying.
  • the method 700 may comprise further steps representing features of the described concept mentioned above.
  • Some embodiments according to the invention relate to an apparatus for generating an overview image of a plurality of images combining the features of the apparatus shown in Fig. 1 and the features of the apparatus shown in Fig. 6 .
  • the image processor 130 of apparatus 100 may re-adjust the assigned position of the preprocessed new image based on a comparison of an image information of relevant common feature points of the new image with an image information of each corresponding relevant common feature point of a stored processed image without considering common feature points located beyond a predefined maximum distance of relevance to a reference plane. For this, the image processor 130 may determine feature points of the preprocessed new image and may compare the determined feature points of the preprocessed new image with feature points of a stored processed image to identify common feature points and to obtain 3-dimensional positions of the common feature points. Further, the image processor 130 may determine common feature points located within a predefined maximum distance of relevance to a reference plane based on the 3-dimensional positions of the common feature points to identify relevant common feature points.
  • the apparatus 100 may realize further features, for example, mentioned in connection with the apparatus 600 shown in Fig. 6 .
  • the apparatus 600 may comprise additionally an image preprocessor 110 and the image processor 620 may comprise an accuracy information input 134 and a controllable processing engine 132 as mentioned in connection with the apparatus 100 shown in Fig. 1 .
  • each image of the plurality of images may comprise associated meta-data.
  • the image preprocessor 110 may preprocess a new image by assigning the new image to a position in the overview image based on a position information contained by the meta-data of the new image.
  • the storage unit 610 may store a plurality of preprocessed or processed images of the overview image, wherein each preprocessed or processed image of the plurality of preprocessed or processed images comprises an assigned position in the overview image.
  • the storage unit 610 may provide an overview image containing the plurality of preprocessed or processed images at their assigned positions for displaying. Further, the image processor 620 may determine an overlap region of the preprocessed new image and a stored preprocessed or stored processed image within the overview image based on the assigned position of the preprocessed new image and the assigned position of the stored preprocessed or stored processed image.
  • the controllable processing engine 132 may process the preprocessed image by re-adjusting the assigned position of the preprocessed new image according to the originally defined image processor 610, but with the restriction that the controllable processing engine 132, and in that way the image processor 610, is controlled by the accuracy information of the position information received by the accuracy information input, so that a maximal re-adjustment of the assigned position of the preprocessed new image is limited based on the received accuracy information of the position information.
  • the apparatus 600 may realize further, for example, features mentioned in connection with apparatus 100 shown in Fig. 1 or apparatus 200 shown in Fig. 2 .
  • the images are taken by an unmanned aerial vehicle, although the described inventive concept may also be used or implemented, if images are provided, for example, by a storage unit (e.g. memory card or multimedia card of a digital camera or a hard disk of a computer).
  • a storage unit e.g. memory card or multimedia card of a digital camera or a hard disk of a computer.
  • a very simple and naive approach is to align the images based on the camera's position.
  • the world coordinates of the camera are mapped to corresponding pixel coordinates in the generated overview image.
  • o world (lat; Ion; alt) T in world coordinates
  • all image coordinates are related to this origin on the local tangential plane (LTP) by approximation to the earth model WGS84.
  • LTP local tangential plane
  • FIG. 12 An example of the resulting overview image is depicted in Figure 12 utilizing the placement function with transformation being just a simple translation for each image.
  • This approach reasonably accurate position information is assumed and a nadir view but do not take into account the camera's orientation. Obviously, effects introduced by non-planar surfaces can not be compensated with this approach.
  • a more advanced approach is to extend the naive position-based alignment by compensating the camera's orientation deviation (i.e., roll, pitch, yaw angles).
  • the placement function of the individual images to generate the overview image is the same as before. But instead of considering only translation, a perspective transformation with eight degrees of freedom may be used.
  • image-based alignment can be categorized into pixel-based, and feature-based methods. The idea is to find transformations T i and consequently the position of each new image which maximizes the quality function: ⁇ ⁇ Merge ⁇ I res , i - 1 , T i I i
  • the pixel-based approaches are computationally more expensive because the quality function is computed from all pixels in the overlapping parts of two images.
  • Feature-based approaches try to reduce the computational effort by first extracting distinctive feature points and then match the feature points in overlapping parts. Depending on the chosen degree of freedom the resulting transformation ranges from a similarity transformation to a perspective transformation. The benefit of this approach is that the generated overview image is visually more appealing. But on the other hand, the major disadvantages are that the search space grows with the number of images to be stitched and the images may get distorted. An example is shown in Fig. 14 .
  • multiple small-scale UAVs may be deployed to support first responders in disaster assessment and disaster management.
  • quadrocopters may be used since they are agile, easy to fly and very stable in the air due to sophisticated on-board control.
  • the UAV may be equipped with an RGB camera (red, green blue).
  • the intended use-case can be sketched as follows: The operator first specifies the areas to be observed on a digital map and defines the quality parameters for each area (see for example " M. Quaritsch, E. Stojanovski, C. Bettstetter, G. Friedrich, H. Hellwagner, M. Hofbaur, M. Shah, and B. Rinner. Collaborative Microdrones: Applications and Research Challenges. In Proceedings of the Second International Conference on Autonomic Computing and Communication Systems (Autonomics 2008), page 7, Turin, Italy, September 2008 "). Quality parameters include, for example, the spatial and temporal resolution of the generated overview image, and the minimum and maximum flight altitude, among others.
  • the system Based on the user's input, the system generates plans for the individual drones to cover the observation areas (see for example " M. Quaritsch, K. Kruggl, D. Wischounig-Strucl, S. Bhattacharya, M. Shah, and B. Rinner. Networked UAVs as Aerial Sensor Network for Disaster Management Applications. e&i Journal, 127(3):56-63, March 2010 "). Therefore, the observation areas are partitioned into smaller areas covered by a single picture taken from a UAV flying at a certain height. The partitioning has to consider a certain overlap of neighboring images which is required by the stitching process. Given a partitioning the continuous areas to be covered can be discretized to a set of so-called picture-points. The picture-points are placed in the center of each partition at the chosen height. The pictures are taken with the camera pointing downwards (nadir view).
  • the mission planner component e.g. unmanned aerial vehicle control unit
  • the mission planner component generates routes for individual UAVs such that each picture-point is visited taking into account the UAV's resource limitations.
  • the images together with metadata i.e., the position and orientation of the camera
  • the major goal is to generate an overall image I res;n of the target area given a set of n individual images ⁇ I i ⁇ .
  • This mosaicking can be described as an optimization problem, in which T i has to be found in a way that it maximizes a quality function X(I res , i ).
  • This quality function based on the system use case, balances the visual appearance (improving the smoothness of transitions) and the geo-referencing accuracy (accuracy of preserving distances). While in some applications it is more important to have a visually appealing overview image, other applications may require accurate geo-referencing in the overview image.
  • a quality function may be used that is a combination of the correlation between overlapping images and relative distances in the generated overview image compared to the ground truth.
  • the UAV's auxiliary sensors such as GPS, IMU and altimeter are used to determine its position and orientation.
  • auxiliary sensors in small-scale UAVs provide only limited accuracy which is not comparable with larger aircrafts. As consequence, it can not be relied on accurate and reliable position, orientation and altitude data of the UAV. Hence it has to be dealt with this inaccuracy in the mosaicking process.
  • the resources such as computation power and memory on-board the UAVs but also on the ground station may be very limited. In disaster situations it is usually not possible to have a huge computing infrastructure available.
  • the base-station may consist of notebooks and standard desktop PCs. But at the same time, the overview image should be presented as quick as possible
  • the individual images may be taken from multiple UAVs in an arbitrary order.
  • An incremental approach may be used to present the user the available image data as early as possible while the UAVs are still on their mission. The more images are taken the better the overview image gets. This also means that a new image may require to adjust the position of already processed image to improve the overall quality.
  • an appropriate transformation T i for each image I i captured at a picture-point may be found in order to solve the mosaicking problem.
  • the metadata approach exploits auxiliary sensor information to derive the position and orientation of the camera which is then used to compute the transformations.
  • auxiliary sensor data i.e., GPS, IMU, altitude and time
  • the image-based approach only exploits image data to compute the transformations.
  • the proposed concept may realize a combination of metadata-based and image-based methods enhancing the metadata-based alignment with image-based alignment.
  • the presented approaches vary in their resource requirements and their achieved results.
  • An aspect is to first place the new images based on the camera's position (position information) and orientation information (e.g. roll, nick, yaw) on the already generated overview image.
  • image-based methods are used to correct for inaccurate position and orientation information and at the same time improve the visual appearance. Since the approximate position of the image is already known from the camera's position, the search-space can be significantly reduced (by limiting the maximal re-adjustment).
  • the transformation T i mentioned before may be split into two transformation whereas the T i,pos represents the transformation based on the camera's position and orientation and T i,img represents the transformation which optimizes the alignment using the image-based method.
  • Transformations T i , img and T i,pos which maximize the quality function may be preferred: ⁇ ⁇ Merge ⁇ I res , i - 1 , T i , img ⁇ T i , pos I i ,
  • the search space may be limited to a reduced set of possible positions based on the expected inaccuracy of position and orientation information (accuracy information of the position information and optionally the accuracy information of the roll, nick or yaw).
  • FIG. 8 A possible process 800 of aligning images taken from mico-UAVs flying at low altitude is sketched in Figure 8 .
  • the input for the whole processing pipeline is a new image taken from the camera on-board the UAV and a set of meta-data.
  • the meta-data contains information on the (e.g. GPS-) position where the photo has been taken as well as information on the UAV's and thus the camera's pose (yaw, nick, roll) - also known as extrinsic camera parameters. Due to the limited capabilities of the UAVs-and the small and relatively cheap inertial measurement units (IMU)-the meta-data is inaccurate to some degree. For example the accuracy of GPS-positions is in the range of up to 10 meters and also the measured angles have errors.
  • IMU inertial measurement units
  • the image preprocessing 810 may include several steps to improve the original images taken by a camera. For example, lens distortions are corrected.
  • an image transformation based on meta-data is performed 820 (assigning the new image to a position in the overview image). Given the image's meta-data, quickly the transformation T i,pos can be computed which is solely based on the meta-data.
  • This transformation basically includes a translation and rotation (to take into account the UAV's heading(yaw)) and optionally warps the image as it has been taken at a nadir view (i.e., correct nick and roll angles).
  • intersecting images may be found 830 (determining an overlap region). Based on the image position in the pool of already processed images those that intersect with the current one are identified. If no image can be found, further processing is postponed until at least one intersecting image is available. In this case the current image is placed in the overview image only based on the meta-data (T i,pos ).
  • a (rough) 3D structure is computed 840.
  • the original image (new image) and the set of images that intersect (overlap) the original one are considered.
  • First feature-points from the new image are extracted 842 (determine feature points). Different algorithms can be used to compute feature-points (e.g., SIFT, SURF, ). For the already processed images the feature-points can be stored separately and thus avoid repeated computation. Consequently, a set of feature-points for each image to consider is obtained.
  • the feature-points from the individual images are matched 844 (determine common feature points) using some correlation function (e.g., nearest neighbor).
  • mapping which feature-point in one image is (expected to be) the same point in the other images is established.
  • methods of multi-view geometry can be exploited to estimate both, the camera's positions and the 3D position of the feature-points 846.
  • the transformation is refined 850 based on the 3D structure and image data.
  • the input for this step is the set of intersecting images and the rough 3D structure of the scene computed in the previous step.
  • rough 3D structure a point-cloud in 3D space with their corresponding image coordinates is understood.
  • One major goal is to maintain geo-referencing as good as possible and at the same time compute a visually appealing overview image.
  • the transformation is computed based on point-matches on a common plane (reference plane, e.g. preferably the ground plane) and ignore regions which are at different elevation levels (e.g., cars, trees, etc.). For this, given the 3D point-cloud a common plane may be extracted.
  • horizontal planes may be fit 852 into the point-cloud and select a common plane (e.g., the plane far away from the camera, or the plane with the highest number of points lying on it, etc.). For the next steps only those points that lie on the selected common plane (relevant common feature points) may be considered. Then, the transformation T i,img that optimizes the placement of the image further may be computed. This can be done either by only considering the selected point-correspondences, or by considering a certain region around the point-correspondences, or both.
  • a transformation that minimizes the distance between corresponding points can be computed.
  • a region around the corresponding points can be defined and the correlation within corresponding regions can be maximized.
  • Various methods can be used, e.g. simple pixel differencing or optical flow methods.
  • the transformation based on corresponding points may be used as a starting point which is then further refined by correlation-based methods.
  • the sensor model restricts the transformation T i,img .
  • This step can be considered optional and also depends on the method used to refine T i , img . For example if the GPS-position is known to be up to 2m inaccurate point correspondences (relevant common feature points) that suggest a position error of 7m can be ignored. Or for the correlation based approach the search range can be limited 854 to the according range.
  • the output of the algorithm described above are two transformations T i,pos and T i,img .
  • the first one can be computed very quickly which allows fast placement of the individual images on an overview map (overview image).
  • the latter one includes somewhat more computational effort and thus requires more time.
  • images can be displayed immediately based on the initial transformation T i,pos and later on refined.
  • the final transformation T i is the combination 860 (re-adjustment of the position of the new image) of T i,pos and T i,img .
  • ⁇ I res ⁇ ⁇ ⁇ spat I res + 1 - ⁇ ⁇ ⁇ corr I res
  • CC X ⁇ Y Covariance X ⁇ Y ⁇ X ⁇ ⁇ Y
  • d i is the actual distance measured between two ground control points
  • d ⁇ l is the estimated distance extracted from overview image
  • m is the number of considered distances.
  • ⁇ spat and ⁇ corr are all in the range of (0, 1].
  • the total quality function ⁇ is a weighted combination of ⁇ spat and ⁇ corr (0 ⁇ a ⁇ 1).
  • ⁇ spat represents the accuracy of spatial distances while ⁇ corr shows the correlation in areas of overlapping images, which is a measure for the seamlessness mosaicking.
  • the weight a 0:5.
  • a triangle is chosen, spanning significant points (P 3 ; P 6 ; P 11 ) for simplified spatial evaluation in the reduced set of eight images (e.g. shown in Fig. 10a ).
  • the table above shows a spatial accuracy and quality parameters of three known and the proposed mosaicking approaches.
  • the resulting spatial quality and the correlation quality ⁇ (I Res ) are presented and combined according to Equation 1 to a final quality characteristic to compare the presented approaches.
  • Metadata-based approaches like the position-based approach and the position-based approach with rotation retain geo-referencing, if only similarity transformations are used.
  • the total error range used in Figure 10a is GPS error + tan( ⁇ ) x height ⁇ 7m in real world distance at the ground level, which is approximately equivalent to 1/4 of the image width. Yet, in a complete nadir view, orthogonality will be reduced when getting away from the optical axis. Somehow it gives an idea that the middle parts of an images contain more reliable data. So for more pleasant result in the mosaicking (generation of the overview image), it may be made sure that the central part of each image under each picture-point is not masked by the border parts of other images.
  • Figure 11a shows the relation between correlation of the overlapping parts of two adjacent images in different approaches.
  • FIG. 11b indicates the relative distance from the estimated position to the corresponding GPS position on each image in the hybrid approach. By comparing these two graphs, it can be seen that if the estimated position of an image is close to its indicated GPS position it results in a higher correlation and vice versa.
  • Figure 10b shows a snapshot from a possible user interface. The operator defines the target area, then single images are placed iteratively over background map. The line shows the flight path of the UAV. Any kind of existing geo-referenced digital maps (e.g., from Google earth or Microsoft virtual earth) or an empty background can be used.
  • Fig. 10a shows an illustration of a mosaicking result of images taken from a roundtrip mission using the described concept
  • Fig. 10b shows an illustration of a screen shot of a possible graphical user interface (GUI) of an unmanned aerial vehicle system. Captured images are incrementally stitched over the (partially outdated) background image.
  • GUI graphical user interface
  • Some embodiments according to the invention relate to a method for generating an overview image from a set of individual images or an incremental mosaicking of images from autonomous, small-scale unmanned aerial vehicles.
  • Unmanned aerial vehicles have been recently deployed in various civilian applications such as environmental monitoring, aerial imaging or surveillance.
  • Small-scale UAVs are of special interest for first responders since they can rather easily provide bird's eye view images of disaster areas.
  • a concept to mosaic an overview image of the area of interest given a set of individual images captured by UAVs flying at low altitude is proposed among others.
  • the approach combines metadata-based and image-based stitching methods in order to overcome the challenges of low-altitude, small-scale UAV deployment such as non-nadir view, inaccurate sensor data, non-planar ground surfaces and limited computing and communication resources.
  • geo-referencing is preserved as much as possible, since this is an important requirement, for example, for disaster management applications.
  • the mosaicking method may be implemented on a UAV system and evaluated based on a quality metric.
  • Unmanned aerial vehicles are, for example, widely used in the military domain. Advances in technology, material science and control engineering made the development of small-scale UAVs possible and affordable. Such small-scale UAVs with a total weight of approximately 1 kg and a diameter of less than 1 m are getting prominent in civilian applications too and pose new research questions.
  • These UAVs are equipped with sensors such as accelerometers, gyroscopes, and barometers to stabilize the flight attitude and GPS receivers to obtain accurate position information. Additionally, UAVs can carry payloads such as cameras, infrared cameras, or other sensors.
  • UAVs enable to obtain a bird's eye view of an area which is helpful in applications such as environmental monitoring, surveillance and law enforcement, and disaster assessment and disaster management (see for example " M. Quaritsch, E. Stojanovski, C. Bettstetter, G. Friedrich, H. Hellwagner, M. Hofbaur, M. Shah, and B. Rinner. Collaborative Microdrones: Applications and Research Challenges. In Proceedings of the Second International Conference on Autonomic Computing and Communication Systems (Autonomics 2008), page 7, Turin, Italy, September 2008 ").
  • each application domain has different requirements.
  • One goal is to support first responders in disaster assessment and disaster management since this is the most challenging application domain.
  • first responders can not rely on a fixed infrastructure and the available information (e.g., maps) may no longer be valid. It is important to provide the first responders a quick and accurate overview of the affected area, typically spanning hundreds of thousands of square meters.
  • This overview image may be refined (according to one aspect of the invention) and updated over time and can be augmented with additional information such as detected objects or the trajectory of moving objects.
  • the overview image needs to be generated from dozens of individual images.
  • a number of UAVs equipped with cameras may be employed instead of a single UAV to cope with the stringent time constraints and the limited flight time.
  • the UAVs flying at low altitudes of up to 100 m, provide images of the affected area which are stitched to an accurate overview image.
  • the hybrid approach for image mosaicking may take both the position information and the image data into account.
  • the described mosaicking approach may be evaluated using a quality metric which is based on a spatial metric and a correlation metric.
  • a method for generating an overview image (“mosaic”) from a set of individual aerial images which have been captured from low altitudes. Position and orientation data from the aerial cameras may be exploited to generate a fast alignment of the individual images on the ground surface. Further, this alignment may be refined by applying different alignment procedures such as maximizing the pixel correlations within the image overlaps or exploiting estimated depth information to identify corresponding points at the same elevation level. On the one hand this method is able to provide a quick overview image which can be refined later on. On the other hand, it achieves a high geo-reference quality and reduces the processing time of the refinement process by limiting the search space through the preceding alignment steps.
  • the proposed concept may enable, for example, an online mosaicking method that increases the resulting geo-referencing accuracy incrementally.
  • Online execution means that the mosaic is created while flying and continuously taking images.
  • a first snapshot may be presented immediately - potentially with high geo-reference deviation and even from images without any overlap.
  • GPS and IMU computing the camera position and view
  • the focus may lie on geo-referencing and projection accuracy instead of merging or fusing images to a nice looking panorama.
  • An appealing fusion is optional later. But any waste or reduction of data by fusion may be considered as information about the scene (e.g. moving objects).
  • small-scale quadrocopters with worse stability and device accuracy than planes (even small planes) but with big advantages in respect of the intended use case may be used (e.g. disaster recovery). They are easy to transport and to operate. Additionally, the approach may have a short feedback loop, determining uncovered or badly covered areas already during flight, due the mosaicking is done online. The UAVs can react immediately on changes.
  • a system for mosaicking high-resolution overview images of large areas with high geometric accuracy from a set of images taken from small-scale UAVs may be realized. This may take the use of small-scale UAVs flying at low altitude into account.
  • a hybrid approach is described that combines inaccurate information on the camera's position and orientation, and the image data. Thus, it can maintain geometric accuracy and at the same time enhance the visual appearance.
  • the evaluations show that the approach results in a higher correlation between overlapping image regions and retains spatial distances with an error of less than 30 cm. Further, the computation time for a set of 37 images may be reduced by approximately 70 % compared to an image-based mosaicking.
  • More dynamic and interactive methods of mosaicking images may be included to increase the quality of the overview image, i.e., as new images are taken the transformation of already mosaicked images are refined.
  • the proposed method may be applied also for larger areas and use images from multiple UAVs.
  • Fig. 9 shows a block diagram of an unmanned aerial vehicle 900 according to an embodiment of the invention.
  • the unmanned aerial vehicle 900 comprises a camera 910, a sensor system 920, an accuracy determiner 930 and a transmitter 940.
  • the camera 910, the sensor system 920 and the accuracy determiner 930 are connected to the transmitter 940.
  • the camera 910 is able to take an image during a flight of the unmanned aerial vehicle 900.
  • the sensor system 920 determines a position of the unmanned aerial vehicle 900 at the time, when the image is taken, to obtain a position information associated to the taken image.
  • the accuracy determiner 930 determines an accuracy of the determination of the position of the unmanned aerial vehicle 900 to obtain an accuracy information associated to the determined position information.
  • the transmitter 940 transmits the taken image together with associated meta-data containing the position information of the taken image and the accuracy information of the position information.
  • an overview image may be generated according to the concept described above.
  • the image is taken means that the determination of the position and the accuracy information of the position may be triggered by taking an image. This may be done by a control unit.
  • the position and the accuracy information may be determined at the same time, the image is taken, with a tolerance depending on the system implementation (e.g. within 1s, 100ms, 10ms or less).
  • the sensor system 920 receives a GPS signal (Global Positioning System Signal) and determines the position of the unmanned aerial vehicle 900 based on the GPS signal. Further, the accuracy determiner may determine the accuracy of the determination of the position also based on the received GPS signal.
  • GPS signal Global Positioning System Signal
  • the accuracy determiner 930 may store a determined accuracy of the determination of the position and provide this accuracy information for all taken images until the accuracy of the determination of the position changes more than a threshold.
  • the transmitter may be a wireless transmitter, so that taken images with associated meta-data may be transmitted during the flight of the unmanned aerial vehicle 900.
  • the transmitter 940 may be a wire bound transmitter comprising a storage device for storing images and meta-data during the flight of the unmanned aerial vehicle 900.
  • the taken images and the meta-data may be transmitted (e.g. to an apparatus for generating an overview image of a plurality of images) after the unmanned aerial vehicle 900 has landed.
  • the sensor system 920 may determine additionally a yaw, a nick and a roll of the unmanned aerial vehicle 900 at the time the image is taken to obtain a yaw information, a nick information and a roll information associated to the taken image. Further, the accuracy determiner 930 may determine an accuracy of the determination of the yaw, the nick and the roll of the unmanned aerial vehicle 900 to obtain an accuracy information associated to the determined yaw information, the nick information and the roll information. The transmitter 940 may transmit the taken image together with associated meta-data containing the position information, the yaw information, the nick information and the roll information of the taken image and the accuracy information of the position information, the yaw information, the nick information and the roll information.
  • the sensor system 920 may determine one or more environmental parameters (e.g. wind force). This environmental parameter may be transmitted together with the other meta-data by the transmitter 940 or may be used by the accuracy determiner 930 to determine the accuracy of the determination of the position of the unmanned aerial vehicle 900 based on a determined environmental parameter. For example, if there is strong wind, the accuracy of the determination of the position may be lower than under weak wind conditions.
  • environmental parameters e.g. wind force
  • Some embodiments of the invention relate to a method for providing an image together with associated meta-data taken by an unmanned aerial vehicle, according to claim 12.
  • aspects of the described concept have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus.
  • embodiments of the invention can be implemented in hardware or in software.
  • the implementation can be performed using a digital storage medium, for example a floppy disk, a DVD, a Blue-Ray, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium may be computer readable.
  • Some examples according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.
  • embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer.
  • the program code may for example be stored on a machine readable carrier.
  • inventions comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.
  • an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.
  • a further example of the inventive methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein.
  • a further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.
  • the data stream or the sequence of signals may for example be configured to be transferred via a data communication connection, for example via the Internet.
  • a further example comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.
  • a processing means for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.
  • a further example comprises a computer having installed thereon the computer program for performing one of the methods described herein.
  • a programmable logic device for example a field programmable gate array
  • a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein.
  • the methods are preferably performed by any hardware apparatus.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Processing (AREA)

Claims (15)

  1. Vorrichtung (100, 200) zum Erzeugen eines Übersichtsbilds (124) einer Mehrzahl von Bildern, wobei jedes Bild der Mehrzahl von Bildern zugeordnete Metadaten aufweist, wobei die Vorrichtung folgende Merkmale aufweist:
    einen Bildvorprozessor (110), der ausgebildet ist, um ein neues Bild (102) durch Zuweisen des neuen Bilds (102) zu einer Position in dem Übersichtsbild (124) basierend auf einer Positionsinformation, die durch die Metadaten des neuen Bilds (102) beinhaltet ist, vorzuverarbeiten;
    eine Speichereinheit (120), die ausgebildet ist, um eine Mehrzahl vorverarbeiteter oder verarbeiteter Bilder des Übersichtsbilds (124) zu speichern, wobei jedes vorverarbeitete oder verarbeitete Bild (112, 122) der Mehrzahl vorverarbeiteter oder verarbeiteter Bilder einer Position in dem Übersichtsbild (124) zugewiesen ist, wobei die Speichereinheit (120) ausgebildet ist, um das Übersichtsbild (124), das die Mehrzahl vorverarbeiteter oder verarbeiteter Bilder an deren zugewiesenen Positionen beinhaltet, zum Anzeigen bereitzustellen; und
    einen Bildprozessor (130), der eine Genauigkeitsinformationseingabe (134) zum Empfangen einer Genauigkeitsinformation (104) der Positionsinformation und eine steuerbare Verarbeitungsmaschine (132) aufweist, wobei die Genauigkeitsinformation (104) der Positionsinformation durch Bestimmen einer Genauigkeit der Bestimmung der Position eines unbemannten Luftfahrzeugs unter Verwendung eines oder mehrerer Umweltparameter, die durch ein Sensorsystem (920) des unbemannten Luftfahrzeugs bestimmt werden, erhalten wird, wobei, wenn starker Wind geht, die Genauigkeit der Bestimmung der Position niedriger sein kann als unter Schwachwindbedingungen, wobei der Bildprozessor (130) ausgebildet ist, um eine Überlappungsregion des vorverarbeiteten neuen Bilds (112) und eines gespeicherten vorverarbeiteten oder gespeicherten verarbeiteten Bilds (112, 122) innerhalb des Übersichtsbilds (124) basierend auf der zugewiesenen Position des vorverarbeiteten neuen Bilds (112) und der zugewiesenen Position des gespeicherten vorverarbeiteten oder gespeicherten verarbeiteten Bilds (112, 122) zu bestimmen,
    wobei die steuerbare Verarbeitungsmaschine (132) ausgebildet ist, um das vorverarbeitete neue Bild (112) durch Neueinstellen der zugewiesenen Position des vorverarbeiteten neuen Bilds (112) basierend auf einem Vergleich von Merkmalen der Überlappungsregion des vorverarbeiteten Bilds (112) und Merkmalen der Überlappungsregion des gespeicherten vorverarbeiteten oder gespeicherten verarbeiteten Bilds (112, 122) zu verarbeiten, wobei die steuerbare Verarbeitungsmaschine (132) durch eine Genauigkeitsinformation (104) der Positionsinformation, die durch die Genauigkeitsinformationseingabe (134) empfangen wird, gesteuert wird, so dass eine maximale Neueinstellung der zugewiesenen Position des vorverarbeiteten neuen Bilds (112) basierend auf der empfangenen Genauigkeitsinformation (104) der Positionsinformation eingeschränkt ist,
    wobei die Speichereinheit (120) ausgebildet ist, um das verarbeitete neue Bild (122) mit der neu eingestellten zugewiesenen Position zu der Mehrzahl vorverarbeiteter oder verarbeiteter Bilder hinzuzufügen.
  2. Vorrichtung gemäß Anspruch 1, bei der der Bildvorprozessor (110) ausgebildet ist, um das neue Bild (102) durch Korrigieren einer Orientierung und einer Perspektivenverzerrung des neuen Bilds (102) basierend auf einer Rollen-Information, einer Stampfen-Information und einer Gieren-Information vorzuverarbeiten, wobei die Rollen-Information, die Stampfen-Information und die Gieren-Information durch die Metadaten des neuen Bilds (102) beinhaltet sind.
  3. Vorrichtung gemäß Anspruch 1 oder 2, bei der die steuerbare Verarbeitungsmaschine (132) ausgebildet ist, um das vorverarbeitete neue Bild (112) durch Neueinstellen der zugewiesenen Position, einer Orientierung oder einer Perspektivenverzerrung des vorverarbeiteten neuen Bilds (112) basierend auf dem Vergleich der Merkmale der Überlappungsregion zu verarbeiten, wobei die steuerbare Verarbeitungsmaschine (132) durch eine Genauigkeitsinformation (104) einer RollenInformation, einer Stampfen-Information oder eine Gieren-Information, die durch die Genauigkeitsinformationseingabe (134) empfangen wird, gesteuert wird, so dass die maximale Neueinstellung der zugewiesenen Position des verarbeiteten neuen Bilds (112), eine maximale Neueinstellung der Orientierung oder der Perspektivenverzerrung des vorverarbeiteten neuen Bilds (102) basierend auf der Genauigkeitsinformation (104) der Rollen-Information, der Stampfen-Information oder der GierenInformation eingeschränkt ist.
  4. Vorrichtung gemäß einem der Ansprüche 1 bis 3, bei der die Speichereinheit (120) ausgebildet ist, um das vorverarbeitete neue Bild (112) mit der zugewiesenen Position zu der Mehrzahl vorverarbeiteter oder verarbeiteter Bilder hinzuzufügen, und bei der die Speichereinheit (120) ausgebildet ist, um das Übersichtsbild, das das vorverarbeitete neue Bild (112) an der zugewiesenen Position beinhaltet, zum Anzeigen bereitzustellen, bevor das vorverarbeitete neue Bild (112) durch den Bildprozessor (130) verarbeitet wird.
  5. Vorrichtung gemäß Anspruch 4, bei der die Speichereinheit (120) ausgebildet ist, um das verarbeitete neue Bild (122) durch Ersetzen des gespeicherten vorverarbeiteten neuen Bilds (112) durch das verarbeitete neue Bild (122) hinzuzufügen, und bei der die Speichereinheit (120) ausgebildet ist, um das Übersichtsbild (124), das das verarbeitete neue Bild (122) an der neu eingestellten zugewiesenen Position beinhaltet, zum Anzeigen bereitzustellen.
  6. Vorrichtung gemäß einem der Ansprüche 1 bis 5, bei der der Bildvorprozessor (110) ausgebildet ist, um nacheinander eine Mehrzahl neuer Bilder (102) zu empfangen, und ausgebildet ist, um jedes empfangene neue Bild (102) vorzuverarbeiten, wobei die Speichereinheit (120) ausgebildet ist, um jedes vorverarbeitete neue Bild (112) zu speichern, und ausgebildet ist, um ein aktualisiertes Übersichtsbild (124) bereitzustellen, und zwar nach einem Speichern einer vordefinierten Anzahl vorverarbeiteter neuer Bilder (112).
  7. Vorrichtung gemäß einem der Ansprüche 1 bis 6, die eine Steuereinheit (240) für ein unbemanntes Luftfahrzeug aufweist, die ausgebildet ist, um ein Steuersignal (242) zum Lotsen eines unbemannten Luftfahrzeugs zu einer Region zu erzeugen, die einem Bereich des Übersichtsbilds (124) entspricht, der nicht durch ein Bild der Mehrzahl vorverarbeiteter oder verarbeiteter Bilder abgedeckt ist.
  8. Vorrichtung gemäß einem der Ansprüche 1 bis 7, bei der der Bildprozessor (110) ausgebildet ist, um Merkmalspunkte des vorverarbeiteten neuen Bilds (112) zu bestimmen, und ausgebildet ist, um die bestimmten Merkmalspunkte des vorverarbeiteten neuen Bilds (112) mit Merkmalspunkten eines gespeicherten verarbeiteten Bilds (122) zu vergleichen, um gemeinsame Merkmalspunkte zu identifizieren und dreidimensionale Positionen der gemeinsamen Merkmalspunkte zu erhalten, wobei der Bildprozessor (130) ausgebildet ist, um gemeinsame Merkmalspunkte, die sich innerhalb einer vordefinierten maximalen Entfernung von Relevanz zu einer Referenzebene befinden, basierend auf den dreidimensionalen Positionen der gemeinsamen Merkmalspunkte zu bestimmen, um relevante gemeinsame Merkmalspunkte zu identifizieren, wobei der Bildprozessor (130) ausgebildet ist, um die Position des vorverarbeiteten neuen Bilds (112) in dem Übersichtsbild (124) basierend auf einem Vergleich einer Bildinformation jedes relevanten gemeinsamen Merkmalspunkts des vorverarbeiteten neuen Bilds (112) mit einer Bildinformation jedes entsprechenden relevanten gemeinsamen Merkmalspunkts des gespeicherten verarbeiteten Bilds (122) neu einzustellen, ohne gemeinsame Merkmalspunkte zu berücksichtigen, die sich jenseits der vordefinierten maximalen Entfernung von Relevanz zu der Referenzebene befinden.
  9. Vorrichtung gemäß Anspruch 8, bei der der Bildprozessor (130) ausgebildet ist, um die Referenzebene basierend auf den dreidimensionalen Positionen der gemeinsamen Merkmalspunkte zu bestimmen.
  10. Vorrichtung gemäß Anspruch 8 oder 9, bei der die Speichereinheit (120) ausgebildet ist, um das neue verarbeitete Bild (122) zusammen mit den bestimmten Merkmalspunkten des neuen verarbeiteten Bilds (122) zu der Mehrzahl vorverarbeiteter oder verarbeiteter Bilder hinzuzufügen.
  11. Unbemanntes Luftfahrzeug (900), das folgende Merkmale aufweist:
    eine Kamera (910), die ausgebildet ist, um während eines Flugs des unbemannten Luftfahrzeugs (900) ein Bild aufzunehmen;
    ein Sensorsystem (920), das ausgebildet ist, um eine Position des unbemannten Luftfahrzeugs (900) zu einer Zeit, zu der das Bild aufgenommen wird, zu bestimmen, um eine Positionsinformation zu erhalten, die dem aufgenommenen Bild zugeordnet ist, wobei das Sensorsystem (920) ausgebildet ist, um ein Gieren, ein Nicken oder ein Rollen des unbemannten Luftfahrzeugs (900) zu einer Zeit, zu der das Bild aufgenommen wird, zu bestimmen, um eine Gieren-Information, eine NickenInformation oder eine Rollen-Information, die dem aufgenommenen Bild zugeordnet ist, zu erhalten;
    einen Genauigkeitsbestimmer (930), der ausgebildet ist, um eine Genauigkeit der Bestimmung der Position des unbemannten Luftfahrzeugs (900) unter Verwendung eines oder mehrerer Umweltparameter zu bestimmen, die durch das Sensorsystem (920) bestimmt werden, um eine Genauigkeitsinformation zu erhalten, die der bestimmten Positionsinformation zugeordnet ist, wobei, wenn starker Wind geht, die Genauigkeit der Bestimmung der Position niedriger sein kann als unter Schwachwindbedingungen, wobei der Genauigkeitsbestimmer (930) ausgebildet ist, um eine Genauigkeit der Bestimmung des Gierens, des Nickens oder des Rollens des unbemannten Luftfahrzeugs (900) zu bestimmen, um eine Genauigkeitsinformation zu erhalten, die der bestimmten Gieren-Information, Nicken-Information oder RollenInformation zugeordnet ist;
    einen Sender, der ausgebildet ist, um das aufgenommene Bild zusammen mit zugeordneten Metadaten an die Vorrichtung aus Anspruch 1 zu senden, wobei die Metadaten die Positionsinformation des aufgenommenen Bilds und die Genauigkeitsinformation der Positionsinformation beinhalten, sowie die Gieren-Information, die Nicken-Information, die Rollen-Information, die Genauigkeitsinformation der Gieren-Information, die Genauigkeitsinformation der Nicken-Information und die Genauigkeitsinformation der Rollen-Information.
  12. Verfahren zum Bereitstellen eines Bilds zusammen mit zugeordneten Metadaten, das durch ein unbemanntes Luftfahrzeug aufgenommen wird, das folgende Schritte aufweist:
    Aufnehmen eines Bilds während eines Flugs des unbemannten Luftfahrzeugs;
    Bestimmen einer Position des unbemannten Luftfahrzeugs zu einer Zeit, zu der das Bild aufgenommen wird, um eine Positionsinformation zu erhalten, die dem aufgenommenen Bild zugeordnet ist;
    Bestimmen eines Gierens, eines Nickens oder eines Rollens des unbemannten Luftfahrzeugs zu einer Zeit, zu der das Bild aufgenommen wird, um eine GierenInformation, eine Nicken-Information oder eine Rollen-Information, die dem aufgenommenen Bild zugeordnet ist, zu erhalten;
    Bestimmen einer Genauigkeit der Bestimmung der Position des unbemannten Luftfahrzeugs unter Verwendung eines oder mehrerer Umweltparameter, die durch ein Sensorsystem (920) des unbemannten Luftfahrzeugs bestimmt werden, um eine Genauigkeitsinformation, die der bestimmten Positionsinformation zugeordnet ist, zu erhalten, wobei, wenn starker Wind geht, die Genauigkeit der Bestimmung der Position niedriger sein kann als unter Schwachwindbedingungen;
    Bestimmen einer Genauigkeit der Bestimmung des Gierens, des Nickens oder des Rollens des unbemannten Luftfahrzeugs, um eine Genauigkeitsinformation zu erhalten, die der bestimmten Gieren-Information, Nicken-Information oder Rollen-Information zugeordnet ist;
    Senden des aufgenommenen Bilds zusammen mit zugeordneten Metadaten an die Vorrichtung aus Anspruch 1, wobei die Metadaten die Positionsinformation des aufgenommenen Bilds und die Genauigkeitsinformation der Positionsinformation beinhalten, sowie die Gieren-Information, die Nicken-Information, die Rollen-Information, die Genauigkeitsinformation der Gieren-Information, die Genauigkeitsinformation der Nicken-Information und die Genauigkeitsinformation der Rollen-Information.
  13. Verfahren (500) zum Erzeugen eines Übersichtsbilds einer Mehrzahl von Bildern, wobei jedes Bild der Mehrzahl von Bildern zugeordnete Metadaten aufweist, wobei das Verfahren folgende Schritte aufweist:
    Vorverarbeiten (510) eines neuen Bilds durch Zuweisen des neuen Bilds zu einer Position in dem Übersichtsbild basierend auf einer Positionsinformation, die durch die Metadaten des neuen Bilds beinhaltet ist;
    Speichern (520) einer Mehrzahl vorverarbeiteter oder verarbeiteter Bilder des Übersichtsbilds, wobei jedes vorverarbeitete oder verarbeitete Bild der Mehrzahl vorverarbeiteter oder verarbeiteter Bilder eine zugewiesene Position in dem Übersichtsbild aufweist;
    Bestimmen (530) einer Überlappungsregion des vorverarbeiteten neuen Bilds und eines gespeicherten vorverarbeiteten oder gespeicherten verarbeiteten Bilds innerhalb des Übersichtsbilds basierend auf der zugewiesenen Position des vorverarbeiteten neuen Bilds und der zugewiesenen Position des gespeicherten vorverarbeiteten oder gespeicherten verarbeiteten Bilds;
    Verarbeiten (540) des vorverarbeiteten Bilds durch Neueinstellen der zugewiesenen Position des vorverarbeiteten neuen Bilds basierend auf einem Vergleich von Merkmalen der Überlappungsregion des vorverarbeiteten neuen Bilds und des gespeicherten vorverarbeiteten oder gespeicherten verarbeiteten Bilds, wobei eine maximale Neueinstellung der zugewiesenen Position des vorverarbeiteten neuen Bilds basierend auf einer Genauigkeitsinformation der Positionsinformation eingeschränkt ist, wobei die Genauigkeitsinformation der Positionsinformation durch Bestimmen einer Genauigkeit der Bestimmung der Position eines unbemannten Luftfahrzeugs unter Verwendung eines oder mehrerer Umweltparameter, die durch ein Sensorsystem (920) des unbemannten Luftfahrzeugs bestimmt werden, erhalten wird, wobei, wenn starker Wind geht, die Genauigkeit der Bestimmung der Position niedriger sein kann als unter Schwachwindbedingungen;
    Hinzufügen (550) des verarbeiteten neuen Bilds mit der neu eingestellten zugewiesenen Position zu der Mehrzahl vorverarbeiteter oder verarbeiteter Bilder; und
    Bereitstellen (560) des Übersichtsbilds, das die Mehrzahl vorverarbeiteter oder verarbeiteter Bilder an deren zugewiesenen Positionen beinhaltet, zum Anzeigen.
  14. Verfahren gemäß Anspruch 13, das ferner folgende Schritte aufweist:
    Hinzufügen des vorverarbeiteten neuen Bilds zu der Mehrzahl vorverarbeiteter oder verarbeiteter Bilder des Übersichtsbilds;
    Anzeigen des Übersichtsbilds, das ein vorverarbeitetes neues Bild an der zugewiesenen Position beinhaltet; und
    Anzeigen des Übersichtsbilds, das das verarbeitete neue Bild an der neu eingestellten zugewiesenen Position beinhaltet.
  15. Computerprogramm mit einem Programmcode zum Durchführen des Verfahrens gemäß einem der Ansprüche 12 bis 14, wenn das Computerprogramm auf einem Computer oder einer Mikrosteuerung abläuft.
EP10174053.8A 2010-08-25 2010-08-25 Vorrichtung und Verfahren zur Erzeugung eines Übersichtsbildes einer Vielzahl von Bildern mittels einer Genauigkeitsinformation Not-in-force EP2423871B1 (de)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP10174053.8A EP2423871B1 (de) 2010-08-25 2010-08-25 Vorrichtung und Verfahren zur Erzeugung eines Übersichtsbildes einer Vielzahl von Bildern mittels einer Genauigkeitsinformation
US13/215,366 US8797400B2 (en) 2010-08-25 2011-08-23 Apparatus and method for generating an overview image of a plurality of images using an accuracy information

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
EP10174053.8A EP2423871B1 (de) 2010-08-25 2010-08-25 Vorrichtung und Verfahren zur Erzeugung eines Übersichtsbildes einer Vielzahl von Bildern mittels einer Genauigkeitsinformation

Publications (2)

Publication Number Publication Date
EP2423871A1 EP2423871A1 (de) 2012-02-29
EP2423871B1 true EP2423871B1 (de) 2014-06-18

Family

ID=43902868

Family Applications (1)

Application Number Title Priority Date Filing Date
EP10174053.8A Not-in-force EP2423871B1 (de) 2010-08-25 2010-08-25 Vorrichtung und Verfahren zur Erzeugung eines Übersichtsbildes einer Vielzahl von Bildern mittels einer Genauigkeitsinformation

Country Status (2)

Country Link
US (1) US8797400B2 (de)
EP (1) EP2423871B1 (de)

Families Citing this family (52)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2013105926A1 (en) * 2011-03-22 2013-07-18 Aerovironment Inc. Invertible aircraft
EP2511658A1 (de) * 2011-04-14 2012-10-17 Hexagon Technology Center GmbH Vermessungssystem und Verfahren zur Neupunktbestimmung
EP2527787B1 (de) * 2011-05-23 2019-09-11 Kabushiki Kaisha TOPCON Luftbildaufnahmeverfahren und Luftbildaufnahmevorrichtung
US8817067B1 (en) * 2011-07-29 2014-08-26 Google Inc. Interface for applying a photogrammetry algorithm to panoramic photographic images
WO2013033954A1 (zh) 2011-09-09 2013-03-14 深圳市大疆创新科技有限公司 陀螺式动态自平衡云台
US20140327733A1 (en) * 2012-03-20 2014-11-06 David Wagreich Image monitoring and display from unmanned vehicle
JP6122591B2 (ja) 2012-08-24 2017-04-26 株式会社トプコン 写真測量用カメラ及び航空写真装置
US20140098296A1 (en) * 2012-10-04 2014-04-10 Ati Technologies Ulc Method and apparatus for changing a perspective of a video
US10771748B2 (en) 2012-11-27 2020-09-08 Metropolitan Life Insurance Co. System and method for interactive aerial imaging
US8903568B1 (en) 2013-07-31 2014-12-02 SZ DJI Technology Co., Ltd Remote control method and terminal
US9460554B2 (en) * 2013-09-09 2016-10-04 International Business Machines Corporation Aerial video annotation
DE102013015596A1 (de) * 2013-09-19 2015-03-19 Jörg Brinkmeyer Automatische Kamera-Nachführung durch einen Sender am sich bewegenden Point-of-Interest mit GPS Unterstützung
CN104903790B (zh) 2013-10-08 2018-07-24 深圳市大疆灵眸科技有限公司 用于增稳和减振的装置和方法
US9562771B2 (en) * 2013-12-18 2017-02-07 Sharper Shape Ltd Analysis of sensor data
TW201534125A (zh) * 2014-02-24 2015-09-01 Hon Hai Prec Ind Co Ltd 遠端視訊監控系統及方法
US20170059317A1 (en) * 2014-05-07 2017-03-02 Datumate Ltd. System and method for automatically generating cad field sketches and automatic validation thereof
KR102165450B1 (ko) * 2014-05-22 2020-10-14 엘지전자 주식회사 무인 항공기를 제어하는 포터블 디바이스 및 그 제어 방법
CN104021535B (zh) * 2014-06-11 2016-09-21 中国科学院电子学研究所 步进分幅ccd图像拼接的方法
ES2908557T3 (es) 2014-06-13 2022-05-03 Twitter Inc Vehículo aéreo no tripulado con capacidad de mensajería
US9665094B1 (en) * 2014-08-15 2017-05-30 X Development Llc Automatically deployed UAVs for disaster response
US9463875B2 (en) 2014-09-03 2016-10-11 International Business Machines Corporation Unmanned aerial vehicle for hazard detection
US10489407B2 (en) * 2014-09-19 2019-11-26 Ebay Inc. Dynamic modifications of results for search interfaces
US10752378B2 (en) * 2014-12-18 2020-08-25 The Boeing Company Mobile apparatus for pest detection and engagement
CN106029501B (zh) 2014-12-23 2019-05-14 深圳市大疆灵眸科技有限公司 Uav全景成像
CN111506109B (zh) * 2014-12-31 2023-08-25 深圳市大疆创新科技有限公司 传感器数据的选择性处理
US9471059B1 (en) 2015-02-17 2016-10-18 Amazon Technologies, Inc. Unmanned aerial vehicle assistant
CN108370416A (zh) * 2015-12-17 2018-08-03 皇家Kpn公司 从视频流生成输出视频
FR3046228B1 (fr) * 2015-12-29 2018-01-05 Thales Procede de representation graphique tridimensionnelle d'une piste d'atterrissage sur un dispositif de visualisation d'aeronef
AU2016234213A1 (en) * 2016-03-31 2017-10-19 Komatsu Ltd. Control system for work machine, work machine, and control method for work machine
JP6852851B2 (ja) * 2016-08-30 2021-03-31 エスゼット ディージェイアイ テクノロジー カンパニー リミテッドＳｚ Ｄｊｉ Ｔｅｃｈｎｏｌｏｇｙ Ｃｏ．，Ｌｔｄ 画像処理方法及び移動体を制御するシステム
KR20180068411A (ko) * 2016-12-14 2018-06-22 삼성전자주식회사 무인 비행 전자 장치의 운행 제어 방법 및 이를 지원하는 전자 장치
US10602056B2 (en) * 2017-01-13 2020-03-24 Microsoft Technology Licensing, Llc Optimal scanning trajectories for 3D scenes
US9986233B1 (en) * 2017-03-16 2018-05-29 Amazon Technologies, Inc. Camera calibration using fixed calibration targets
US10554950B1 (en) 2017-03-16 2020-02-04 Amazon Technologies, Inc. Collection of camera calibration data using augmented reality
US10447995B1 (en) 2017-03-16 2019-10-15 Amazon Technologies, Inc. Validation of camera calibration data using augmented reality
WO2018198634A1 (ja) * 2017-04-28 2018-11-01 ソニー株式会社 情報処理装置、情報処理方法、情報処理プログラム、画像処理装置および画像処理システム
JP7046506B2 (ja) * 2017-06-12 2022-04-04 キヤノン株式会社 情報処理装置、情報処理方法及びプログラム
JP6791387B2 (ja) * 2017-08-08 2020-11-25 日本電気株式会社 飛行体、飛行体制御装置、飛行体制御方法および飛行体制御プログラム
US10638109B2 (en) * 2017-09-15 2020-04-28 Elphel, Inc. Method for the FPGA-based long range multi-view stereo with differential image rectification
KR101854181B1 (ko) 2017-10-20 2018-05-04 이병섭 무인항공기를 이용한 타겟 탐색 시스템
EP3707892B1 (de) 2017-11-10 2023-08-16 Koninklijke KPN N.V. Erhalten von bilddaten von einem objekt in einer szene
ES2848851T3 (es) * 2018-04-30 2021-08-12 Tata Consultancy Services Ltd Procedimiento y sistema para la construcción de imágenes basados en la costura de fotogramas en un entorno interior
US10970924B2 (en) * 2018-06-17 2021-04-06 Foresight Ai Inc. Reconstruction of a scene from a moving camera
US11125800B1 (en) 2018-10-31 2021-09-21 United Services Automobile Association (Usaa) Electrical power outage detection system
US11854262B1 (en) 2018-10-31 2023-12-26 United Services Automobile Association (Usaa) Post-disaster conditions monitoring system using drones
US11789003B1 (en) 2018-10-31 2023-10-17 United Services Automobile Association (Usaa) Water contamination detection system
US11012750B2 (en) * 2018-11-14 2021-05-18 Rohde & Schwarz Gmbh & Co. Kg Method for configuring a multiviewer as well as multiviewer
TWI693829B (zh) * 2018-12-24 2020-05-11 緯創資通股份有限公司 電子裝置及其影像測距方法
US11221626B2 (en) * 2019-04-23 2022-01-11 HERE Global, B.V. Drone-based collection of location-related data
CN110503622B (zh) * 2019-08-23 2022-07-01 上海圭目机器人有限公司 基于定位数据的图像全局定位优化拼接方法
US20220334789A1 (en) * 2021-04-20 2022-10-20 Streem, Llc Augmented reality video stream synchronization
CN113344030B (zh) * 2021-05-11 2023-11-03 淮阴工学院 一种基于判决相关分析的遥感图像特征融合方法及系统

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6819982B2 (en) * 2002-11-26 2004-11-16 The Boeing Company Uninhabited airborne vehicle in-flight refueling system
US7418320B1 (en) * 2005-01-24 2008-08-26 International Business Machines Corporation Navigating a UAV having an on-board digital camera to capture desired geographic area
WO2008002875A2 (en) * 2006-06-26 2008-01-03 Lockheed Martin Corporation Method and system for providing a perspective view image by intelligent fusion of a plurality of sensor data
US8497905B2 (en) * 2008-04-11 2013-07-30 nearmap australia pty ltd. Systems and methods of capturing large area images in detail including cascaded cameras and/or calibration features
US8219267B2 (en) * 2010-05-27 2012-07-10 Honeywell International Inc. Wind estimation for an unmanned aerial vehicle

Also Published As

Publication number Publication date
US8797400B2 (en) 2014-08-05
EP2423871A1 (de) 2012-02-29
US20120050524A1 (en) 2012-03-01

Similar Documents

Publication Publication Date Title
EP2423871B1 (de) Vorrichtung und Verfahren zur Erzeugung eines Übersichtsbildes einer Vielzahl von Bildern mittels einer Genauigkeitsinformation
EP2423873B1 (de) Vorrichtung und Verfahren zur Erzeugung eines Übersichtsbildes einer Vielzahl von Bildern mittels Referenzebene
Yahyanejad et al. Incremental mosaicking of images from autonomous, small-scale uavs
JP7485749B2 (ja) ビデオベースの位置決め及びマッピングの方法及びシステム
Nassar et al. A deep CNN-based framework for enhanced aerial imagery registration with applications to UAV geolocalization
Majdik et al. Air‐ground matching: Appearance‐based GPS‐denied urban localization of micro aerial vehicles
Yahyanejad et al. A fast and mobile system for registration of low-altitude visual and thermal aerial images using multiple small-scale UAVs
US9530235B2 (en) Aligning panoramic imagery and aerial imagery
US8547375B2 (en) Methods for transferring points of interest between images with non-parallel viewing directions
Caballero et al. Improving vision-based planar motion estimation for unmanned aerial vehicles through online mosaicing
Tanchenko et al. UAV navigation system Autonomous correction algorithm based on road and river network recognition
US11513211B2 (en) Environment model using cross-sensor feature point referencing
Karantzalos et al. Model-based building detection from low-cost optical sensors onboard unmanned aerial vehicles
Avola et al. Homography vs similarity transformation in aerial mosaicking: which is the best at different altitudes?
Tsao et al. Stitching aerial images for vehicle positioning and tracking
Shukla et al. Automatic geolocation of targets tracked by aerial imaging platforms using satellite imagery
Shahbazi Professional drone mapping
US12000703B2 (en) Method, software product, and system for determining a position and orientation in a 3D reconstruction of the earth's surface
Han Test and evaluation of an image-matching navigation system for a UAS operating in a GPS-denied environment
US20240085186A1 (en) A method, software product, and system for determining a position and orientation in a 3d reconstruction of the earth's surface
Yahyanejad Orthorectified mosacking of images from small-scale unmanned aerial vehicles
Castillo-Carrión et al. Autonomous 3D metric reconstruction from uncalibrated aerial images captured from UAVs
Wischounig-Strucl et al. A structure based mosaicking approach for aerial images from low altitude of non-planar scenes
He 3d reconstruction from passive sensors
Mirisola Exploiting attitude sensing in vision-based navigation, mappping and tracking including results from an airship

Legal Events

Date Code Title Description
17P Request for examination filed

Effective date: 20110817

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME RS

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

INTG Intention to grant announced

Effective date: 20140103

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 673706

Country of ref document: AT

Kind code of ref document: T

Effective date: 20140715

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602010016742

Country of ref document: DE

Effective date: 20140731

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140919

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140918

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

REG Reference to a national code

Ref country code: NL

Ref legal event code: VDEP

Effective date: 20140618

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 673706

Country of ref document: AT

Kind code of ref document: T

Effective date: 20140618

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20141020

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20141018

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602010016742

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140825

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20140831

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20140831

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20140831

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

26N No opposition filed

Effective date: 20150319

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20140825

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20100825

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 7

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 8

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 9

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140618

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20190826

Year of fee payment: 10

Ref country code: DE

Payment date: 20190716

Year of fee payment: 10

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20190827

Year of fee payment: 10

REG Reference to a national code

Ref country code: DE

Ref legal event code: R119

Ref document number: 602010016742

Country of ref document: DE

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20200825

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210302

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200831

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200825