KR101837403B1 - Method and Apparatus for Fast mosaicking of Unmanned Aerial Vehicle Images - Google Patents

Abstract

The present invention relates to a method and device for high-speed image mosaicking and, more specifically, relates to a method and device to generate a mosaic image of a target area at high speed by using images photographed through an unmanned aircraft. According to the present invention, the device for mosaicking of an image photographed through an unmanned aircraft includes: an image obtaining part obtaining images, which are overlapped and photographed about a target area by the unmanned aircraft flying along a preset route; a mosaic image plane determining part determining a reference image which becomes a mosaic image plane; a conversion relation deriving part deriving a conversion relation between each of the obtained images and the mosaic image plane corresponding to the reference image; a geometric correction part performing geometric correction on the obtained images by using the conversion relation between each of the images and the mosaic image plane; and an image collecting part collecting the images, which have gone through the geometric correction, as a single mosaic image.

Description

TECHNICAL FIELD [0001] The present invention relates to a high-speed image mosaicking method and apparatus,

The present invention relates to a high-speed image mosaicking method and apparatus for an image photographed on an unmanned airplane, and more particularly, to a method and apparatus for generating a mosaic image for a target area at high speed using images photographed from an unmanned airplane.

Unmanned aerial vehicles with high spatial resolution and period resolution are bringing innovation to many applications, including disaster management, agricultural monitoring, tracking and surveillance. However, the high spatial resolution of unmanned aerial photographs, on the contrary, limits the surface area that individual images can cover. Therefore, in order to monitor a wide area using an unmanned aerial vehicle image, it is essential that an image mosaicking process of collecting a plurality of images of a target area into a single image is essential.

The mosaicking method of the unmanned airplane image can be divided into DSM (Digital Surface Model) based, GCP (Ground Control Point) based, and correspondent point based method depending on the presence or absence of prior information about the target area and its type. Although DSM and GCP based methods enable accurate mosaic image generation, they have limitations in terms of availability because they require advance information on the target area. For this reason, considering the versatility of the methodology in many image monitoring fields, a correspondence point based method capable of generating a mosaic image based only on the correspondence point information between adjacent images is widely used as a method of mosaicing the image.

Corresponding point based image mosaicking method generally consists of image acquisition, corresponding point extraction, image geometry correction, image radiation correction, and image acquisition. Conventional correspondence-based image mosaicking method assumes high redundancy between adjacent images as a prerequisite for the mosaic. This is a way to increase the stability and accuracy of estimation by using a sufficient number of correspondence points when estimating the relative transformation relation between two adjacent images. However, the high degree of redundancy of the image greatly increases the number of image pairs to be compared for the corresponding point extraction as well as the search area for the corresponding point extraction in the individual image pairs. As a result, it causes the calculation amount of the correspondence point extraction process which requires the most processing time in the image mosaicking process to be increased.

Although an attempt has been made to quickly generate a mosaic image by avoiding the above-described problem, an essential solution has not been proposed, for example, assuming acquisition of a precise external facial expression element for an image.

As a related prior art, Korean Patent No. 10-1323099 (published on May 28, 2013) and the like can be referred to.

It is an object of the present invention to provide a method and apparatus for high-speed mosaicking of a plurality of unmanned aerial photographs of a large target area.

A high-speed image mosaicking apparatus for an image photographed by an unmanned airplane according to the present invention is characterized in that images captured by the unmanned airplane moving along a predetermined flight path so as to be overlapped with each other are acquired from the unmanned airplane A mosaic image plane determining unit for determining a reference image that is a mosaic image plane among the acquired images, a mosaic image plane determining unit for determining a mosaic image plane corresponding to the reference image, An image geometry correcting unit for performing geometric correction on the acquired images by using a conversion relation derived between each of the obtained images and a mosaic image plane corresponding to the reference image, And an image collecting unit for collecting the performed images into a single mosaic image .

In one embodiment, the mosaic image plane determining unit determines the reference image set to the mosaic plane by using an external facial expression element acquired at the time of capturing images photographed by the UAV, And three-dimensional position information and rotation angle information of an image sensor for photographing the images.

In an exemplary embodiment, the reference image may be the most centrally located image in the image distribution of the acquired images analyzed based on the three-dimensional position information.

In an exemplary embodiment, the mosaic image plane determining unit may extract correspondence points between images of the acquired images that are adjacent to each other, extract a corresponding maximum number of corresponding points from a maximum spanning tree (MST) ), And determines the reference image as a node that minimizes the depth of the tree.

In one embodiment, the mosaic image plane determining unit may extract an overlapping portion from the adjacent images from the row and column information of each of the acquired images based on the flight path, And at least one corresponding point extraction area is determined.

In an exemplary embodiment, the corresponding points extraction is performed between images in which captured positions of the acquired images are mutually adjacent, only in the region where the determined corresponding point extraction regions are overlapped with each other.

A high-speed image mosaicking method for an image photographed on an unmanned airplane according to the present invention is a method for acquiring images taken from an unmanned airplane moving along a predetermined flight path so as to be overlapped with each other with respect to a target area, Determining a reference image that is a mosaic image plane of the acquired images, deriving a mosaic image plane corresponding to the reference image, and a conversion relation of each of the acquired images, Performing a geometric correction on each of the acquired images by using a conversion relation derived between each of the reference images and a mosaic image plane corresponding to the reference image and collecting the geometrically corrected images into a single mosaic image The method comprising the steps of:

The high-speed image mosaicking method and apparatus according to the present invention can generate a mosaic image at high speed by overcoming a pre-constraint problem such as a speed reduction problem due to a high degree of redundancy between neighboring images and securing accurate external facial expression elements.

Further, the method and apparatus for high-speed image mosaicking according to the present invention grasp, in advance, which part of the image is overlapped with the adjacent images from the row and column information of each image on the flight path, The corresponding points can be extracted by considering only the overlapping regions. Therefore, it is possible to improve the extraction speed and accuracy of the corresponding point.

1 is a view for explaining a high-speed image mosaicking apparatus according to an embodiment of the present invention.
2 is a flowchart illustrating an operation of a high-speed image mosaicking apparatus according to an embodiment of the present invention.
3 is a reference diagram for explaining the image acquisition process of FIG.
4 is a reference view for explaining the mosaic plane determination process of FIG.
5 is a reference diagram for explaining the conversion relationship derivation process of FIG.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like or similar components are denoted by the same reference numerals, and redundant explanations thereof will be omitted. The suffix "module" and " part "for the components used in the following description are given or mixed in consideration of ease of specification, and do not have their own meaning or role. In the following description of the embodiments of the present invention, a detailed description of related arts will be omitted when it is determined that the gist of the embodiments disclosed herein may be blurred. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. , ≪ / RTI > equivalents, and alternatives.

Terms including ordinals, such as first, second, etc., may be used to describe various elements, but the elements are not limited to these terms. The terms are used only for the purpose of distinguishing one component from another.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The singular expressions include plural expressions unless the context clearly dictates otherwise.

In the present application, the terms "comprises", "having", and the like are used to specify that a feature, a number, a step, an operation, an element, a component, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Reference will now be made in detail to the preferred embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout.

1 is a view for explaining a high-speed image mosaicking apparatus 100 according to an embodiment of the present invention. 1, a high-speed image mosaicking apparatus 100 according to an embodiment of the present invention includes an image acquisition unit 110, a mosaic image plane determination unit 120, a conversion relation deriving unit 130, A correction unit 140, and an image processor 150.

The image acquisition unit 110 acquires images photographed by the unmanned air vehicle 10.

The unmanned airplane 10 moves along the previously planned flight path 20 (predetermined flight path), captures an image so that the object region (or the object region) is overlapped with the object region (or the object region) And transmits the image. At this time, the external facial expression element information can be further acquired at the time of photographing, and the acquired external facial expression element information can also be transmitted to the image capturing unit 110. [

The external facial expression element information may be acquired from an image sensor mounted on an unmanned airplane when capturing an image and at least one other sensor interlocked with the image sensor.

, κ)을 포함한다.Here, the external facial expressions include three-dimensional positions (Tx, Ty, Tz) of the image sensor and rotational angles (ω,

, K).

The image acquisition unit 110 records the image and external facial expression element information received from the UAV 10 (S110 in FIG. 2).

The mosaic image plane determination unit 120 determines a reference image to be set as a mosaic image plane among the acquired images (S120 in FIG. 2).

That is, the determined reference image is a mosaic image plane.

Here, the reference image is determined by using the external facial expression elements of the images, or when the external facial expression element information is absent, the corresponding points are extracted from the adjacent images and their connectivity is analyzed and determined.

For example, when determining that the reference image is available, the mosaic image plane determining unit 120 uses the location information of the external facial expression element to determine the closest image to the center of the image distribution as a reference image .

On the other hand, when there is no external facial expression information, the correspondence point between neighboring images is extracted, and the maximum spanning tree (MST) is calculated by weighting the number of corresponding points extracted. .

3, the mosaic image plane determining unit 120 extracts the corresponding points from the row and column information of each image for the entire flight path 20, as shown in FIG. 3, It is determined which portion of the target image overlaps.

The mosaic image plane determining unit 120 can distinguish between overlapping portions and non-overlapping portions between images using various positional information such as a position at which the image is captured, a position of the sensor where the image is captured, and an angle of the sensor .

The mosaic image plane determination unit 120 does not extract the corresponding point in the portion of the image that is the object of the corresponding point extraction and in which the adjacent images are not overlapped.

FIG. 4 is a conceptual diagram conceptually showing the shape of a matching area mask that each of the individual images can have, which is indexed from -1 to 15. FIG. The mosaic image plane determination unit 120 extracts corresponding points from the overlapping portions of adjacent images for each matching region.

For example, each of the matching regions may be divided into an upper edge region 1a, a left edge region 1b, a lower edge region 1c, a right edge region 1b, And a spot area 1d. For example, in the matching area mask indexed with the number "1 ", only the upper edge region 1a may be adjacent to the adjacent images, and the mosaic image plane determining unit 120 may determine the upper edge region 1a), it is possible to perform the corresponding point extraction with the adjacent images.

Therefore, according to the present invention, the time required for extracting corresponding points can be reduced.

As a result, the correspondence point extraction region for each image can be predefined as a total of 16 types as shown in FIG. 4. The correspondence point extraction with neighboring images considers only the overlapping regions, Can be improved.

The conversion relation deriving unit 130 derives a conversion relation for each image to a mosaic image plane using an external facial expression element or corresponding point information of the image.

For example, when deriving the conversion relation to the mosaic image plane, the conversion relation deriving unit 130 may determine that the conversion relation of one image exists in the reference Is calculated as the product of the inverse of the transformation matrix from the image coordinate system to the ground coordinate system of the image and the transformation matrix from the image coordinate system to the ground coordinate system of the corresponding image.

[Equation 1]

는 i번째 영상의 영상 좌표계에서 지상 좌표계로 가는 3×3크기의 변환행렬,

는 기준영상의 영상 좌표계에서 지상 좌표계로 가는 3×3크기의 변환행렬,

는 i번째 영상 좌표계에서 모자이크 영상 평면으로 가는 3×3크기의 변환행렬.In Equation (1)

Is a 3x3 transform matrix from the image coordinate system of the i-th image to the ground coordinate system,

Is a 3x3 transform matrix from the image coordinate system of the reference image to the ground coordinate system,

Is a 3x3 transform matrix from the i-th image coordinate system to the mosaic image plane.

On the other hand, when the external facial expression information is not available, the conversion relation from the relative transformation relations between the neighboring images estimated using the corresponding point information to the mosaic image plane for each image is derived. For example, when a total of seven images (A to G) are acquired as shown in FIG. 5 and a reference image C that minimizes the maximum extension tree and tree depth from the number n of corresponding points extracted between adjacent images is determined , And the transformation relation of the image B with respect to the mosaic image plane is calculated as a product of relative transformation relations connected to each other from the image B to the reference image C as shown in Equation (2).

&Quot; (2) "

는 영상 B의 영상 좌표계에서 영상 A의 영상 좌표계로 가는 3×3크기의 변환행렬,

는 영상 A의 영상 좌표계에서 영상 C(기준영상)의 영상 좌표계로 가는 3×3크기의 변환행렬,

는 영상 B의 영상 좌표계에서 모자이크 영상 평면으로 가는 3×3크기의 변환행렬.In Equation (2)

A 3x3 transform matrix from the image coordinate system of the image B to the image coordinate system of the image A,

Is a 3x3 transform matrix from the image coordinate system of image A to the image coordinate system of image C (reference image)

Is a 3x3 transform matrix from the image coordinate system of image B to the mosaic image plane.

Here, the relative transformation relationship estimation can be performed using a central projection transformation, which is a nonlinear model with eight degrees of freedom.

Also, in the case where the degree of overlap with the adjacent image is not limited as in the present invention, in order to ensure the stability of the model estimation, it is possible to apply the affine transformation, which is a linear model having six degrees of freedom, The conversion model can be selected adaptively.

On the other hand, the image geometry correcting unit 140 corrects the images using the conversion relation of each image for the mosaic image plane determined by the conversion relation deriving unit 130. [

The image geometry correcting unit 140 defines the size of the resulting image from the transformation relation of each image, calculates the pixel position of the input image corresponding to the individual pixel position, and then performs the interpolation of the pixel value, Perform calibration.

The image controller 150 aggregates the images corrected by the image geometry corrector 140 into a single mosaic image. Herein, the image processing apparatus may further include a bulletin board for minimizing the brightness value and the geometrical inconsistency generated between the adjacent images according to the required result quality.

While the invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. This is possible. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the equivalents of the claims, as well as the claims.

Claims (7)

The present invention relates to a high-speed image mosaicking apparatus for an image photographed on a UAV,
An image acquisition unit for acquiring, from the unmanned airplane, images photographed so as to be overlapped with respect to a target area in the unmanned airplane moving along a predetermined flight path;
A mosaic image plane determining unit for determining a reference image that is a mosaic image plane among the acquired images;
A transformation relation deriving unit for deriving a transformation relation between each of the acquired images and a mosaic image plane corresponding to the reference image;
An image geometry correcting unit for performing geometric correction on the acquired images by using a conversion relation derived between each of the obtained images and a mosaic image plane corresponding to the reference image; And
And an image collecting unit for collecting the images subjected to the geometric correction into a single mosaic image,
Wherein the mosaic image plane determining unit comprises:
The corresponding points are extracted between the captured images of the acquired images and the maximum number of corresponding points is used as a weight to calculate the maximum spanning tree (MST) to minimize the depth of the tree And determining the node as the reference image. The method according to claim 1,
Wherein the mosaic image plane determining unit comprises:
Determining the reference image set to the mosaic image plane using an external facial expression element acquired at the time of photographing of the images photographed on the unmanned airplane,
Wherein the external facial expression element includes three-dimensional position information and rotation angle information of an image sensor for photographing the images. 3. The method of claim 2,
Wherein the reference image is the most centrally located image in the image distribution of the acquired images analyzed based on the three-dimensional location information. delete The method according to claim 1,
Wherein the mosaic image plane determining unit comprises:
Extracting an overlapping portion from the adjacent images from the row and column information of each of the acquired images based on the flight path, and determining at least one corresponding point extraction region based on the extracted overlapping portion. High-speed image mosaic device. 6. The method of claim 5,
Wherein the corresponding point extraction is performed between images in which captured positions of the acquired images are mutually adjacent, only in an area in which the determined corresponding point extraction regions overlap with each other. The present invention relates to a method for high-speed image mosaicking of an image taken on an unmanned aerial vehicle,
Acquiring, from the unmanned airplane, images photographed so as to be overlapped with each other with respect to a target area in the unmanned airplane moving along a predetermined flight path;
Determining a reference image that is a mosaic image plane among the acquired images;
Deriving a mosaic image plane corresponding to the reference image and a conversion relation of each of the acquired images;
Performing a geometric correction on the acquired images using a conversion relation derived between each of the acquired images and a mosaic image plane corresponding to the reference image; And
And aggregating the images subjected to the geometric correction into a single mosaic image,
In the step of determining the reference image,
The corresponding points are extracted from the captured images of the acquired images and the maximum number of corresponding points is used as a weight to calculate the maximum spanning tree (MST), and the depth of the tree is minimized And determining a node as the reference image.

Images