KR102613590B1 - Method of determining the location of a drone using 3D terrain location information and a drone thereof - Google Patents

Abstract

The present invention is a method of determining the location of a drone using 3D feature location information, comprising the steps of receiving a 2D tile map related to an image taking route to acquire the 3D feature information, and capturing images of the image taking route at a regular interval. receiving and using a signal received from outside to calculate two-dimensional plane coordinates where the drone is located at the same time the image was received, classifying features of interest included in the received image, and classifying the classified Extracting sequence information of the feature of interest using image coordinates between features of interest, wherein the sequence information of the feature of interest extracted from the image and the sequence information of the feature included in a subgroup of the two-dimensional tile map are at a certain level. A step of first extracting a subgroup of the two-dimensional tile map that matches the above, calculating the direction angle of the drone using the calculated two-dimensional plane coordinates of the drone at time t and the coordinate value of time t + 1, and Secondary extracting a 2D tile map subgroup in which the difference between the calculated orientation angle and the orientation angle of the subgroup of the 2D tile map satisfies a certain level or less, and the extraction of features included in the secondary extracted 2D tile map subgroup. Comprising the step of calculating the location of the drone by calculating the plane coordinates and height value of the camera that captured the image using the plane coordinates and height value, wherein the two-dimensional tile map subgroup is the image capture route. By including orientation angle information and utilizing the 2D plane coordinates and height values of major features included in the 2D tile map, the precise location of the drone and 3D location information of surrounding features can be extracted in real time.

Description

Method of determining the location of a drone using 3D terrain location information and a drone using the same {Method of determining the location of a drone using 3D terrain location information and a drone thereof}

The present invention relates to a method for determining the position of a drone, and more specifically, to a method for determining the position of a drone using 3D feature location information that can extract the precise location of the drone and 3D location information of surrounding features in real time, and a method for determining the position of a drone using the same. It's about drones used.

Because updating a 3D map by rephotographing roads and terrain takes a lot of time and money, a technology is needed that can update the 3D map by using existing 3D map information but only detecting changes in objects of interest. At this time, in order to increase the accuracy of the updated 3D map, it is necessary to increase the location accuracy of the drone that photographs roads and terrain features. Conventionally, the location of a mobile device is determined using an expensive precise positioning module or using a specific location context identifier or a G sensor that detects a speed bump as in the following prior art.

Prior Art 1 (Korean Patent Publication KR 10-2012-0117896 A) is a patent on map handling for location-based services associated with localized environments, and refers to a specific localization associated with a specific location context identifier. Stores map information corresponding to the localized environment. Prior art 1 has a specific location context identifier to determine the location of the mobile device.

Prior Art 2 (Korean Patent Publication KR 10-2019-0061135 A) is a patent on a device and method for estimating the position of a driving vehicle, and includes a G sensor that detects speed bumps while the vehicle is running, and the data input from the G sensor. When a speed bump is detected from the detection information, map information around the vehicle is extracted from the map data unit based on the estimated vehicle location to determine the location of the speed bump, and the estimated vehicle location is matched with the location of the speed bump to determine the vehicle The position of is being corrected.

Therefore, the first problem that the present invention aims to solve is to extract the precise location of the drone and the 3D location information of surrounding features in real time by utilizing the 2D plane coordinates and height values of major features included in the 2D tile map. It provides a method for determining the location of a drone using 3D feature location information.

The second problem that the present invention aims to solve is to create a high-accuracy drone using existing measured 3D location information of terrain features, camera-based object recognition technology, and optical flow information of objects of interest, even without an expensive precision positioning module. It provides a drone that determines the location using 3D location information of geographical features that can calculate the location.

In addition, the object is to provide a computer-readable recording medium on which a program for executing the above-described method is recorded on a computer.

In order to achieve the first task, the present invention provides a method for determining the position of a drone using 3D feature location information, comprising: receiving a 2D tile map related to an image taking route to acquire the 3D feature information; Receiving images of the image capture route at regular intervals and calculating two-dimensional plane coordinates where the drone is located at the same time when the images are received, using a signal received from an external source; Classifying features of interest included in the received image and extracting sequence information of the features of interest using image coordinates between the classified features of interest; First extracting a subgroup of a two-dimensional tile map in which sequence information of a feature of interest extracted from the image matches sequence information of a feature included in a subgroup of the two-dimensional tile map to a certain level or more; The heading angle of the drone is calculated using the calculated two-dimensional plane coordinates of the drone at time t and the coordinate value of time t + 1, and the difference between the calculated heading angle and the heading angle of the subgroup of the two-dimensional tile map is Secondary extraction of 2D tile map subgroups that meet a certain level or less; And calculating the location of the drone by calculating the plane coordinates and height value of the camera that captured the image using the plane coordinates and height value of the feature included in the secondary extracted 2D tile map subgroup. And, the 2D tile map subgroup provides a method of determining the location of a drone using 3D feature location information including directional angle information of the image capture route.

According to an embodiment of the present invention, the two-dimensional tile map is divided into subgroups and stored based on the change range of the height value of the feature, and the subgroups may include orientation angle information on two-dimensional plane coordinates.

In addition, when calculating the camera plane coordinates and height value, the plane coordinates and height information of the top of the feature having a height value above a certain level included in the subgroup of the secondary extracted two-dimensional tile map are used to receive the The camera plane coordinates and height values of the captured image can be calculated.

According to another embodiment of the present invention, the camera plane coordinates and height value of the image can be stored as attribute information of the image.

In order to achieve the second task, the present invention includes a tile map receiver that receives a 2-dimensional tile map related to an image taking route to acquire the 3-dimensional feature information; an image receiving unit that receives images of the image taking route at regular intervals; a sequence extraction unit that classifies the feature of interest included in the received image and extracts sequence information of the feature of interest using image coordinates between the classified features of interest; A subgroup primary decision unit that first extracts a subgroup of the two-dimensional tile map where the sequence information of the feature of interest extracted from the image and the sequence information of the feature included in the subgroup of the two-dimensional tile map match a certain level or more. ; Using a signal received from outside, calculate the two-dimensional plane coordinates where the drone is located at the same time when the image was received, and use the coordinate values of time t and time t+1 of the calculated two-dimensional plane coordinates of the drone. A subgroup secondary decision unit that calculates the orientation angle of the drone and secondarily extracts a 2D tile map subgroup in which the difference between the calculated orientation angle and the orientation angle of the subgroup of the 2D tile map satisfies a certain level or less. ; and a position calculation unit that calculates the location of the drone by calculating the plane coordinates and height values of the camera that captured the image using the plane coordinates and height values of the features included in the secondary extracted 2D tile map subgroup. It provides a drone using 3D geographical feature location information.

According to an embodiment of the present invention, the two-dimensional tile map is divided into subgroups and stored based on the change range of the height value of the feature, and the subgroups may include orientation angle information on two-dimensional plane coordinates.

In addition, when calculating the plane coordinates and height value of the camera, the plane coordinates and height information of the top of the feature having a height value above a certain level included in the subgroup of the secondarily extracted two-dimensional tile map are used to calculate the plane coordinates and height value of the camera. The camera plane coordinates and height values of the received image can be calculated.

According to another embodiment of the present invention, the camera plane coordinates and height value of the image can be stored as attribute information of the image.

In order to solve the above-described other technical problems, the present invention provides a computer-readable recording medium that records a program for executing on a computer the method of determining the position of a drone using the above-described 3D feature location information.

According to the present invention, the precise location of the drone and 3D location information of surrounding features can be extracted in real time by utilizing the 2D plane coordinates and height values of major features included in the 2D tile map.

In addition, according to the present invention, even without an expensive precise positioning module, high-accuracy drone positioning can be calculated using existing measured 3D location information of terrain features, camera-based object recognition technology, and optical flow information of the object of interest. It may be possible.

Furthermore, according to the present invention, the management of major geographical features of rivers, forests, and cities can be updated using a crowdsourcing method, and information on major geographical features can be efficiently managed using drones with cameras and communication functions.

Figure 1 shows a 2D tile map according to an embodiment of the present invention.
Figure 2 is a diagram for explaining sequence information of features included in a 2D tile map.
Figure 3 is a diagram illustrating the concept of direction angle information included in a subgroup of a 2D tile map.
Figure 4 is a configuration diagram showing a drone and a tile map server that determines a location using 3D feature location information according to a preferred embodiment of the present invention.
Figure 5 is a diagram showing the relationship between camera coordinates and real-space 3D coordinates.
Figure 6 is a diagram for explaining a method of determining camera center coordinates using feature location information of a 2D tile map.
Figure 7 is a diagram showing the relationship between camera frames and image frames.
Figure 8 is a diagram showing the relationship between an image frame and a real-space frame for calculating the location of a new feature.
Figure 9 shows two very short images of the drone's direction of travel.
Figure 10 is a flowchart of a method for determining a drone location according to a preferred embodiment of the present invention.
Figure 11 is a flowchart of a method for updating geographical feature location information according to a preferred embodiment of the present invention.
Figure 12 is an example of designing a shooting route to reflect the contours of geographical features and prevent sudden changes in the height value of the drone.

Before explaining the specific contents of the present invention, for convenience of understanding, an outline of a solution to the problem that the present invention seeks to solve or the core of the technical idea is first presented.

According to an embodiment of the present invention, the tile map server receives the starting point and destination information from the drone, considers the image capturing path to be captured by the drone, converts the information on the terrain into a 2D tile map, and provides it to the drone.

The drone can extract the precise location of the drone and the 3D location information of surrounding features in real time by utilizing the 2D plane coordinates and height values of major features included in the 2D tile map. At this time, the desired feature information can be effectively extracted using the sequence information between features extracted from the captured image and the sequence information of features included in the 2D tile map.

Hereinafter, the present invention will be described in more detail with reference to preferred embodiments. However, these examples are for illustrating the present invention in more detail, and it will be apparent to those skilled in the art that the scope of the present invention is not limited thereto.

Figure 1 shows a 2D tile map according to an embodiment of the present invention.

The 2D tile map in Figure 1 shows the area of the Earth in the form of tiles. Level 1 can be selected to have an area of 6144 km * 6144 km. In this case, level 3 has a size of 1536 km * 1536 km, which is 1/4.

In areas with dense terrain, such as urban areas, it is desirable to select the size of 1.5 km * 1.5 km corresponding to level 13 or 725 m Ⅹ 725 m corresponding to level 14 as the basic 2D tile map information.

Height value, color, classification information, etc. can be stored as separate attribute information based on the plane coordinate value of the geographical feature (river, building, iron tower, etc.) classified as object of interest information in the 2D tile map information.

According to an embodiment of the present invention, the 2D tile map includes feature location information and attribute information and can be used to determine the real-time precise location of the drone.

The 2D tile map may include attribute information (feature classification information, color, shape, text information, etc.) of the feature based on the location information (plane coordinates and height value) of the feature. In addition, it is desirable to provide level information based on a certain area and classify the 2D tile map into subgroups based on the change range of the height value of the feature.

As another embodiment, the 2D tile map may be classified into subgroups by considering the change in height value of the feature and the direction angle information of the drone shooting route. By using the location information of the feature, it is possible to know the sequence information of the feature including the arrangement information of continuous features.

Figure 2 is a diagram for explaining sequence information of features included in a 2D tile map.

Referring to Figure 2, information on major geographical features is shown at level 13 of the 2D tile map. Each geographical feature consists of information about a circular building, a square building 1, a square building 2, and a steel tower in counterclockwise order centered on the river. Each geographical feature can be grouped and divided into one subgroup spatial information, and the height information of each geographical feature can be included in the subgroup spatial information. The information included in the 2D tile map is divided into the level information of the 2D tile map and subgroup spatial information formed along rivers or roads, and the 2D plane coordinates of each feature and the matching height, color, and classification for each subgroup spatial information. You can store attribute information such as code and text. Therefore, if at least one of the drone's location information, the height information of the feature, or the direction angle information of the drone's shooting route is known, the closest or closest subgroup is extracted from the subgroup of the 2D tile map, and the feature included in the subgroup is extracted. Location information of water can be extracted. Meanwhile, it is preferable that the subgroup includes height information of geographical features or direction angle information of the drone shooting route.

First, subgroups that exist within a certain distance within 50m of the drone's GPS location and WiFi location are extracted, and each subgroup's spatial information and a series of feature location information are extracted as sequence information and extracted from the current image information. Subgroup information matching the sequence information of a series of geographical features can be initially extracted.

For example, in Figure 2, a circular building is placed below the stream, and square building 1 and square building 2 are placed above the stream. Using a series of sequence information according to the location of each feature, the current location information of the drone can be quickly calculated.

The 2D tile map information used for position calculation can be managed by organizing it into a separate layer so that it can be extracted more quickly. Additionally, in order to quickly extract the features that appear in the image, the features used to calculate the location of the drone can be configured to include features with a certain height value. For example, if the terrain information is composed only of features over 10m, screen blocking by forests, etc. can be minimized as the drone moves.

Meanwhile, in order to classify and extract desired features in the image, an object recognition module can be used that can automatically acquire the image coordinates of the features by extracting the features through supervised/unsupervised learning using machine learning methods. If 3D location information matching the image coordinates of the acquired feature is included in the 2D tile map, it is possible to calculate the current location information of the drone's camera, that is, the center coordinates of the camera, using the location information of the feature. At this time, it is possible to calculate the camera plane position and height value of the current image by using the plane position and height value of the feature used in calculating the drone's position and the pinhole model of the camera.

Figure 3 is a diagram illustrating the concept of direction angle information included in a subgroup of a 2D tile map.

Referring to FIG. 3, directional angle information of the drone photography route or the direction of movement of the drone is shown. The direction of drone movement can be calculated from the directional angle information of the drone shooting route, and vice versa. The angle formed with the direction of movement based on a determined reference direction may be the direction angle.

In addition, the amount of rotation angle change between continuously captured images is calculated using the optical flow information of the verification feature in the captured image, that is, information about the distance and direction of relative movement in adjacent images for the same object. , the 3D location coordinates of the verification feature can be calculated.

If the calculated 3D location information of the verification feature satisfies a certain level of location accuracy, the existing 2D tile map is updated and stored as a 2D tile map with changed or new features added, and then sent to the tile map server. The tile map server uses the camera's center coordinates (X, Y, height value) and 2D tile map update information received through crowdsourcing from multiple drones to post-process the data to provide 3D location information of the feature. It is possible to update the 3D map by acquiring .

Figure 4 is a configuration diagram showing a drone and a tile map server that determines a location using 3D feature location information according to a preferred embodiment of the present invention.

Referring to FIG. 4, the drone and tile map server that determines the location using 3D feature location information according to this embodiment consists of a drone 10 and a tile map server 20.

The drone 10 includes a tile map receiver 100, an initial location information generator 105, a subgroup filtering unit 110, a subgroup primary decision unit 115, a subgroup secondary decision unit 120, and a location Calculator 125, position update unit 130, image receiver 135, sequence extractor 140, subgroup extractor 145, position calculator 150, rotation angle calculator 155, topography It consists of a water location calculation unit 160, a tile map information update unit 165, and a transmission unit 170.

The tile map server 20 consists of a shooting route determination unit 200, a tile map transmission unit 210, a 3D data post-processing unit 220, a tile map information reception unit 230, and a tile map update unit 240.

When the tile map server 20 according to this embodiment receives the origin and destination information from the drone 10, it provides a 2D tile map corresponding to the expected shooting route to the drone 10, and provides the 2D tile map corresponding to the expected shooting route to the drone 10. The precise location of the drone 10 is determined in real time using the tile map server 20, which performs data post-processing using image information and updated 2D tile map information, and the 2D tile map received from the tile map server 20. It consists of a drone that calculates changes in location information of existing features and extracts location information of new features.

The tile map receiver 100 receives a 2D tile map reflecting the shooting route information for acquiring geographical feature information from the tile map server 20, and transmits the received 2D tile map to the subgroup filtering unit 110.

The initial location information generator 105 generates the initial location information of the drone 10 using the GPS module included in the drone 10 or a received WiFi signal, or using the current location information of the drone calculated externally. You can. The initial location information generator 105 generates the current location information of the drone 10, which is the basis for extracting a subgroup of the 2D tile map corresponding to the current location of the drone 10.

The subgroup filtering unit 110 can extract a subgroup of the 2D tile map that exists within a certain distance on the shooting route by reflecting the initial location information received from the initial location information generation unit 105 of the drone 10. there is.

The subgroup filtering unit 110 may separate the 2D tile map information into subgroups and store them, reflecting the length of the river or road that exists in the shooting route. The subgroup contains attribute information of major features based on two-dimensional plane coordinates that reflect a specific map coordinate system. The attribute information may include information such as the type of feature, height value, color, and text. Additionally, each subgroup may include information on the direction of a river or road or the directional angle of a possible path. Since it can be assumed that major geographical features are mainly arranged based on a specific route, the directional angle information of rivers, roads, and possible routes can be stored as subgroup information. Additionally, sequence information between features may be stored by reflecting the location information between features constituting the subgroup.

Since the initial location information of the drone 10 generated by the initial location information generation unit 105 has a low location accuracy of several meters, the subgroup primary decision unit 115 uses sequence information between geographical features to determine the current drone location information. Subgroup candidates of the 2D tile map to be used for position calculation in (10) can be extracted.

Using the continuous sequence information of the relative location information of the feature of interest automatically classified by the sequence extraction unit 140, the subgroup primary decision unit 115 can determine a subgroup candidate for the 2D tile map. At this time, the sequence information of the feature classified in the current image and the sequence information of the feature stored in the received 2D tile map may be different, so when comparing the sequence information of the two features, if the sequence information matches at a certain rate, the corresponding 2D tile map A tile map can be selected as a subgroup candidate.

The sequence extraction unit 140 automatically classifies the feature of interest photographed by the image receiver 135 by applying an image object recognition module using supervised learning or unsupervised learning to automatically classify the feature of interest in the image. The two-dimensional coordinates of the feature of interest can be acquired.

The subgroup secondary decision unit 120 compares the orientation angle information of the location information of the drone 10 with the orientation angle information of the subgroup candidate in the 2D tile map received from the subgroup primary decision unit 115, and determines a certain Subgroups that satisfy the level can be extracted secondarily.

Even when the location accuracy of the drone's initial location information is very low, when the amount of change in 2D plane coordinates extracted at certain times is constant, the calculated direction angle can be considered similar to the direction angle of location information with high precision.

The position calculation unit 125 uses the 3D position information of the feature of the 2D tile map subgroup, which is secondarily extracted by reflecting the orientation angle information, to calculate the focus position of the camera, that is, the plane coordinate of the center coordinate of the camera and the height value ( X, Y, height values) can be calculated. The plane coordinates and height value of the camera correspond to the plane coordinates and height value of the drone.

The camera calibration information needed at this time requires information on the pixel size, focal length, and skew variable value of the camera image corresponding to the camera's internal information.

If the camera's internal calibration information exists, automatically classify the feature of interest using an object recognition module in the image taken along the shooting route, obtain the image coordinates of the feature, and determine the actual 3D position matching the image coordinates. Information can be extracted from the subgroup information of the 2D tile map, the relationship between the image frame and the real space frame can be calculated using the camera pinhole model, and the camera's center coordinates (X, Y, height value) can be calculated using the least squares method. . The camera pinhole model is a model in which an external image is projected as an image through a single pinhole. At this time, this pinhole corresponds to the center of the lens, and the distance from this point to where the image on the back is formed is the camera focal length.

At this time, since the image coordinates of the feature of interest are needed, it is easier to obtain the top image coordinates than the bottom image coordinates of the feature of interest. Therefore, the height value provided by the subgroup of the 2D tile map is the height value of the top. In addition, it is easier to extract image coordinates for features with a height value above a certain height, so the coordinates and attribute information of features used for precise drone positioning are classified into a separate 2D tile map layer for calculation of real-time mobile applications. Speed can be improved.

The location update unit 130 updates the location information of the drone using the location information of the drone received from the location calculation unit 125 and the location calculation unit 150.

The location update unit 130 may recalculate the location information of the drone by assigning different weights to the location information of the drone received from the location calculation unit 125 and the location calculation unit 150.

The transmitter 170 transmits the drone's location information updated by the location update unit 130 to the 3D data post-processing unit 220 of the tile map server 20.

Figure 5 is a diagram showing the relationship between image coordinates and real space 3D coordinates.

Figure 5 shows a method for calculating the plane position and height value of an object of interest in a camera pinhole model. The formula for calculating the camera’s location information is as follows.

D1'' is the actual distance to the bottom of the object of interest, and D1 is the actual distance to the top. f is the focal length of the camera, and Hc is the height value of the camera. D1' is the distance between the camera and the point where a line perpendicular to the camera movement direction meets when it is drawn from the object of interest perpendicular to the camera movement direction.

With respect to the object of interest based on the camera frame, Ψ is the rotation angle of the X-axis, and θ is the rotation angle of the Y-axis. Therefore, referring to FIG. 5, the following equation is derived.

x, y are the x, y coordinates on the image of the object of interest. Therefore, the image coordinate y has a proportional relationship with H-Hc.

Figure 6 is a diagram for explaining a method of determining camera center coordinates using feature location information of a 2D tile map.

Referring to FIG. 6, the method of determining the location of the drone using the distance information of the feature acquired from the image is as follows.

It is desirable that the 3D location information of the feature is information extracted from the received 2D tile map.

The image coordinates of the features of interest OBJ1 and OBJ2 are (P1x _t , P1y _t ) and (P2x _t , P2y _t ), and the plane coordinates and height values in real space are OBJ1(X1, Y1, H1) and OBJ2(X2, Y2). , H2), and the height value of the camera is Hc.

You can calculate the camera's 3D location information coordinates (X, Y, Hc) using Equation 2 and Equation 3 below.

α, β correspond to the elevation angle on the Y axis from the camera to the top of the features of interest OBJ1 and OBJ2, and can be calculated in the same way as calculating θ in Equation 1.

D1, D2 are the actual distances to the top of the feature of interest, and D1", D2" are the actual distance to the bottom of the feature of interest.

Equation 2 shows the positional relationship between OBJ2 and the camera.

Equation 3 shows the relationship between the camera and OBJ1.

Equation 2 and Equation 3 can be changed to the determinant B = Ax as shown below, and the real-time precise position (Xt, Yt, Hc) of the camera's focal position, that is, the center coordinate of the camera, can be obtained from Equation 4.

At this time, it is desirable to calculate the precise position of the camera/center coordinates of the camera using the least squares method using the location information of at least 4 features.

here, is matrix B, is matrix A, is the matrix x.

By converting Equation 4 to a determinant, Error can be defined as follows.

In order to minimize the above error, if repeated calculations are made using the least squares method until an error value equal to or smaller than a certain standard is obtained, the camera center coordinate matrix x can be extracted.

Meanwhile, the image receiver 135 receives images of the image capture route at regular intervals.

The sequence extractor 140 classifies the feature of interest included in the image received from the image receiver 135 and extracts sequence information of the feature of interest using image coordinates between the classified features of interest. Additionally, as an example, the sequence extractor 140 may call the subgroup extractor 145 after the precise position of the drone 10 or the center coordinates (X, Y, height value) of the camera are calculated. there is. In this case, it is possible to extract subgroups based on the more accurate location of the drone 10.

The subgroup extraction unit 145 extracts a subgroup of the 2D tile map by comparing the sequence information of the feature of interest extracted from the image with the sequence information of the feature included in the subgroup of the 2D tile map.

Additionally, the subgroup extraction unit 145 extracts a subgroup of the 2D tile map if the sequence information of the feature extracted from the image and the feature of the 2D tile map are identical to a certain level or more. As another example, the subgroup extraction unit 145 may be replaced with the subgroup primary determination unit 115.

The difference between the position calculation unit 150 and the position calculation unit 125 is that the direction angle information is not used in the position calculation unit 150.

In addition, the location calculation unit 150 calculates real-time location information of geographical features under the assumption that the drone 10 makes a pure translational movement along the shooting route, and stores and updates 2D tile map information.

Therefore, if the z-axis rotation amount between consecutive images captured by a drone at a certain period meets a certain level or less, the calculated separation distance between the camera's center coordinates (X, Y, height value) is used to determine the actual 3 of the feature of interest. Dimensional position information can be calculated. The z-axis rotation amount of the continuous image refers to the rotation amount of the axis perpendicular to the moving direction of the drone (for example, when the plane coordinate is xy coordinate, it means the change in the z-axis rotation amount).

Figure 7 is a diagram showing the relationship between camera frames and image frames.

Referring to FIG. 7, the relationship between camera coordinates and image coordinates can be expressed as Equation 6.

Here, f is the focal length of the camera, α = f/s, and s is the pixel size of the image. u, v are image coordinates, and x _c , y _c , z _c are camera coordinates.

In Figure 7, it is shown that the image coordinates u and v are multiplied by the pixel size s of the image to become u*s and v*s, and it can be seen that the camera coordinates and the image coordinates are separated by the focal length f.

Meanwhile, the rotation matrix configuration for each axis is as shown in Equation 7 below. Also, assuming that the change in rotation amount is very small, the values are cos(x) ≒ 1, sin(x) ≒ x, and sin(x)sin(y) ≒ 0, so the rotation matrix below can be changed as follows: there is. Additionally, in the case of translational movement without rotation (Yaw) in the Z-axis direction of the camera, the rotation matrix becomes simpler.

With respect to the coordinate axis of the camera coordinate system, Rz _c ,Ψ is the rotation amount by rotating the Z axis, Ry _c ,ф is the rotation amount by rotating the Y axis, and Rx _c ,ф is the rotation amount by rotating the X axis.

In Equation 7, c represents the cosine function and s represents the sine function.

The height Hc (z=h) of the camera can be calculated as shown in Equation 8 below.

In Equation 8, c represents the cosine function and s represents the sine function.

r ₃₁ represents the element value of the 3rd row and 1st column of the rotation matrix R, r ₃₂ represents the element value of the 3rd row and 2nd column of the rotation matrix R, and r ₃₃ represents the element value of the 3rd row and 1st column of the rotation matrix R.

Figure 8 is a diagram showing the relationship between an image frame and a real-space frame for calculating the location of a new feature.

Referring to Figure 8, a pinhole camera model is shown. The model can also be applied to the image-real space model. Here, since the real space coordinate system and the image coordinate system have different origins, they can be expressed as follows. R is the rotation matrix, and C is the separation distance matrix between the image coordinate system and the real coordinate system.

Here _,

The reason for applying the rotation matrix is that when taking an image, the rotational movement of the camera is reflected in Yaw (Z-axis), Pitch (Y-axis), and Roll (X-axis) to capture the image.

When a feature of interest exists in two consecutive images, the rotation angle calculation unit 155 uses the rotation angle change of the camera, the change in image coordinates of the feature of interest, and the movement amount of the center coordinate of the camera to use the SVD algorithm. Using , you can calculate 3D location information of the feature of interest.

Figure 9 shows two very short images of the drone's direction of travel.

Referring to FIG. 9, in the case of pure translation, the change in rotation angle in the drone's moving direction in two very short images is close to 0, and the same vanishing point (epipole) exists. If the drone moves while keeping its directional angle constant and images are captured in a very short cycle, the change in directional angle between two images taken continuously can be assumed to be 0. The white line represents the epipolar line.

As a result of the verification by the feature location calculation unit 160, if the 3D location information of the verified feature matching the image coordinates satisfies a certain level of accuracy or higher, the tile map information update unit 165 updates the 2D tile information in real time. .

Based on the 3D location information of the feature included in the subgroup of the 2D tile map, the currently captured image information is used for the verification feature for which the 3D location information is already known through the function of the rotation angle calculation unit 155. By calculating 3D location information, location information accuracy can be calculated based on the difference between the calculated location information and the previously stored location information.

Therefore, real-time 2D tile information with location information accuracy that satisfies a certain level is stored and updated, and the stored information is received from the tile map information receiver 230 of the tile map server 20.

The 3D data post-processing unit 220 of the tile map server 20 uses 2D tile map update information received from multiple drones using a crowdsourcing method to post-process the data to extract more precise location information of geographical features. do.

The tile map information updating unit 165 updates the 2D tile map stored in the tile map server 20 with the tile map information updated from the tile map information receiving unit 230.

Meanwhile, the tile map server 20 consists of a driving route decision unit 200, a tile map transmitter 210, a 3D data post-processing unit 220, a tile map information reception unit 230, and a tile map update unit 240. do.

The driving route decision unit 200 receives the starting point and destination information from the drone 10 and determines the image capturing route to be captured by the drone.

The tile map transmitter 210 transmits a 2D tile map including the determined filming route to the drone 10.

The 3D data post-processing unit 220 uses image information transmitted together with real-time 2D tile information acquired from multiple drones to calculate 3D location information with higher accuracy for the feature of interest.

With 3D location information collected from multiple drones through crowdsourcing, changes in the location information of geographic features or location information of new features can be processed using statistical methods.

Since different 3D location information is transmitted for the same feature, the least squares method can be used to extract the 3D location information of the feature that satisfies a certain level of location accuracy and update the information in the 2D tile map. .

The tile map information receiving unit 230 receives updated 2D tile map information from the tile map information updating unit 165.

The tile map update unit 240 updates the existing 2D tile map using the 2D tile map, location information, and attribute information received from the 3D data post-processing unit 220 and the tile map information receiving unit 230.

Figure 10 is a flowchart of a method for determining a drone location according to a preferred embodiment of the present invention.

Referring to FIG. 10, the drone location determination method according to this embodiment consists of steps processed in time series in the drone location determination device shown in FIG. 4. Therefore, even if the content is omitted below, the content described above regarding the drone positioning device shown in FIG. 4 also applies to the drone positioning method according to this embodiment.

In step 1000, the drone positioning device receives a 2D tile map related to the image taking route to acquire 3D terrain information. For the route on which geographical feature information is to be collected, 2D tile map information including the attributes and height value information of the geographical feature may be received from the tile map server 20.

In step 1005, the drone positioning device divides the two-dimensional tile map into subgroups and stores them based on the change range of the height value of the feature.

For images taken from a drone, the pixel resolution of the image depends on the height of the drone from the ground. Therefore, it is necessary to design the shooting route so that there is no sudden change in the drone's height value by reflecting the contour lines of the terrain features.

Figure 12 is an example of designing a shooting route to reflect the contours of geographical features and prevent sudden changes in the height value of the drone.

In Figure 12, the shooting area is divided into areas A, B, and C according to the distribution of the height values of the features. C terrain is designed differently from A and B terrains because the change in height value of the feature in the left and right directions is less than a certain standard value.

Therefore, for the filming area, areas with similar altitude values within a certain range are classified as shown in Figure 12, a filming route is designed in detail for the area, and for each filming route, the 2D direction angle and height value of the filming route are calculated. Save and provide.

In step 1010, the drone positioning device receives images of the image capture route at regular intervals and uses signals received from the outside to calculate two-dimensional coordinates where the drone is located at the same time when the images are received.

The drone positioning device can filter and extract 2D tile map subgroup information contained within a certain distance from the drone's initial location information. By extracting the subgroup required for the current location within the filtered 2D tile map, the current location of the drone is determined.

The drone's initial location information can be roughly calculated from signals from the GPS module or Wi-Fi. In urban areas, the location accuracy of drones can be improved by using both GPS information and Wi-Fi information. However, in the case of location accuracy using the GPS module or Wi-Fi signal, location errors of several meters may occur, so when collecting location information of geographical features, expensive GPS must be used to ensure that the device location accuracy is within 10 cm. It is desirable to secure and also calculate the drone's position through post-processing.

On the other hand, when utilizing 3D location information of the plane coordinates and height value of a feature obtained using equipment such as Lidar, the screen coordinates of the feature of interest are extracted from the drone's mono camera image and matched accordingly. It is possible to calculate the real-time precise position of the camera using a model between three-dimensional coordinates in real space. At this time, it is assumed that the position of the camera is the same as that of the drone.

In step 1020, the drone positioning device classifies the feature of interest included in the received image and extracts sequence information of the feature of interest using image coordinates between the classified features of interest. It is possible to automatically recognize and classify features of interest from images received through a camera on a drone, and extract sequence information based on location information between features of interest.

At this time, the recognition of features of interest included in the image is possible using an object recognition module learned through deep learning and machine learning.

In addition, when recognizing a geographical feature in an image, it is necessary to obtain the coordinates of the lowest image adjacent to the river, road, and installation plane of the geographical feature in order to calculate the distance information between the actual camera and the geographical feature. However, when taking images, there are many cases where it is impossible to take pictures of the lowest point of a geographical feature due to limitations in physical shooting locations and moving objects along the shooting route. Therefore, in the present invention, it is desirable to calculate camera location information using the top location information of the feature.

In step 1030, the drone positioning device first extracts a subgroup of the two-dimensional tile map where the sequence information of the feature of interest extracted from the image and the sequence information of the feature included in the subgroup of the two-dimensional tile map match at least a certain level. do.

The sequence information of features included in the subgroup of the filtered 2D tile map can be compared with the sequence information of the feature of interest extracted from the image, and subgroup candidates of the 2D tile map whose sequence information is the same at a certain rate or more can be extracted.

If the accuracy of the drone's initial location information is very low, a subgroup of the 2D tile map can be extracted by comparing the continuous sequence information between the location information of the features in the received 2D tile information and the sequence information of the features acquired from the image. When comparing sequence information, differences may occur between the features of the 2D tile map and the sequence information extracted from the current image, so if a certain ratio is satisfied, they can be extracted as subgroup candidates.

In step 1040, the drone positioning device calculates the heading angle of the drone using the calculated two-dimensional plane coordinates of the drone at time t and the coordinate values of time t+1, and uses the calculated heading angle and the two-dimensional tile map. A 2D tile map subgroup whose orientation angle difference between subgroups satisfies a certain level or less is secondarily extracted.

In other words, the direction angle of the drone is calculated using the amount of change in the drone's location information, the direction angle of the subgroup candidate in the 2D tile map is compared with the direction angle of the drone, and the subgroup candidate in the 2D tile map whose difference is below a certain level is selected. It can be re-extracted. Therefore, it is desirable that the 2D tile map subgroup includes orientation angle information of the corresponding subgroup. Since the drone's shooting route is set according to the direction of the river or road, the direction angle when the drone moves will also be set according to the direction of the river or road.

In order to extract the optimal subgroup from multiple 2D tile maps, it is currently possible to extract it using the drone's location information. However, since the location accuracy of mobile devices is at the level of several meters, the drone's movement direction information must be used to extract it. Subgroup candidates from the 2D tile map can be re-extracted.

In this case, the location information of the drone synchronized with the image reception period is extracted, the extracted location information (location information at time t and time t+1) is converted to plane coordinates to extract the direction angle, and the change in direction angle is If it is maintained within a certain level, the heading angle of the current drone can be compared with the heading angle information included in the 2D tile map subgroup, and a subgroup candidate whose difference satisfies a certain level can be re-extracted.

Here, the direction angle of the subgroup of the 2D tile map corresponds to the direction angle of the river, road, or movable space where the feature exists within the subgroup.

In step 1050, the drone positioning device calculates the plane coordinates and height values of the camera that captured the image using the plane coordinates and height values of the features included in the secondary extracted 2D tile map subgroup.

It is desirable to calculate the real-time center coordinates and height value of the shooting camera using the top location information of the feature included in the subgroup of the secondary extracted 2D tile map.

Step 1050 is a step of calculating the focus coordinates of the camera, that is, the center coordinates of the camera, by matching the image coordinates of the feature of interest recognized in the image with the 3D location information of the subgroup of the 2D tile map, using the camera pinhole model. . In this case, it is possible to calculate the camera center coordinates using the least squares method. In this case, the focal length, camera pixel size, and skew variable values are required as the camera's internal calibration variables.

In step 1060, the drone positioning device stores the center coordinate location information of the shooting camera as image attribute information.

This is the step of storing the precise center coordinates of the real-time camera as attribute information of the image saved from the drone. If the center coordinates of the camera are stored, images collected on the server through crowdsourcing can be easily grouped using the center coordinates of the camera, and based on the information in the image, precise 3D location information of geographical features can be statistically collected. By post-processing using this method, 3D location information of geographical features with higher accuracy can be extracted.

Figure 11 is a flowchart of a method for updating geographical feature location information according to a preferred embodiment of the present invention.

Referring to FIG. 11, the method for updating feature location information according to this embodiment consists of steps processed in time series in the feature location information updating device shown in FIG. 4. Therefore, even if the content is omitted below, the content described above regarding the feature location information updating device shown in FIG. 4 also applies to the feature location information updating method according to this embodiment.

In step 1100, the feature location information updating device receives a 2D tile map related to the image taking route for acquiring 3D feature information. For a route on which to collect feature information, it is desirable to receive 2D tile map information including feature attribute and height value information from the tile map server 20.

In step 1110, the feature location information update device receives images of the image capture route at a regular interval.

In step 1105, the drone positioning device divides the two-dimensional tile map into subgroups and stores them based on the change range of the height value of the feature.

For images taken from a drone, the pixel resolution of the image depends on the height of the drone from the ground. Therefore, it is necessary to design the shooting route so that there is no sudden change in the drone's height value by reflecting the contour lines of the terrain features.

In step 1120, the feature location information update device classifies the feature of interest included in the received image and extracts sequence information of the feature of interest using image coordinates between the classified features of interest.

In step 1130, the feature location information update device creates a subgroup of a two-dimensional tile map in which the sequence information of the feature of interest extracted from the image and the sequence information of the features included in the subgroup of the two-dimensional tile map match at least a certain level. Extract.

Before step 1130, the initial location information of the drone may be used to filter and extract 2D tile map subgroup information included within a certain distance from the initial location information. When the drone's precise location information (camera's center coordinates) has been calculated, it is desirable to filter 2D tile map subgroups included within a certain distance.

In step 1140, the feature location information update device determines the received plane coordinates and height values of the camera using the plane coordinates and height values of the features included in the subgroup of the extracted two-dimensional tile map.

In step 1150, the feature location information update device calculates the amount of change in the relative rotation angle of the continuously received images as optical flow information of the verification feature classified in the continuously received images.

If the z-axis rotation amount between consecutive images captured by a drone at a certain period meets a certain level or less, the rotation angle information of two consecutive images is calculated using the extracted center coordinates and optical flow information of the camera, and the consecutive images are You can calculate 3D location information of existing features.

When acquiring images of geographical features using a camera mounted on a drone, two consecutive images can be used as stereo images. Therefore, it is possible to acquire 3D location information of the feature of interest from two consecutive images.

At this time, it can be assumed that there is a very small change in the rotation angle of the camera's Z-axis (vertical axis in the plane) for a very short period of time (approximately 1/10 second or less), and also that the drone changes the shooting direction. When moving while maintaining , it can be assumed to be close to pure translational motion.

At this time, the instantaneous change in camera rotation angle can be calculated using the optical flow information of the feature of interest.

In order to calculate optical flow information, the rotation angle change matrix that occurs between successive images acquired at regular intervals and the separation distance information between images are required.

At this time, since the camera is assumed to be in pure translation, it can be changed to have the values cos(x) ≒ 1, sin(x) ≒ x, and sin(x)sin(y) ≒ 0, and two consecutive images At the moment of filming, the direction of the drone is assumed to be in pure translational motion with little change, and the rotation amount of the camera's Z axis is assumed to be close to 0. By simplifying the rotation matrix, real-time calculation speed can be improved. When taking an image, the rotational movement of the camera is reflected through Yaw (Z-axis), Pitch (Y-axis), and Roll (X-axis) to capture the image.

Assuming that the mobile device performs pure translational motion, the relative camera frame does not change direction with respect to movement, so the change in the Z-axis rotation angle can be assumed to be ψ = 0. Additionally, the rotation amount of the X and Y axes of the camera frame can be assumed to be close to 0.

Therefore, the optical flow follows the formula below, and the unknowns are reduced to ωy = θ, wx = ф. Here, the amount of movement of the focus position of the two images is (Tx, Ty, and Z is the height of the camera. The image coordinates of the object of interest in the image taken at time t, t-1 are (u', v'), ( u, v).

From the above equation, it is possible to obtain the change amount ωy=θ, wx=ф in the relative rotation angle of the continuously received images.

In step 1160, the feature location information update device calculates the 3D location information of the verification feature using the relative rotation angle change between successively received images and the camera plane coordinate movement amount.

In other words, the difference between the 3D location information of the feature included in the 2D tile information and the 3D location information of the same feature calculated from the rotation angle and camera center coordinate movement amount of successive images can be calculated.

Once the camera's external calibration information is known, 3D location information corresponding to the image coordinates of the feature of interest can be extracted. The camera's external calibration information consists of the difference in rotation angle between two consecutive images and the camera center coordinate moving distance.

Among these, the difference in rotation angle can be extracted using the least squares method using the light flux flow information of the feature of interest, and the camera center coordinate moving distance can be extracted using the 3D position coordinates of the feature included in the subgroup of the 2D tile map. By acquiring the center coordinates, the center coordinate movement distance of the camera can be extracted.

In step 1170, if the distance difference between the location information calculated for the verification feature and the 3D location information of the verification feature included in the subgroup of the extracted 2D tile map is below a certain level, the feature location information update device The 3D location information of the changed or new feature is updated in the 2D tile map.

In other words, the 3D location information of the verification feature included in the 2D tile map is calculated from the two consecutive images currently acquired, and if the accuracy of the location information is satisfied, the location information of the changed feature or new feature in the 2D tile map is calculated. Update .

Step 1170 includes verifying whether the accuracy of the 3D location information of the verification feature calculated in step 1160 satisfies a certain level.

It is desirable to calculate the 3D location information of the verification feature present in two consecutive images in which the feature of the received 2D tile map is captured and compare it with the location information present in the 2D tile map received from the server. Through the above comparison, if the location accuracy exceeds a certain level is satisfied, the information of the existing 2D tile map is updated and stored. Updating the information of a 2D tile map includes changing or deleting existing features and adding location information of new features.

According to an embodiment of the present invention, a crowdsourcing method can be used as a method for detecting changes in geographical features. The crowdsourcing method uses a drone's camera to photograph route information and analyzes the captured images to update real-time geographic feature information where changes are detected. By collecting 2D tile map information received from multiple drones, it is possible to precisely modify the 2D tile map information stored in the tile map server 20.

The term '~unit' used in this embodiment refers to software or hardware components such as FPGA (field-programmable gate array) or ASIC, and the '~unit' performs certain roles. However, '~part' is not limited to software or hardware. The '~ part' may be configured to reside in an addressable storage medium and may be configured to reproduce on one or more processors. Therefore, as an example, '~ part' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and 'parts' may be combined into a smaller number of components and 'parts' or may be further separated into additional components and 'parts'. Additionally, components and 'parts' may be implemented to regenerate one or more CPUs within a device or a secure multimedia card.

All of the above-described functions can be performed by a processor such as a microprocessor, controller, microcontroller, application specific integrated circuit (ASIC), etc. according to software or program code coded to perform the above functions. The design, development and implementation of the above code will be apparent to those skilled in the art based on the description of the present invention.

Although the present invention has been described above with reference to embodiments, those skilled in the art can implement the present invention by making various modifications and changes without departing from the technical spirit and scope of the present invention. You will understand. Therefore, it is not limited to the above-described embodiments, and the present invention includes all embodiments within the scope of the following claims.

Claims (9)

A method for determining the location of a drone using 3D feature location information,
Receiving a two-dimensional tile map related to an image taking route to acquire the three-dimensional feature location information;
Receiving images of the image capture route at regular intervals and calculating two-dimensional plane coordinates where the drone is located at the same time when the images are received, using a signal received from an external source;
Classifying features of interest included in the received image and extracting sequence information of the features of interest using image coordinates between the classified features of interest;
First extracting a subgroup of a two-dimensional tile map in which sequence information of a feature of interest extracted from the image matches sequence information of a feature included in a subgroup of the two-dimensional tile map to a certain level or more;
The heading angle of the drone is calculated using the calculated two-dimensional plane coordinates of the drone at time t and the coordinate value of time t + 1, and the difference between the calculated heading angle and the heading angle of the subgroup of the two-dimensional tile map is Secondary extraction of 2D tile map subgroups that meet a certain level or less; and
Comprising the step of calculating the location of the drone by calculating the plane coordinates and height value of the camera that captured the image using the plane coordinates and height value of the feature included in the secondary extracted 2D tile map subgroup; ,
A method of determining the location of a drone using 3D feature location information, wherein the 2D tile map subgroup includes directional angle information of the image capture route. According to claim 1,
The two-dimensional tile map is divided into subgroups and stored based on the change range of the height value of the feature, and the subgroups include orientation angle information in two-dimensional plane coordinates using three-dimensional feature location information. How to determine the location of a drone. According to claim 1,
When calculating the camera plane coordinates and height value, the plane coordinates and height information of the top of the feature having a height value above a certain level included in the subgroup of the secondary extracted two-dimensional tile map are used to obtain the received image. A method for determining the location of a drone using 3D feature location information, characterized by calculating camera plane coordinates and height values. According to claim 1,
A method of determining the location of a drone using 3D feature location information that stores the camera plane coordinates and height value of the image as attribute information of the image. In a drone using 3D feature location information,
a tile map receiver that receives a 2-dimensional tile map related to an image taking route to acquire the 3-dimensional feature location information;
an image receiving unit that receives images of the image taking route at regular intervals;
a sequence extraction unit that classifies the feature of interest included in the received image and extracts sequence information of the feature of interest using image coordinates between the classified features of interest;
A subgroup primary decision unit that first extracts a subgroup of the two-dimensional tile map where the sequence information of the feature of interest extracted from the image and the sequence information of the feature included in the subgroup of the two-dimensional tile map match a certain level or more. ;
Using a signal received from outside, calculate the two-dimensional plane coordinates where the drone is located at the same time when the image was received, and use the coordinate values of time t and time t+1 of the calculated two-dimensional plane coordinates of the drone. A subgroup secondary decision unit that calculates the orientation angle of the drone and secondarily extracts a 2D tile map subgroup in which the difference between the calculated orientation angle and the orientation angle of the subgroup of the 2D tile map satisfies a certain level or less. ; and
A position calculation unit that calculates the location of the drone by calculating the plane coordinates and height values of the camera that captured the image using the plane coordinates and height values of the features included in the secondary extracted 2D tile map subgroup. A drone that determines its location using 3D location information of geographical features. According to clause 5,
The two-dimensional tile map is divided into subgroups and stored based on the change range of the height value of the feature, and the subgroups use 3D feature location information, wherein the subgroups include orientation angle information on two-dimensional plane coordinates. A drone that determines its location. According to clause 5,
When calculating the plane coordinates and height value of the camera, the received plane coordinates and height information of the top of the feature having a height value above a certain level included in the subgroup of the secondary extracted two-dimensional tile map are used. A drone that determines its location using 3D feature location information, characterized by calculating the camera plane coordinates and height value of the image. According to clause 5,
A drone that determines its location using 3D feature location information that stores the camera plane coordinates and height value of the image as attribute information of the image. A computer-readable recording medium recording a program for executing the method of any one of claims 1 to 4 on a computer.