WO2024096691A1

WO2024096691A1 - Method and device for estimating gps coordinates of multiple target objects and tracking target objects on basis of camera image information about unmanned aerial vehicle

Info

Publication number: WO2024096691A1
Application number: PCT/KR2023/017562
Authority: WO
Inventors: 이규만; 이주혁
Original assignee: 경북대학교 산학협력단
Priority date: 2022-11-04
Filing date: 2023-11-03
Publication date: 2024-05-10

Abstract

Provided are a method and device for estimating the GPS coordinates of multiple target objects and tracking the target objects on the basis of camera image information from an unmanned aerial vehicle according to an embodiment. The device comprises: a detection unit that, through a pre-trained artificial intelligence deep learning model, detects an object corresponding to an assignment set in the unmanned aerial vehicle, and generates a bounding box including the detected target object according to the size of the target object; a setting unit that sets a reference pixel that is a target point represented as a pixel on a camera image plane; a calculation unit that calculates reference pixel coordinates of the target object in a two-dimensional pixel coordinate system; and a conversion unit that converts the reference pixel coordinates of the target object into three-dimensional coordinates by using at least one of altitude information about the unmanned aerial vehicle or depth information about the target object, and converts the coordinate system of the converted three-dimensional coordinates to acquire the GPS coordinates of the target object.

Description

Method and device for estimating GPS coordinates of multiple targets and tracking targets based on camera image information from unmanned aerial vehicles

The present disclosure relates to a method and device for estimating and tracking the GPS coordinates of a target. Specifically, the GPS coordinates of multiple targets are estimated based on camera image information of an unmanned aerial vehicle, and the pixels corresponding to the targets are located at the center of the image plane. It relates to a method and device for tracking a target by controlling a drone to locate it.

Unless otherwise indicated herein, the material described in this section is not prior art to the claims of this application, and is not admitted to be prior art by inclusion in this section.

In order to perform missions such as searching for missing people or calculating the global location of targets that are not captured by anti-drone radar, it is important to accurately determine the location coordinates of multiple targets. To this end, the location coordinates of the target may be obtained through a location sensor or terminal installed on the target, but in real situations such as searching and detecting missing persons, the location coordinates of the target must be obtained even in an uncooperative environment. An uncooperative environment refers to a case in which the location of a target cannot be determined through wireless communication, for example, if the target is impossible to communicate with each other or is not equipped with a system such as ADS-B (Automatic Dependent Surveillance - Broadcast). In such an uncooperative environment, it is generally difficult to estimate the location of the target.

Meanwhile, an unmanned aerial vehicle (UAV) refers to an aircraft that flies autonomously or semi-automatically according to a preset route or remotely controlled from the ground without a pilot directly boarding the aircraft. Unmanned aerial vehicles are also called drones.

These drones are increasingly using cameras to detect or track objects or people on the ground. During this filming process, drones can detect targets such as objects or people and perform tracking missions.

Accordingly, there is a need for an invention that can more simply and precisely fix a target, such as a specific point or a specific object, and track the captured target while shooting.

An apparatus and method for estimating GPS coordinates of a target based on camera image information of an unmanned aerial vehicle according to an embodiment calculates the GPS coordinates of multiple targets detected based on camera images mounted on an unmanned aerial vehicle, and identifies multiple targets on a three-dimensional map. Estimate the exact global location of the target.

In the embodiment, targets corresponding to given tasks, such as detection of missing persons and vehicles violating traffic laws, are automatically detected through a pre-trained artificial intelligence deep learning model. In the embodiment, a bounding box is created according to the target size including the detected target. In the embodiment, the reference pixel of the bounding box is set according to the detection location (ground, sky) of the detected target, the reference pixel coordinates of the target are calculated, and the calculated reference pixel coordinates are converted to a coordinate system including the unmanned aerial vehicle coordinate system. Afterwards, the target GPS coordinates of the unmanned aerial vehicle are finally acquired.

An apparatus and method for estimating GPS coordinates of a target based on camera image information of an unmanned aerial vehicle according to an embodiment can select a desired target reference pixel within a bounding box according to the user's purpose. In addition, the apparatus and method for estimating the GPS coordinates of a target based on camera image information of an unmanned aerial vehicle according to an embodiment calculates depth information through altimeter information mounted on the unmanned aerial vehicle when the detected target is a ground target; In the case of an airborne target, depth information on the target can be acquired by mounting a stereo camera or depth camera.

An apparatus and method for tracking a ground target according to an embodiment uses a camera mounted on a drone to control the ground target so that the pixel corresponding to the ground target is located at the center of the image plane. It provides a ground target tracking method using a drone equipped with a tracking camera.

A device for estimating the GPS coordinates of a target based on camera image information of an unmanned aerial vehicle according to an embodiment detects a target corresponding to the mission set for the unmanned aerial vehicle through a trained (pre-trained) artificial intelligence deep learning model ( a detection unit that detects and creates a bounding box containing the detected target; a setting unit that sets a reference pixel, which is a target point expressed as a pixel on a camera image plane; a calculation unit that calculates the reference pixel coordinates of the target in a two-dimensional pixel coordinate system; A conversion unit that converts the reference pixel coordinates of the target into 3D coordinates using at least one of the altitude information and depth information of the target, and converts the coordinate system of the converted 3D coordinates to obtain GPS coordinates of the target; Includes.

The ground target tracking method according to the embodiment is a method of tracking a ground target using a drone equipped with a camera, and the center pixel of the ground target captured using a camera mounted on the drone is selected on the image plane. Checking whether it has been done; If the center pixel of the ground target is checked as selected, calculating a yaw angle for a yaw rotation; Aligning in the horizontal direction through yawing rotation by the calculated yawing angle; calculating a distance for movement of the drone; Aligning in the vertical direction by moving forward or backward by the calculated distance; and tracking the ground target based on the aligned horizontal and vertical directions.

The device and method for estimating the GPS coordinates of a target based on the camera image information of an unmanned aerial vehicle as described above automatically determines the global location of the target through the camera image regardless of the number of targets or the detection location of the target, such as in the sky or on the ground. By displaying it on a map, it allows more accurate performance of various missions, such as searching for missing people or calculating the global location of targets eluded by radar for anti-drone purposes.

The device and method for estimating the GPS coordinates of a target based on camera image information of an unmanned aerial vehicle according to an embodiment are derived from the fields of computer vision, artificial intelligence, robotics, aerospace, and flight dynamics coordinate conversion, and are used for searching missing persons and invading unmanned aerial vehicles. It can be used for detection, etc.

An apparatus and method for estimating GPS coordinates of a target based on camera image information of an unmanned aerial vehicle according to an embodiment includes searching for missing persons and belongings using an unmanned aerial vehicle, estimating location information of people in a dangerous area, and landing and delivering an unmanned aerial vehicle. It can be used in the process of estimating the GPS coordinates of the destination, obtaining location information of the inspection target when inspecting facilities, and estimating the GPS coordinates of an enemy unmanned aerial vehicle located in the sky.

According to the ground target tracking method and device according to the embodiment, a camera is mounted on a drone, and the ground target is tracked by controlling the drone so that the pixel corresponding to the ground target is located at the center of the image plane. You can.

The effects of the present invention are not limited to the effects described above, and should be understood to include all effects that can be inferred from the configuration of the invention described in the detailed description or claims of the present invention.

Figure 1 is a diagram showing the data processing configuration of an apparatus 100 for estimating GPS coordinates of a target based on camera image information of an unmanned aerial vehicle according to an embodiment of the present invention.

Figure 2 is a diagram showing the data processing configuration of the conversion unit 170 according to an embodiment of the present invention.

Figure 3 is a diagram showing the positional relationship between an unmanned aerial vehicle and a ground target according to an embodiment of the present invention.

Figure 4 is a diagram showing the positional relationship between an unmanned aerial vehicle and an airborne target according to an embodiment of the present invention.

Figure 5 is a diagram showing the X _B Y _B plane of FRD (Front - Right - Down) according to an embodiment of the present invention.

Figure 6 is a diagram showing the data processing flow of a method for estimating GPS coordinates of a target based on camera image information of an unmanned aerial vehicle according to an embodiment of the present invention.

Figure 7 is a diagram showing the data processing process in step S400 according to an embodiment of the present invention.

Figure 8 is a block diagram for explaining a ground target tracking device using a drone equipped with a camera according to another embodiment of the present invention.

FIG. 9 is a diagram illustrating a state before the selected pixel is horizontally centered on the image plane according to another embodiment of the present invention.

FIG. 10 is a diagram illustrating a state after a selected pixel is horizontally centered on an image plane according to another embodiment of the present invention.

Figure 11 is a flowchart illustrating a ground target tracking method using a drone equipped with a camera according to another embodiment of the present invention.

12 to 14 are images for explaining ground target pixel tracking simulation results according to another embodiment of the present invention.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely provided to ensure that the disclosure of the present invention is complete and to provide common knowledge in the technical field to which the present invention pertains. It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

In describing embodiments of the present invention, if it is determined that a detailed description of a known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. The terms described below are terms defined in consideration of functions in embodiments of the present invention, and may vary depending on the intention or custom of the user or operator. Therefore, the definition should be made based on the contents throughout this specification.

An apparatus and method for estimating GPS coordinates of a target based on camera image information of an unmanned aerial vehicle according to an embodiment estimates position coordinates, which are GPS coordinates of multiple targets detected based on camera images mounted on an unmanned aerial vehicle, and provides three-dimensional Displays the exact global location of multiple targets on the map.

In the embodiment, targets corresponding to given tasks, such as detection of missing persons and vehicles violating traffic laws, are automatically detected through a pre-trained artificial intelligence deep learning model. In the embodiment, a bounding box is created according to the target size including the detected target. In the embodiment, the reference pixel of the bounding box is set according to the detection location (ground, sky) of the detected target, the reference pixel coordinates of the target are calculated, the calculated reference pixel coordinates are converted into 3D coordinates, and then the unmanned aerial vehicle By appropriately converting coordinates using GPS coordinates, GPS coordinates are finally obtained.

An apparatus and method for estimating GPS coordinates of multiple targets based on camera image information of an unmanned aerial vehicle according to an embodiment can select a desired target reference pixel within a bounding box according to the user's purpose. In addition, an apparatus and method for estimating GPS coordinates of multiple targets according to an embodiment calculates depth information through altimeter information mounted on an unmanned aerial vehicle when the detected target is a ground target, and calculates depth information using a stereo camera or a stereo camera when the detected target is an airborne target. Depth information of a target can be obtained through a depth camera. When converting coordinates, depth information is required. When converting coordinates from a 2D image to a 3D space (2D to 3D), 3D restoration using 2D information means creating one more dimensional information. In this case, scale ambiguity occurs and the exact scale is unknown, so it can be restored proportionally. If you know the value of one axis as a reference, which is precisely known depth information, you can no longer convert proportionally because you know the scale accurately, but you can change the exact coordinates from a two-dimensional image to a three-dimensional space (2D to 3D). Conversion can be done. Since the unmanned aerial vehicle knows altitude information using an altimeter, it uses the measured altitude value as depth information to know the scale of the ground target. In the case of an airborne target, depth information can be obtained through camera sensors that can measure depth information, such as a mounted stereo camera or depth camera.

Figure 1 is a diagram showing the data processing configuration of a target GPS coordinate estimation device 100 for an unmanned aerial vehicle according to an embodiment.

Referring to FIG. 1, the target GPS coordinate estimation device 100 of an unmanned aerial vehicle according to an embodiment will be comprised of a detection unit 110, a setting unit 130, a calculation unit 150, and a conversion unit 170. You can. The term 'part' used in this specification should be construed to include software, hardware, or a combination thereof, depending on the context in which the term is used. For example, software may be machine language, firmware, embedded code, and application software. As another example, hardware may be a circuit, processor, computer, integrated circuit, integrated circuit core, sensor, Micro-Electro-Mechanical System (MEMS), passive device, or a combination thereof.

The detection unit 110 detects a target corresponding to the mission set for the unmanned aerial vehicle through a pre-trained artificial intelligence deep learning model. Missions set for unmanned aerial vehicles may include searching for missing persons, detecting vehicles, and detecting enemy aircraft in the sky. Additionally, in the embodiment, the detection unit 110 may perform multi-targeting to detect at least one target.

In the embodiment, the detection unit 110 can detect both ground targets and airborne targets. A ground target is a target located on the ground, and an aerial target is a target located in the sky. In the embodiment, the detection unit 110 may detect ground targets and airspace targets through an artificial intelligence deep learning model trained by separating the ground target detection process and the airspace target detection process. In an embodiment, the artificial intelligence deep learning model includes, but is not limited to, at least one of a Deep Neural Network (DNN), a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), and a Bidirectional Recurrent Deep Neural Network (BRDNN). In an embodiment, an artificial intelligence deep learning model can classify various target detection processes according to the type of ground target and learn the classified target detection process. Types of targets may include, but are not limited to, people, articles, hats, bicycles, etc.

The detection unit 110 creates a bounding box containing the detected target according to the size of the target.

Afterwards, the setting unit 130 sets the reference pixel of the target included in the bounding box. In an embodiment, the reference pixel of the target may be a target point represented by a pixel on the camera image plane. That is, the setting unit 130 may set a reference pixel to obtain a target point within the bounding box.

In an embodiment, the setting unit 130 may set the reference pixel of the target as the center point of the bounding box.

Additionally, in the embodiment, the setting unit 130 may set the reference pixel of the target within the bounding box to a desired pixel according to the user's purpose. For example, if the user's goal is to shoot down the target and the target is an aircraft, the wing part of the aircraft may be set as the reference pixel of the target, and if the target is a building, the core part of the building may be set as the reference pixel. Additionally, if the user's purpose is a search and the target is a person, the reference pixel of the target may be set according to the person's posture. For example, if the person is standing, location information on the ground is needed, so the foot side of the detected person can be set as the reference pixel of the target. If the person is lying down, the person's torso can be set as the reference pixel for the target.

The calculation unit 150 calculates the reference pixel coordinates of the target in a two-dimensional pixel coordinate system. The calculation unit 150 calculates the reference pixel of the target set in the setting unit 130 as a coordinate value in a two-dimensional pixel coordinate system.

The conversion unit 170 converts the reference pixel coordinates of the target into three-dimensional coordinates using at least one of the altitude information and depth information of the unmanned aerial vehicle. Here, the 3D coordinates are 3D values based on the camera coordinate system. To convert this to the unmanned aerial vehicle body coordinate system, an extrinsic parameter is used. The external parameter is a variable that indicates where the camera is mounted and in what direction it is mounted relative to the center of the unmanned aerial vehicle body, and may include the camera tilt angle.

That is, the coordinate system of the converted 3D coordinates in the embodiment may be the unmanned aerial vehicle body coordinate system used in the unmanned aerial vehicle. Afterwards, the conversion unit 170 converts the three-dimensional coordinate system and obtains the target GPS coordinates of the unmanned aerial vehicle.

Hereinafter, the process of acquiring target GPS coordinates in the conversion unit 170 will be described in detail.

Figure 2 is a diagram showing the data processing configuration of the conversion unit 170 according to an embodiment.

Referring to FIG. 2, the conversion unit 170 according to the embodiment may be configured to include a dimension conversion unit 171, a coordinate system conversion unit 173, and a GPS coordinate acquisition unit 175.

In the embodiment, the dimension transformation unit 171 transforms the coordinate system of the reference pixel coordinates. The dimension conversion unit 171 may convert the reference pixel coordinates of the target from a two-dimensional pixel coordinate system to a three-dimensional unmanned aerial vehicle body coordinate system.

More specifically, in the case of a ground target, the dimension conversion unit 171 uses the altitude information of the unmanned aerial vehicle as depth information of the ground target and converts the reference pixel coordinates of the target calculated in a two-dimensional pixel coordinate system into three dimensions.

In the embodiment, the dimension conversion unit 171 calculates the three-dimensional position difference of the camera coordinate system with respect to the reference pixel coordinates of the target, which are two-dimensional coordinates, through the camera model of the unmanned aerial vehicle, and calculates the three-dimensional position difference between the camera and the unmanned aerial vehicle CG (Center of Gravity). Using the external parameter value, the target's reference pixel coordinates are converted into three-dimensional coordinates for the body coordinate system of the unmanned aerial vehicle.

Figure 3 is a diagram showing the positional relationship between an unmanned aerial vehicle and a ground target according to an embodiment.

Referring to FIG. 3, in the embodiment, the dimension conversion unit 171 sets the camera mounted on the unmanned aerial vehicle as a pinhole camera model and converts the two-dimensional reference pixel coordinates into three-dimensional coordinates of the camera through Equation 1. You can.

(K: camera's intrinsic matrix, P _u (u,v,1) ^T : 3-dimensional vector representing the 2-dimensional coordinates of the pixel coordinate system as a homogeneous coordinate system, P _c : converted to the camera coordinate system 3D coordinates)

Afterwards, the dimensional conversion unit 171 assumes that the origin of the camera coordinate system and the unmanned aerial vehicle body coordinate system are located at the same location, and converts the three-dimensional coordinate P _c converted to the camera coordinate system into a vector of the unmanned aerial vehicle body coordinate system using Equation 2. Convert to P _B.

(R _BC : 3D rotation transformation matrix from the camera coordinate system to the unmanned aerial vehicle body coordinate system, P _c : 3D coordinates converted to the camera coordinate system)

Since the camera mounted on the unmanned aerial vehicle is tilted in the range of 0 degrees to 90 degrees in the ground direction, the dimensional conversion unit 171 uses a three-dimensional rotation transformation matrix R _BC from the camera coordinate system to the unmanned aerial vehicle body coordinate system. It can be calculated through Equation 3.

(Ry: y-axis rotation transformation matrix, Rz: z-axis rotation transformation matrix Rx: x-axis rotation transformation matrix)

Afterwards, the dimension conversion unit 171 calculates the vector P _B of the unmanned aerial vehicle body coordinate system using equation 4 derived through equation 1 and equation 2.

(K: camera's internal parameter (intrinsic matrix), P _u (u,v,1) ^T : 3-dimensional vector representing the 2-dimensional coordinates of the pixel coordinate system as a homogeneous coordinate)

On the other hand, when the detected target is an airspace target, the dimension conversion unit 171 By collecting depth information of an aerial target through a stereo camera or depth camera mounted on an unmanned aerial vehicle, the GPS coordinates of the aerial target can be estimated.

For example, when depth information (z _c ) of an aerial target is given, the dimension conversion unit 171 calculates the 3D position information P _c of the aerial target expressed in a camera coordinate system using Equation 5, and using Equation 6 Calculate the vector P _B of the unmanned aerial vehicle body coordinate system.

(P _c : 3-dimensional coordinates converted to the camera coordinate system, K: internal parameters of the camera (intrinsic matrix), z _c : depth information of the target, I: identity matrix, P _u : 2-dimensional coordinates of the pixel coordinate system are converted to a homogeneous coordinate system ( 3D vector expressed in homogeneous coordinates)

(P _B : Vector of the unmanned aerial vehicle body coordinate system, R _BC : 3D rotation transformation matrix from the camera coordinate system to the unmanned aerial vehicle body coordinate system, K: Camera's internal parameter (intrinsic matrix), z _c : Target depth information, I: Unit matrix, P _u : 3-dimensional vector representing the 2-dimensional coordinates of the pixel coordinate system as a homogeneous coordinate system)

In the embodiment, when the dimensional conversion unit 171 converts the 2D reference pixel coordinates of the target into the 3D body coordinate system of the unmanned aerial vehicle, the coordinate system conversion unit 173 converts the target coordinates into the 3D body coordinate system of the unmanned aerial vehicle based on the target coordinates. The relative position difference between the unmanned aerial vehicle and the target is calculated, and the three-dimensional coordinates of the target are converted to the ENU (East-North-Up) coordinate system.

FIG. 4 is a diagram showing the positional relationship between an unmanned aerial vehicle and an airborne target according to an embodiment, and FIG. 5 is a diagram showing the X _B Y _B plane of FRD (Front-Right-Down) according to an embodiment. Referring to Figures 4 and 5, in the embodiment, the coordinate system conversion unit 173 converts the reference pixel coordinates of the target into the unmanned aerial vehicle body coordinate system and converts the three-dimensional coordinates into P _t = (x _t , y _t , z _t ) ^T Assuming that, the angle Ψ formed between the target and the unmanned aerial vehicle on the X _B Y _B plane of the unmanned aerial vehicle body coordinate system is calculated through Equation 7.

At this time, the coordinate system conversion unit 173 assumes that the ground target is located on the ground at the same altitude as the sea level altitude of the ground, which is the reference for the altitude of the unmanned aerial vehicle.

In an embodiment, the reference pixel of the ground target may be set to the center point of the automatically detected bounding box, unless specifically selected by the user. Accordingly, the coordinate system conversion unit 173 calculates the distance d _t to the ground target on the X _B Y _B plane of the unmanned aerial vehicle body coordinate system using altimeter information mounted on the unmanned aerial vehicle through Equation 8.

(h: altitude of the unmanned aerial vehicle, d _t : distance from the foot of the perpendicular line where the origin of the unmanned aerial vehicle's body coordinate system is placed on the ground to the target)

Meanwhile, unless the user specifically selects the reference pixel of the target in the sky, the center point of the bounding box is set as the reference pixel of the target in the sky. Accordingly, the coordinate system conversion unit 173 calculates the distance d _t from the sky target on the X _B Y _B plane of the unmanned aerial vehicle body coordinate system through Equation 9.

(x _t : x coordinate, y _t : y coordinate)

In addition, given the altimeter information (h) of the unmanned aerial vehicle, the coordinate system conversion unit 173 calculates the altitude (h _t ) of the target in the sky using Equation 10.

(h: altitude of the unmanned aerial vehicle, z _t : z coordinate of 3D coordinates converted to unmanned aerial vehicle body coordinates)

Afterwards, the coordinate system conversion unit 173 calculates both the ground target and the airborne target, the angle Ψ formed between the target and the unmanned aerial vehicle on the X _B Y _B plane of the calculated unmanned aerial vehicle body coordinate system _, _and the Using the distance d _t to the target, the body coordinate system (FRD) of the unmanned aerial vehicle is converted to an inertial coordinate system (ENU) through Equation 11 and Equation 12.

In the embodiment, the coordinate system conversion unit 173 calculates the target x coordinate (x _ENU ) in the ENU coordinate system using Equation 11.

(d _t : Distance to target on the X _B _Y _B plane of the unmanned _aerial vehicle body coordinate system, Ψ: Angle formed between the target and the unmanned aerial vehicle on the Heading angle of unmanned aerial vehicle (0≤α <360))

The coordinate system conversion unit 173 calculates the target y coordinate (y _ENU ) in the ENU coordinate system using Equation 12.

₍ d _t : Distance _to the target _in _the sky on the Heading angle of the drone in the direction (0≤α <360))

In the embodiment, the GPS coordinate acquisition unit 175 converts target coordinates expressed in the ENU coordinate system into GPS coordinates. GPS coordinates are coordinates that include latitude, longitude, and altitude. For example, the GPS coordinate acquisition unit 175 first converts the coordinates of the target expressed in the ENU coordinate system to the ECEF (Earth-Centered Earth-Fixed) coordinate system, and then converts the ECEF coordinate system to the LLA coordinate system (GPS coordinates) to obtain the target GPS coordinates. It can be obtained.

In an embodiment, the GPS coordinate acquisition unit 175 converts the target coordinates from the ENU coordinate system to the ECEF coordinate system using the current GPS coordinates of the unmanned aerial vehicle, and then converts them back to the LLA (Latitude-Longitude-Altitude) coordinate system, Finally, obtain the aircraft's target GPS coordinates. Specifically, the GPS coordinate acquisition unit 175 converts the target coordinates of the ENU (East-North-Up) coordinate system calculated by the coordinate system conversion unit 173 into the ECEF coordinate system using Equations 13 to 18 .

In the embodiment, the GPS coordinate acquisition unit 175 first calculates t using Equation 13 to convert from the ENU coordinate system to the ECEF coordinate system.

(z _ENU : z-coordinate of the target expressed in the ENU coordinate system, y _ENU : y-coordinate of the target expressed in the ENU coordinate system, Φ: latitude of the unmanned aerial vehicle)

t calculated in this way is a temporary variable used in Equations 14 and 15 for conversion from the ENU coordinate system to the ECEF coordinate system.

Afterwards, the GPS coordinate acquisition unit 175 calculates d _x , which is the relative position difference on the x-axis between the unmanned aerial vehicle and the target in the ECEF coordinate system using Equation 14.

(λ: Longitude of unmanned aerial vehicle)

Afterwards, the GPS coordinate acquisition unit 175 calculates d _y , which is the relative position difference on the y axis between the unmanned aerial vehicle and the target in the ECEF coordinate system using Equation 15.

(λ: Longitude of unmanned aerial vehicle)

Afterwards, the GPS coordinate acquisition unit 175 calculates x ₀ , which is the x-axis position of the unmanned aerial vehicle expressed in the ECEF coordinate system, using Equation 16.

(Φ: latitude of the unmanned aerial vehicle, N(Φ): radius of curvature of the Earth's ellipsoid according to latitude (Φ), λ: longitude of the unmanned aerial vehicle)

Afterwards, the GPS coordinate acquisition unit 175 calculates y ₀ , which is the y-axis position of the unmanned aerial vehicle expressed in the ECEF coordinate system, using Equation 17.

In the embodiment, the radius of curvature N(Φ) of the Earth's ellipsoid according to latitude (Φ) may be calculated by the position coordinate acquisition unit 175 through Equation 18.

(a: long axis of the Earth ellipsoid (WGS84), b: short axis of the Earth ellipsoid (WGS84), Φ: latitude of the unmanned aerial vehicle)

Afterwards, the GPS coordinate acquisition unit 175 _calculates

(x ₀ : position of the unmanned aerial vehicle in the ECEF coordinate system, d _x : relative position difference between the unmanned aerial vehicle and the target in the ECEF coordinate system)

The GPS coordinate acquisition unit 175 calculates y _ECEF, which is the y coordinate of the ECEF coordinate system, using Equation 20 through the unmanned aerial vehicle location information expressed in the ECEF coordinate system and the relative position difference between the unmanned aerial vehicle and the target.

(y ₀ : position of the unmanned aerial vehicle in the ECEF coordinate system, d _y : relative position difference between the unmanned aerial vehicle and the target in the ECEF coordinate system)

That is, in the embodiment, the GPS coordinate acquisition unit 175 can convert the target's location into the ECEF coordinate system using Equation 19 and Equation 20.

Afterwards, the GPS coordinate acquisition unit 175 converts the coordinates of the target shown in the ECEF coordinate system into GPS coordinates (ECEF to LLA) and finally obtains the GPS coordinates of the target of the unmanned aerial vehicle.

In the embodiment, the GPS coordinate acquisition unit 175 converts the coordinates of the target converted into the ECEF coordinate system obtained using Equation 19 and Equation 20 into GPS coordinates using Equation 21 to Equation 26, and converts the converted Coordinates are estimated as target GPS coordinates.

In the embodiment, the GPS coordinate acquisition unit 175 calculates the size r of the target position vector in the ECEF coordinate system using Equation 21.

Afterwards, the GPS coordinate acquisition unit 175 calculates the linear eccentricity E using Equation 22.

(a: Long axis of the Earth's ellipsoid (WGS84), b: Short axis of the Earth's ellipsoid (WGS84))

Afterwards, the GPS coordinate acquisition unit 175 calculates u using Equation 23.

Afterwards, the GPS coordinate acquisition unit 175 calculates β ₀ using Equation 24.

(u: minor axis of the confocal ellipsoid, β: angle from the center of the Earth to the reference ellipsoid, β ₀ : angle from the center of the Earth to the confocal ellipsoid)

Afterwards, the GPS coordinate acquisition unit 175 calculates Δβ using Equation 25.

Afterwards, the GPS coordinate acquisition unit 175 calculates β using Equation 26.

Below, the method for estimating the target GPS coordinates of an unmanned aerial vehicle will be explained in turn. Since the operation (function) of the target GPS coordinate estimation method of the unmanned aerial vehicle according to the embodiment is essentially the same as the function of the target GPS coordinate estimation system of the unmanned aerial vehicle, descriptions overlapping with FIGS. 1 to 5 will be omitted.

Figure 6 is a diagram showing the data processing flow of a method for estimating GPS coordinates of a target for an unmanned aerial vehicle according to an embodiment.

Referring to FIG. 6, in step S100, a target corresponding to the mission set for the unmanned aerial vehicle is detected through an artificial intelligence deep learning model trained (pre-trained) in the detection unit, and bounding including the detected target is performed. A bounding box is created according to the size of the target. In step S200, the setting unit sets a reference pixel, which is a target point expressed as a pixel on the camera image plane. In step S300, the calculation unit calculates the reference pixel coordinates of the target in a two-dimensional pixel coordinate system. In step S400, the conversion unit converts the reference pixel coordinates of the target into 3D coordinates using at least one of the target's altitude information and the target's depth information, and converts the coordinate system of the converted 3D coordinates. Afterwards, in step S500, the GPS coordinates of the target are obtained from the conversion unit.

Figure 7 is a diagram showing the data processing process in step S400 according to an embodiment.

Referring to FIG. 7, in step S410, the dimension conversion unit converts the coordinate system of the reference pixel coordinates. More specifically, in step S410, the reference pixel coordinates of the target are converted from a 2D pixel coordinate system to a 3D camera coordinate system and then into a 3D body coordinate system (FRD coordinate system). When converting the 2D pixel coordinate system to the 3D camera coordinate system in step S410, a pinhole camera model can be used, and at this time, altitude information of the unmanned aerial vehicle for ground targets and target depth information for airborne targets can be used.

Additionally, when converting the 3D camera coordinate system to the 3D body coordinate system, external parameters between the camera and the unmanned aerial vehicle CG can be used.

In step S430, the coordinate system conversion unit 173 calculates the relative position difference between the unmanned aerial vehicle and the target based on the target coordinates converted to the unmanned aerial vehicle body coordinate system, and converts the target's coordinates to the ENU (East-North-Up) coordinate system. . In the embodiment, in step S430, the coordinates of the target are converted from the unmanned aerial vehicle body coordinate system (FRD coordinate system) to the inertial coordinate system, which is the ENU coordinate system.

In step S450, the GPS coordinate acquisition unit 175 converts the target coordinates expressed in the ENU coordinate system into GPS coordinates. Specifically, in step S450, the ENU coordinate system is converted to the ECEF coordinate system and the LLA coordinate system (target GPS coordinates). In the embodiment, when converting the ENU coordinate system to the ECEF coordinate system, the GPS coordinate value of the unmanned aerial vehicle is used.

The device and method for estimating the target GPS coordinates of an unmanned aerial vehicle as described above automatically maps and displays the global location of the target through camera images regardless of the number of targets or the detection location of the target, such as in the sky or on the ground, thereby rescuing missing people or anti-virus software. It allows various missions to be performed more accurately, such as calculating the global location of targets that cannot be detected by drone-level radar.

The device and method for estimating GPS coordinates of a target for an unmanned aerial vehicle according to an embodiment include searching for missing people and belongings using an unmanned aerial vehicle, estimating location information of people in a dangerous area, estimating GPS coordinates of a destination for landing and delivery of an unmanned aerial vehicle, and It can be used in the process of obtaining location information of the inspection target during inspection and estimating the GPS coordinates of enemy drones in the sky.

Below, a ground target tracking device and method using a drone equipped with a camera according to another embodiment of the present invention will be described.

First, in order to control a drone equipped with a camera so that the pixel on the camera image plane is at the center of the image plane, the two-dimensional coordinates of the ground target pixel expressed in the pixel coordinate system are expressed based on the drone body coordinate system. It must be converted to dimensional coordinates.

For this transformation, a total of three coordinate systems are needed: a pixel coordinate system, a camera coordinate system, and a drone body coordinate system. At this time, the drone body coordinate system

uses the FRD (Front-Right-Down) coordinate system.

The origin of the camera coordinate system is the drone body coordinate system.

It exists on a plane and is in the camera coordinate system.

The axis is in the drone body coordinate system.

parallel to the axis Additionally, the camera coordinate system

The axis is in the drone body coordinate system for tracking ground targets.

Angle in the direction of the ground relative to the axis

(

) is tilted.

Hereinafter, coordinate system transformation will be described.

The conversion from the 2D pixel coordinate system to the 3D camera coordinate system utilizes camera intrinsic parameters. Assuming that the camera mounted on the drone is a pinhole camera model, the two-dimensional pixel coordinates can be converted as shown in Equation 27.

Here K is the intrinsic matrix of the camera,

is a three-dimensional vector representing the two-dimensional coordinates of the pixel coordinate system as a homogeneous coordinate system, and

is a three-dimensional coordinate converted to the camera coordinate system.

Next, for convenience of calculation, assuming that the origin of the camera coordinate system and the drone body coordinate system are located at the same location, the extrinsic parameter between the two coordinate systems is used to convert to the camera coordinate system as shown in Equation 28. 3D coordinates

the drone body coordinate system

It can be converted to .

here,

is a three-dimensional rotation transformation matrix from the camera coordinate system to the drone body coordinate system, and can be calculated as in Equation 29 below.

Finally, the following equation 30 is established by equation 27 and equation 28.

Here K is the intrinsic matrix of the camera,

is a 3-dimensional vector representing the 2-dimensional coordinates of the pixel coordinate system as a homogeneous coordinate system,

is a three-dimensional rotation transformation matrix from the camera coordinate system to the drone body coordinate system.

Figure 8 is a block diagram for explaining a ground target tracking device using a drone equipped with a camera according to an embodiment of the present invention.

Referring to FIG. 8, the ground target tracking device using a drone equipped with a camera according to an embodiment of the present invention includes a check unit 210, a yawing angle calculation unit 220, a horizontal alignment unit 230, and a distance calculation unit. It includes a unit 240 and a vertical alignment unit 250. In this embodiment, it is explained that the ground target tracking device is composed of a check unit 210, a yawing angle calculation unit 220, a horizontal alignment unit 230, a distance calculation unit 240, and a vertical alignment unit 250. However, this was only logically divided for convenience of explanation and not hardware-wise. In this embodiment, the camera may be a fixed monocular camera, a gimbal camera, etc.

The check unit 210 checks whether the center pixel of the ground target is located on the image plane.

As an example, a case where the center pixel is checked to be located on the image plane may be a case where the center pixel of a ground target displayed on the image plane is selected according to a user's manipulation.

As another example, the case where the center pixel is checked to be located on the image plane may be a case where the center pixel of the ground target is located on the image plane through an intra-image tracking algorithm. Specifically, if the user specifies the ground target they want to track in the video as ROI (Region Of Interest) only once, the video between frames is analyzed based on existing target tracking algorithms (MOSSE, CSRT, etc.) in the video to determine the ground target. Targets can be continuously tracked within the video. When a drone equipped with a camera is stationary and the object designated by ROI moves, the above-described tracking algorithm tracks the ground target, which is the object designated by ROI, within the video. When this tracking algorithm is operating, the center pixel of a moving ground target can be automatically derived, thereby maintaining continuity of tracking using the drone. In other words, move the drone so that the ground target is at the center of the image to secure and maintain the angle of view.

As another example, if the center pixel is checked to be located on the image plane, the ground target is recognized every frame through artificial intelligence and the center pixel of the ground target located on the image plane is automatically selected. It can be.

When the yawing angle calculation unit 220 checks that the center pixel of the target is located on the image plane, it calculates the yawing angle for yawing rotation.

Specifically, the coordinates obtained by converting the pixel coordinates located on the image plane into the drone body coordinate system are

If defined, the yawing angle calculation unit 120 is the angle at which the yawing rotation should be performed.

Can be calculated as in Equation 31 below.

here

is the x-coordinate value converted from the pixel coordinates located on the image plane to the drone body coordinate system,

is the y coordinate value converted from the pixel coordinates located on the image plane to the drone body coordinate system.

The horizontal alignment unit 230 aligns in the horizontal direction through yawing rotation by the yawing angle calculated by the yawing angle calculation unit 220.

Figure 9 is a diagram to explain the state before the selected pixel is centered horizontally on the image plane.

When the pixel selected in FIG. 9 is located at the center of the image plane horizontal direction through yawing, the positional relationship shown in FIG. 10 occurs when the drone is in a hover state.

Figure 10 is a diagram for explaining the state after the selected pixel is centered horizontally on the image plane.

Referring again to FIG. 8, the distance calculation unit 240 calculates the distance for the drone to move.

Specifically, coordinates converted from pixel coordinates to the drone body coordinate system.

and the height of the camera origin away from the ground

Using , the distance calculation unit 240 calculates the distance from the altitude reference point H of the drone to the ground target T.

can be calculated as in Equation 32.

here,

In addition, the distance calculation unit 240 calculates the distance between the altitude reference point H of the drone and the point O where the camera optical axis meets the ground.

Calculate as shown in Equation 33.

here,

is the height of the camera origin away from the ground,

is the drone body coordinate system.

It is the tilt angle toward the ground relative to the axis.

Next, the distance calculation unit 240 calculates the distance d that the drone must move to align the vertical center of the selected pixel, as shown in Equation 34 below, based on Equation 32 and Equation 33.

here,

,

is the distance between the drone's altitude reference point H and the point 0 where the camera optical axis meets the ground,

is the height of the camera origin away from the ground,

is the y coordinate value converted from the pixel coordinates located on the image plane to the drone body coordinate system,

is the z coordinate value converted from the pixel coordinates located on the image plane to the drone body coordinate system,

is the drone body coordinate system.

It is the tilt angle toward the ground relative to the axis.

The vertical alignment unit 250 aligns in the vertical direction by moving forward or backward by the distance calculated by the distance calculation unit 240.

As described above, according to the present invention, when the center pixel of a ground target is selected on the image plane, it rotates by yawing by the calculated yawing angle to adjust the horizontal direction, and moves forward or backward by the calculated distance to adjust the vertical direction to the ground target. The ground target is tracked by placing the pixel selected as the target at the center of the image plane.

Referring to FIG. 11, it is checked whether the center pixel of the target has been selected (step S610). Checking whether the center pixel is selected may be performed by the check unit 210 shown in FIG. 1. The above-mentioned center pixel may be selected according to the user's manipulation, or may be selected through an object detection program programmed for object selection.

If it is checked that the center pixel is selected in step S610, the yawing angle is calculated for yawing rotation (step S620). The above-described yawing angle calculation may be performed by the yawing angle calculation unit 220 shown in FIG. 8.

Align the horizontal direction by yawing and rotating by the yawing angle calculated in step S620 (step S630). The horizontal alignment described above can be performed by the horizontal alignment unit 230 shown in FIG. 8. In this embodiment, step S620 and step S630 are described as being separated, but step S620 and step S630 may be performed simultaneously.

Calculate the distance for the drone to move (step S640). The above distance calculation may be performed by the distance calculation unit 240 shown in FIG. 8.

Align the vertical direction by moving forward or backward by the distance calculated in step S640 (step S650). The vertical alignment described above can be performed by the vertical alignment unit 250 shown in FIG. 8. In this embodiment, step S640 and step S650 are described as being separated, but step S640 and step S650 may be performed simultaneously.

Then, the experimental verification results will be described below.

The ground target pixel tracking technique according to the present invention was verified in a simulation environment. camera

The altitude of the drone mounted at an angle of

It is deployed to perform ground target tracking.

12 to 14 are images for explaining ground target pixel tracking simulation results. In particular, FIG. 12 shows selection of a ground target to be tracked, FIG. 13 shows after target tracking, and FIG. 14 is a graph showing the error between the target position and the camera center point on the world coordinate system after target tracking.

If the center pixel of the target marker placed on the ground is selected on the image plane as shown in Figure 12, the angle calculated by Equation 31

Adjust the horizontal direction by yawing and rotating, and the distance calculated by Equation 34

Move forward or backward as much as possible to track the ground target in the vertical direction. That is, the pixel selected by the tracking technique according to the present invention is placed at the center of the image plane as shown in FIG. 13.

As a result of performing the same target tracking mission five times under the above-mentioned conditions, as shown in Figure 14, the average pixel error (e.g., the pixel distance between the center point of the final image plane and the target marker center pixel point) and the real distance of 0.62 m Target tracking is possible with error.

As described above, according to the present invention, a camera is mounted on a drone, and the ground target can be tracked by controlling the drone so that the pixel corresponding to the ground target is located at the center of the image plane. .

In addition, because a camera is used, it has the scalability to be mounted on small drones, and the amount of computation can be reduced through a relatively simple algorithm.

The disclosed content is merely an example and can be modified and implemented in various ways by those skilled in the art without departing from the gist of the claims, so the scope of protection of the disclosed content is limited to the above-mentioned specific scope. It is not limited to the examples.

Claims

A detection unit that detects a target corresponding to the mission set for the unmanned aerial vehicle through a pre-trained artificial intelligence deep learning model and creates a bounding box containing the detected target;

a setting unit that sets a reference pixel, which is a target point expressed as a pixel on a camera image plane;

a calculation unit calculating reference pixel coordinates of the target in a two-dimensional pixel coordinate system;

Using at least one of the altitude information of the unmanned aerial vehicle and the depth information of the target, the reference pixel coordinates of the target are converted into 3D coordinates, and the coordinate system of the converted 3D coordinates is converted to obtain the GPS coordinates of the target. A conversion unit that does; A device that estimates the GPS coordinates of a target based on camera image information of an unmanned aerial vehicle.
According to paragraph 1,

The target is

Includes ground and air targets,

The conversion unit; Is

A device for estimating GPS coordinates of a target based on camera image information of an unmanned aerial vehicle, characterized in that it converts the reference pixel coordinates of the target into 3D coordinates by using the altitude information of the unmanned aerial vehicle as depth information of the ground target. .
According to claim 1, wherein the conversion unit; Is

GPS of the target based on the camera image information of the unmanned aerial vehicle, characterized in that the depth information of the target in the air is acquired through a stereo camera or depth camera that measures depth, and the reference pixel coordinates of the target are converted into 3D coordinates. A device for estimating coordinates.
The method of claim 2 or 3, wherein the conversion unit; Is

Through the camera model of the unmanned aerial vehicle, the 3-dimensional position of the camera coordinate system is calculated with respect to the target's reference pixel coordinates, which are 2-dimensional coordinates, and the target's standard is calculated using external parameter values between the camera and the unmanned aerial vehicle CG (Center of Gravity). A device for estimating GPS coordinates of a target based on camera image information of an unmanned aerial vehicle, characterized by converting pixel coordinates into three-dimensional coordinates for the body coordinate system of the unmanned aerial vehicle.
According to claim 4, wherein the conversion unit; Is

Using the current GPS coordinates of the unmanned aerial vehicle, the 3-dimensional coordinates of the target expressed in the unmanned aerial vehicle body coordinate system are converted to the ECEF (Earth-Centered Earth-Fixed) coordinate system and then converted back to the LLA (Latitude-Longitude-Altitude) coordinate system. , A device that estimates the GPS coordinates of a target based on camera image information of an unmanned aerial vehicle, characterized in that it acquires the GPS coordinates of the target.
(A) Through the artificial intelligence deep learning model learned (pre-trained) in the detection department, the target corresponding to the mission set for the unmanned aerial vehicle is detected and the bounding box containing the detected target is created. Generating according to the size of the target;

(B) setting a reference pixel, which is a target point expressed as a pixel on a camera image plane, in a setting unit;

(C) calculating the reference pixel coordinates of the target in a two-dimensional pixel coordinate system in a calculation unit;

(D) The conversion unit converts the reference pixel coordinates of the target into 3D coordinates using at least one of the altitude information of the unmanned aerial vehicle and the depth information of the target, and converts the coordinate system of the converted 3D coordinates. Obtaining GPS coordinates of a target; A method of estimating GPS coordinates of a target based on camera image information of an unmanned aerial vehicle, including.
According to clause 6,

The target is

Includes ground and air targets,

Step (D) above; Is

Estimating GPS coordinates of a target based on camera image information of an unmanned aerial vehicle, characterized in that the altitude information of the unmanned aerial vehicle is used as depth information of the ground target to convert the reference pixel coordinates of the target into 3-dimensional coordinates. method.
The method of claim 6, wherein step (D); Is

Characterized by acquiring depth information of the target in the sky through a stereo camera or depth camera that measures depth, and converting the reference pixel coordinates of the target into three-dimensional coordinates, the target is captured based on the camera image information of the unmanned aerial vehicle. How to estimate GPS coordinates.
The method of claim 7 or 8, wherein step (D); Is

Through the camera model of the unmanned aerial vehicle, the 3-dimensional position of the camera coordinate system is calculated with respect to the target's reference pixel coordinates, which are 2-dimensional coordinates, and the target's standard is calculated using external parameter values between the camera and the unmanned aerial vehicle CG (Center of Gravity). A method of estimating GPS coordinates of a target based on camera image information of an unmanned aerial vehicle, characterized by converting pixel coordinates into three-dimensional coordinates for the body coordinate system of the unmanned aerial vehicle.
The method of claim 9, wherein step (D); Is

Using the current GPS coordinates of the unmanned aerial vehicle, the 3-dimensional coordinates of the target expressed in the unmanned aerial vehicle body coordinate system are converted to the ECEF (Earth-Centered Earth-Fixed) coordinate system and then converted back to the LLA (Latitude-Longitude-Altitude) coordinate system. , A method of estimating the GPS coordinates of a target based on camera image information of an unmanned aerial vehicle, characterized by acquiring the GPS coordinates of the target.
Checking whether the center pixel of a ground target captured using a camera mounted on a drone is located on an image plane;

If the center pixel of the ground target is checked to be located on the image plane, calculating a yaw angle for a yaw rotation;

Aligning in the horizontal direction through yawing rotation by the calculated yawing angle;

calculating a distance for movement of the drone; and

A ground target tracking method using a drone equipped with a camera, including the step of moving forward or backward by a calculated distance and aligning it in the vertical direction.
According to clause 11,

The yawing angle is,
(here,
is the x-coordinate value converted from the pixel coordinates located on the image plane to the drone body coordinate system,
is a y-coordinate value obtained by converting pixel coordinates located on the image plane to the drone body coordinate system. A ground target tracking method using a drone equipped with a camera, characterized in that calculated.
According to clause 11,

The distance above is,
(here,
,
,
is the distance between the altitude reference point H of the drone and the point O where the camera optical axis meets the ground,
is the height of the camera origin away from the ground,
is the x-coordinate value converted from the pixel coordinates located on the image plane to the drone body coordinate system,
is a y-coordinate value obtained by converting the pixel coordinates located on the image plane into the drone body coordinate system,
is the z coordinate value converted from the pixel coordinates located on the image plane to the drone body coordinate system,
is the drone body coordinate system
A method of tracking a ground target using a drone equipped with a camera, which is calculated by a tilt angle in the direction of the ground relative to the axis.
According to clause 11,

The step of aligning in the horizontal direction is,

In order to ensure that the pixel selected as the ground target is located at the center of the image plane, the heading angle of the drone is rotated through yawing in place to align the selected pixel to be located at the horizontal center of the image plane. A ground target tracking method using a drone equipped with a camera.
According to clause 11,

The step of aligning in the vertical direction is,

A ground target tracking method using a drone equipped with a camera, characterized in that the drone is moved forward or backward to align the vertical direction so that the selected pixel is located at the center of the image plane.
a check unit that checks whether the center pixel of a ground target captured using a camera mounted on a drone is located on an image plane;

a yawing angle calculation unit that calculates a yawing angle for yawing rotation when it is checked that the center pixel of the ground target is located on the image plane;

A horizontal alignment unit that aligns in the horizontal direction through yawing rotation by the calculated yawing angle;

a distance calculation unit that calculates the distance for movement of the drone; and

A ground target tracking device using a drone equipped with a camera including a vertical alignment unit that moves forward or backward by a calculated distance and aligns in the vertical direction.
According to clause 16,

The yawing angle calculator,

(here,
is the x-coordinate value converted from the pixel coordinates located on the image plane to the drone body coordinate system,
A ground target tracking device using a drone equipped with a camera, characterized in that the yaw angle is calculated using the formula (y coordinate value converted from the pixel coordinate located on the image plane to the drone body coordinate system).
According to clause 16,

The distance calculator,

(here,
,
,
is the distance between the altitude reference point H of the drone and the point O where the camera optical axis meets the ground,
is the height of the camera origin away from the ground,
is the x-coordinate value converted from the pixel coordinates located on the image plane to the drone body coordinate system,
is a y-coordinate value obtained by converting the pixel coordinates located on the image plane into the drone body coordinate system,
is the z coordinate value converted from the pixel coordinates located on the image plane to the drone body coordinate system,
is the drone body coordinate system
A ground target tracking device using a drone equipped with a camera, characterized in that the distance is calculated using the formula of (tilt angle in the direction of the ground relative to the axis).