KR20230105412A - Method for precision landing of drone and device thereof - Google Patents

Abstract

A drone precision landing method according to the present invention includes the steps of setting a search area on a query image transmitted from a video capture module, aligning the heading direction of a drone with the heading direction when capturing the query image, and selecting a target image from the image capture module. Receiving the message, detecting the landing position in the target image and setting a landing area, setting the landing area as a tracking target area, and adjusting the horizontal position of the tracking target area to be located in the center of the captured image. The step of setting the search area may include automatically generating a search area on the query image by utilizing a template proposal network.

Description

Drone precision landing method and device {METHOD FOR PRECISION LANDING OF DRONE AND DEVICE THEREOF}

The present invention relates to a method and device for precise landing of a drone, and more particularly, by using an image captured by a camera mounted on a drone, even when there is no landing mark or a plurality of identical landing marks exist, the drone precisely takes off from the position. It relates to a method and device for precise landing of a drone capable of landing.

A drone is an unmanned aerial vehicle that flies by induction of radio waves without a human being on board. In recent years, the US aviation regulator has approved the first fully automated commercial drone flight, and the development of autonomous drone operation technology is actively underway. One of the technologies for automating drone operation is autonomous landing technology. In particular, in order to automate drone operation, it is necessary to automatically detect the landing location for autonomous landing.

Currently, most drones fly based on GNSS (Global Navigation Satellite System) information, and can land again at the location from which they took off using GNSS information of the take-off location. However, GNSS has a problem in that an error of up to several tens of meters may occur depending on the environment, and thus precise landing cannot be guaranteed with the take-off position of the drone. However, RTK-GNSS (Real Time Kinematic-Global Navigation Satellite System) has a horizontal/vertical error of less than several centimeters (cm) when the environment around the drone is good. can land smoothly. However, in an area where the satellite reception environment is not good, such as an urban area with many buildings and structures or a mountainous terrain, a problem in that precision may occur. That is, the technology for precise landing using RTK-GNSS information has a problem in not guaranteeing a stable precise landing regardless of the surrounding environment. In addition, there is a disadvantage in that real-time communication for receiving a correction signal must be secured in order to use RTK-GNSS.

Therefore, it is a technology that uses the image of the ground direction taken by the drone for autonomous landing without relying on GNSS. A method for landing is being used.

However, in the conventional method, if there are several identical landing marks in a small area, multiple landing marks are detected in the video taken from the bottom, so the landing mark from which the drone took off cannot be identified. Even if the GNSS information is fused, there is a problem in that the landing marks cannot be distinguished if the distance between the landing marks is within the GNSS error range. In addition, autonomous landing can be performed using the GNSS information stored during takeoff in a place without a landing sign.

An object of the present invention for solving the above problems is to provide a precision landing method and device for a drone that can accurately land on a landing mark from which each drone took off, even if a plurality of identical landing marks exist in a narrow area. there is.

Another object of the present invention to solve the above problems is to provide a precision landing method and device for a drone that allows the drone to accurately land at a location where the drone took off even in a place without a landing sign.

In order to achieve the above object, a method for precise landing of a drone according to an embodiment of the present invention includes setting a search area on a query image transmitted from an image capturing module, aligning the heading direction of a drone with the heading direction when capturing the query image. step of receiving a target image from an image capture module, detecting a landing position in the target image and setting a landing area, setting the landing area as a tracking target area, (a) the tracking target area descending while adjusting the horizontal position to be located at the center of the photographed image, (b) sampling the image captured while descending based on a predetermined time or a predetermined altitude change, and tracking target area in the sampled image It includes reducing and resetting and repeatedly performing steps (a) and (b), and the setting of the search area includes the step of using a template proposal neural network (Template Proposal Network). It is characterized in that a search area is automatically generated on the query image.

According to the present invention, the precise landing method and apparatus of a drone can precisely land on a landing marker from which each drone took off, even if a plurality of identical landing markers exist in a small area.

In addition, the precision landing method and device of the drone can precisely land the drone at the location where the drone took off even in a place without a landing mark.

1 is a conceptual diagram showing a drone precision landing process according to an embodiment of the present invention.
2 is a block diagram of a drone precision landing device according to an embodiment of the present invention.
3 is a flowchart of a drone precision landing device during takeoff according to an embodiment of the present invention.
4 is a flowchart illustrating a landing of a drone precision landing device according to an embodiment of the present invention.
5 is a flowchart illustrating a landing position detection of a drone precision landing device according to an embodiment of the present invention.
6 is a diagram showing template detection results according to an embodiment of the present invention by way of example.
7 is a conceptual diagram of an artificial neural network for detecting a landing position according to an embodiment of the present invention.

Since the present invention can make various changes and have various embodiments, specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention. The terms and/or include any combination of a plurality of related recited items or any of a plurality of related recited items.

In embodiments of the present application, “at least one of A and B” may mean “at least one of A or B” or “at least one of combinations of one or more of A and B”. Also, in the embodiments of the present application, “one or more of A and B” may mean “one or more of A or B” or “one or more of combinations of one or more of A and B”.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this application, they should not be interpreted in an ideal or excessively formal meaning. don't

Hereinafter, with reference to the accompanying drawings, preferred embodiments of the present invention will be described in more detail. In order to facilitate overall understanding in the description of the present invention, the same reference numerals are used for the same components in the drawings, and redundant descriptions of the same components are omitted.

Technology for inducing autonomous landing of drones without relying on GNSS is mainly used for technology that uses images taken from drones in the direction of the ground. Technologies for inducing autonomous landing of drones using images taken from drones on the ground can be largely classified into two technologies. The first is a technology that analyzes ground images taken by drones to automatically determine a safe area for landing and controls the drone to land in that area. For example, by using artificial intelligence technology, it is possible to induce a drone to land by identifying an area where the ground is flat and there are no obstacles around. However, this method has the advantage of being easy to use because there is no need to install a landing guidance device in advance, but there is a limitation that landing cannot be performed if an area that satisfies the landing conditions is not found. In addition, since the drone does not exactly land at the place where it took off, but returns to the air around the place where it took off using GNSS information and lands in an area that is considered safe, there is a possibility that the drone lands in an unsafe place. , there is a problem that a plurality of drones can land in the same area at the same time. For example, when a drone is used to search for a missing person, several drones may simultaneously fly and land at the same time to search for a missing person. At this time, a situation in which several drones collide may occur by determining the same area as a landing location and attempting to land. Therefore, it is suitable for an emergency situation where a drone needs to land urgently, but it is unreasonable to apply it in a general drone operation situation.

The second method of using images for autonomous landing of a drone is to place a landing marker on the ground, detect a landing marker from an image of a lower area taken by a drone, and land the drone at the detected location. However, if there are several identical landing marks in a small area, a plurality of landing marks are detected in an image taken from the bottom, so there is a problem in that a specific location from which the drone took off cannot be identified. If multiple drones attempt to land at the same time, there is a risk of collision while flying toward the same landing marker. In addition, in order to autonomously land based on landing markers, there is a disadvantage in that drones must be operated only in areas where markers are installed in advance or drone operators must carry landing markers recognizable by drones. In particular, the landing sign must be large to identify the landing sign at the flight altitude of the drone, and must have some weight to be unaffected by wind outdoors, so it is not easy to carry, making it difficult for drone operators to carry around. there is.

Distinctive landing marks may be used where multiple landing marks are required in a small area. However, in order to be able to distinguish the landing sign from the image taken at the flight altitude of the drone, the sign must be larger or the resolution of the camera used for filming must be increased. In particular, if a cover with increased complexity is photographed for identification outdoors under strong sunlight, details of the pattern tend not to be maintained in the photographed image, so the cover must be enlarged or a high-resolution camera must be used. In addition, since a high-resolution camera has a narrow FoV (Field of View), a large GNSS error increases the possibility that the landing sign is not included in the image taken from the drone below. In addition, whenever the landing mark used by each drone is changed, new mark information must be reflected in the landing mark detection software.

Meanwhile, the above problem can be solved to some extent by using template matching or template detection technology. After the drone has completed take-off, the center area of the image taken at the bottom and the image taken at the bottom after the drone returns to the location where the drone has completed take-off based on GNSS information are matched to obtain the center of the image taken at take-off. coordinates can be obtained. However, due to GNSS errors and control errors during take-off, there is a problem in that it cannot be guaranteed that the center of the image taken from the bottom is the same as the actual take-off location after take-off is completed. In addition, if the central area of the image taken after take-off has a monotonous structure, such as a lawn, there is a problem in that an accurate take-off location cannot be found by conventional template matching or template detection technology.

In order to solve the above problems, the present invention discloses a drone precise landing method and apparatus capable of landing by detecting a take-off position using a template detection technology without a landing mark.

1 is a conceptual diagram showing a drone precision landing process according to an embodiment of the present invention.

To first define terms used in this specification, a take-off position may mean a horizontal position from which a drone takes off. The take-off location may be expressed as pixel coordinates corresponding to the take-off location in the photographed image. The query image may refer to an image obtained by photographing a vertically lower area with a camera after the drone ascends to a preset altitude and completes take-off. The search area may refer to a predetermined area set around the take-off location in the query image. Returning to the take-off position may mean movement to the altitude and horizontal position at the time of take-off completion of the drone. The target image may refer to an image obtained by photographing a vertically lower area with a camera after returning to the take-off completion position. An area corresponding to the search area may be found within the target image, and the area corresponding to the search area within the target image may be referred to as a landing area. The landing location may mean the center pixel coordinates of the landing area.

Referring back to FIG. 1 , the drone may capture a query image to set and store a search area. Thereafter, the drone may perform a predefined mission, such as flying over route points 2 and 3, and return to the location where the take-off was completed. At this time, since the return based on the GNSS information has an error depending on the surrounding environment, a difference may inevitably occur between regions included in the query image and the target image. The drone precision landing device may find a landing area corresponding to a search area within a target image, derive coordinates of a landing location, and move the drone to actual coordinates corresponding to the landing location to land.

2 is a block diagram of a drone precision landing device according to an embodiment of the present invention.

The drone precision landing device includes an image capture module 110 that captures the lower area of the drone, an image processing module 120 that samples an image received from the image capture module 110 and tracks a specific area within the image, and a location where the drone has completed take-off. After returning to the landing location detection module 130, which determines the landing location from the captured image, the landing control module 140 that allows the drone to land precisely in conjunction with the image processing module 120 and the landing location detection module 130 include

The image capture module 110 may capture an area vertically below the drone and transmit the image to the image processing module 120 . The image capturing module 110 may include a camera for capturing. The camera can be fixed to the drone via a gimbal. In other words, the camera can be connected to a gimbal that can be angled, and the gimbal can be adjusted so that the camera is always pointed at the ground directly below the drone. On the other hand, if the camera is directly attached to the drone without a gimbal, the center of the captured image may not be the vertical downward position of the drone because it cannot accurately capture the vertical downward position when the drone shakes due to wind or the like. Therefore, in the case where there is no gimbal, pixel coordinates corresponding to a vertical lower position of the drone may need to be calculated in consideration of the degree of inclination of the drone. In most cases, there is a difference between the center pixel coordinates of the captured image and the pixel coordinates of the camera principal point, but they can be assumed to coincide in this document.

The image processing module 120 may sample an image transmitted from the image capturing module 110 and may perform processing for tracking a specific region in the sampled image. Image sampling may be based on changes in time or elevation. The image processing module 120 may sample the image received from the image capture module 110 at regular time intervals, or may sample based on an altitude change when the drone takes off or ascends. Even if the drone is flight controlled to take off vertically, it may be difficult to maintain a horizontal position at the start of takeoff due to GNSS errors and various control errors. In this case, there may be a difference between the center of the image captured after take-off and the take-off position. The image processing module 120 may include a center area tracking function for making the center of the image taken after take-off complete to be the take-off position. The center area tracking function is based on the assumption that the center of the image captured at a low altitude after take-off is the same as the actual take-off location, and the center area of the image taken at a low altitude can be set as the tracking area.

The landing location detection module 130 may determine the landing location by analyzing an image taken after take-off completion and an image taken after returning to the take-off completion location. In other words, the landing location detection module 130 may derive the landing area and landing location on the target image captured after returning to the take-off location based on the query image captured after take-off and the search area set on the query image. .

The landing control module 140 may control the drone to accurately land at the landing location determined by the landing location detection module 130 and adjust the drone heading to be suitable for shooting for landing location detection.

3 is a flowchart of a drone precision landing device during takeoff according to an embodiment of the present invention.

When the drone takes off, the image capturing module 110 may photograph the lower side of the drone (S201). The image processing module 120 samples the images taken by the image capturing module 110 while taking off (S203), and continuously tracks the central area in the sampled images to accurately determine the take-off location from the images taken after take-off is completed. It can (S205). The image processing module 120 gradually expands the target area to be tracked while maintaining the center of the tracked area as the shooting altitude increases, since the area to be filmed widens during takeoff and the initial tracking area gradually becomes smaller, making accurate tracking difficult (S207 )can do. Here, the tracking target area extended according to the altitude may be used as the next tracking target area (S209). In addition, the image processing module 120 may store information about a vertical downward image taken after take-off, a take-off location, a heading direction of the drone, and an altitude (S211). Information on the stored video, take-off location, heading direction of the drone, and altitude can be used when landing.

4 is a flow chart of the drone precision landing device according to an embodiment of the present invention upon landing, and FIG. 5 is a flow chart of the drone precision landing device according to an embodiment of the present invention upon detecting a landing position.

Even if the drone is flight controlled to land vertically, it is difficult to maintain the horizontal position at the start of landing due to GNSS errors and various control errors. Therefore, there may be a difference between the horizontal position after landing and the horizontal position at the start of landing. In order to make the horizontal position after landing match the horizontal position at the start of landing, the following center area tracking function is required. The image processing module 120 continuously tracks the center area of the image taken at the start of landing in the images taken while descending, and transfers the error between the center of the tracked area and the center of the captured image to the landing control module 140 The horizontal position can be adjusted (S301). In the process of descending while tracking the center area, the image capturing module 110 may continuously image the lower side of the drone and transmit the captured image to the image processing module 120 (S303). The image processing module 120 may sample images captured while the image capturing module 110 descends (S305), and may continuously track the center area of the sampled images (S307). When the drone is descending and shooting, the area to be filmed gradually narrows, so the initial tracking target area may become too large or exceed the shooting range of the video. Accordingly, when the drone descends and the altitude decreases, the tracked area may be gradually reduced while maintaining the center of the tracked area (S309). In addition, the reduced tracking target area can be reset to the tracking target area and used as the next tracking target area (S311).

Referring to FIG. 5 , the landing location detection module 130 may determine the landing location by analyzing an image taken after take-off completion and an image taken after returning to the take-off completion location (S401). When the drone returns to the location where the take-off was completed after completing the mission, the drone's heading direction may be aligned with the heading direction at take-off (S403). Through this, the query image and the target image can be photographed in a similar direction, thereby increasing the accuracy of image analysis for detecting the landing position. After aligning the drone's heading, the lower area may be photographed and stored as a target image (S405). The landing position detection module 130 may detect the landing position by using the query image, the takeoff position, and the target image stored after takeoff as inputs of an artificial neural network for detecting the landing position (S407). An artificial neural network for landing location detection is described in detail below. After the landing location is detected, a certain area around the landing location in the target image may be set and stored as a tracking target area (S411). The landing process shown in FIG. 4 starts by transferring the tracking target area to the image processing module 120 .

6 is a diagram showing template detection results according to an embodiment of the present invention by way of example.

There are several methods for image matching to find a specific region within a query image in a target image, and among them, a key point-based template matching method is a representative method. However, the key point-based template matching method has a problem in that matching accuracy is lowered if the number of usable key points in an image is insufficient or if the key points are concentrated in a specific region of an image. Therefore, the present invention uses an artificial neural network based template matching method to find a landing position with high accuracy under various conditions.

There are existing studies on artificial neural networks that detect new types of objects or templates that are not included in the training data. The artificial neural network for landing location detection proposed in the present invention is characterized in that it further includes a step of automatically generating a template to be detected by receiving a query image as an input without receiving a template to be detected as an input.

In the present invention, a template may be automatically generated in an artificial neural network without cutting out a part of the center of a query image captured after take-off and using it as a template. Referring to FIG. 6 , if a white square area is used as a landing location template as shown in FIG. 6 (a), it may be difficult to specify exactly one area when searching for a template in an image taken after returning to the take-off location. For example, since several areas along the road look similar as shown in FIG.

On the other hand, if the area further expanded from the center is used as a template as shown in Fig. 6(c), the curved road and part of the forest are included in the template, so in the image taken after returning to the take-off completion position as shown in Fig. 6(d), It is possible to specify exactly one area. Accordingly, it is necessary to determine whether the detection accuracy can be improved by setting an area extended to the vicinity of the center of the query image captured after take-off as a template, ie, a search area, and automatically set the search area.

7 is a conceptual diagram of an artificial neural network for detecting a landing position according to an embodiment of the present invention.

An artificial neural network for detecting a landing location may be composed of a template proposal neural network and a template detection neural network. When a query image is input, the Template Proposal Network may set a search area suitable for detecting a landing location and transmit information about the set search area as an input to the Template Detection Network. In this case, since the center of an image captured after take-off may not be the take-off position, an image cropped so that the take-off position is the center may be used as a query image.

A template detection network performs a function of detecting a search area received from a template proposal network in a target image. More specifically, an area corresponding to the search area of the query image may be detected in the target image, and this area may be a landing area. At this time, in order for the drone to land precisely, it is enough to know only the center coordinates of the area, not the area information, so the template detection network can output the center coordinates of the detected landing area.

The center of the search area output by the Template Proposal Network must be the same as or less different from the take-off location, and must include a feature distinguishing it from other areas in the query image. Therefore, the loss increases when the error between the center of the generated search area and the actual take-off location increases, and the loss increases when the difference between the center coordinates of the landing area output by the Template Detection Network and the actual landing location increases. A loss function can be defined to

Even if the take-off location is not located at the center of the search area, if it is included in the search area, a landing location corresponding to the take-off location can be calculated by calculating a transformation matrix after finding the landing area. However, in order to accurately calculate the transformation matrix, the difference between the landing area and the search area must be small, and since there may be a difference in the shooting direction of the query image and the target image, the landing area information considering the tilt in the Template Detection Network should output That is, the template detection network does not output four values of the center X and Y coordinates of the landing area and the width and width of the area as landing area information, but the X and Y coordinates of the four vertices of the landing area. should output As the number of variables to be determined by the artificial neural network increases, convergence becomes more difficult during learning, and it becomes difficult to use an anchor box, which is widely used in artificial neural networks for object detection. Therefore, it is preferable to use only the center error of the two regions in the loss function rather than making the search region and the landing region coincide.

Also, the template proposal neural network may generate a plurality of search areas instead of one search area. Landing location detection accuracy can be improved by fusing information on a plurality of landing areas found using a plurality of search areas as inputs of a template detection network.

If there is a landing mark, a landing mark detection function is additionally executed in addition to the artificial neural network for landing position detection, and the landing position closest to the landing position identified by the artificial neural network for landing position detection is determined as the final landing position, thereby increasing the landing precision. can be raised

The landing control module 140 may perform a control function to adjust the drone heading to be suitable for shooting for detecting the landing location and accurately land at the landing location determined by the landing location detection module 130 . The landing location is identified by analyzing the quality image taken after take-off and the target image taken after returning to the take-off point. The smaller the difference in the shooting direction of the two images, the higher the analysis accuracy. Therefore, when taking a query image after take-off, the heading direction of the drone is stored, and after returning to the take-off point, the drone's heading is adjusted to the stored direction and a target image is captured.

When the landing location is identified, the drone is controlled to descend toward the landing location. Even if the drone starts landing after moving vertically above the landing location, the actual landing location is likely to be different due to control errors. Referring to FIG. 4 , while the drone descends, the image processing module 120 continuously tracks the landing area, and adjusts the horizontal position of the drone so that the tracked landing area becomes the center of the captured image while descending.

The methods according to the present invention may be implemented in the form of program instructions that can be executed by various computer means and recorded on a computer readable medium. Computer readable media may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on a computer readable medium may be specially designed and configured for the present invention or may be known and usable to those skilled in computer software.

Examples of computer readable media include hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes generated by a compiler. The hardware device described above may be configured to operate with at least one software module to perform the operations of the present invention, and vice versa.

Existing methods of detecting pre-determined landing markers not only require complex landing markers to make a difference between them when the number of drones increases, but also require that the markers are sufficiently complex to distinguish differences at landing start altitudes. There are downsides to growing up. In addition, in order to be able to distinguish the difference between the landing marks in the image taken when the landing start altitude is high, an expensive camera with a high resolution and a wide angle of view must be mounted on the drone, which increases the cost. Moreover, when light shines strongly in an outdoor environment, small differences in cover patterns may disappear from images taken at high altitudes, and small differences may not be revealed in low illumination.

In contrast, the present invention is an invention that can precisely land at the take-off position through template detection using a neural network, and can use a plurality of identical landing signs in a small area, thereby reducing the cost of manufacturing signs and cameras for detecting signs. , it has the effect of reducing the complexity of the label detection algorithm.

In addition, in the case of the existing method, if the same landing marker is used and a plurality of identical landing markers are included in the video taken before the drone starts landing, the same landing marker cannot be distinguished, so several drones try to land on the same marker. Doing so may result in a risk of collision. In order to eliminate the risk of collision during landing while using the same landing marker as described above, a sufficient distance must be secured between the landing markers so that only one marker can be included in the video taken after the drone returns to the take-off location based on GNSS information. . However, in a narrow environment where there are no obstacles such as trees or rocks, there is a problem in that it is difficult to leave a sufficient gap between the signs.

In contrast, the present invention is an invention that can precisely land at the take-off position through template detection using a neural network, and a plurality of drones can safely and precisely land while using the same landing marker even in a narrow area. In particular, the present invention may land at a landing mark closest to the identified landing position, but has the advantage of enabling precise landing at the take-off place even if take-off is made at an arbitrary place without a landing mark.

In addition, the existing methods can only land based on GNSS information if there is no landing sign or other landing guidance device, so the precision is low, and if there are obstacles around, the drone may be damaged due to collision with the obstacle during landing.

On the other hand, the present invention has the advantage of being able to precisely land at the take-off position through template detection using a neural network, thereby performing a precise landing at the take-off position without an additional device for guiding landing.

Although described with reference to the above embodiments, those skilled in the art will understand that the present invention can be variously modified and changed without departing from the spirit and scope of the present invention described in the claims below. You will be able to.

110 video shooting module
120 image processing module
130 landing position detection module
140 Landing Control Module

Claims (1)

In the drone precision landing method,
setting a search area on the query image transmitted from the image capturing module;
aligning the heading direction of the drone with the heading direction when the query image is captured;
receiving a target image from the imaging module;
Detecting a landing position in the target image and setting a landing area;
setting the landing area as a tracking target area; and
Adjusting the horizontal position of the region to be tracked so that the region to be tracked is located at the center of the photographed image and descending;
In the step of setting the search area,
Characterized in that the search area is automatically generated on the query image by utilizing a template proposal network.
How to land a drone with precision.

Images