KR102022695B1 - Method and apparatus for controlling drone landing - Google Patents

Abstract

A method and apparatus for controlling drone landing is disclosed. According to an embodiment, the drone landing control apparatus recognizes a target by using an ultrasonic sensor and a GPS sensor of the drone, detects a first position of the target, and uses an optical sensor and an IR sensor of the drone to make an RGB image of the target. Acquires an IR image and recognizes the target as a landing target for the drone to land from the RGB image and the IR image, using a recognizer trained to recognize targets to be landed by the drone from the RGB images and the IR images. The second position of the target is detected, and the first position and the second position are used to generate the precise position of the target that is the landing target, and based on the precise position of the target that is the landing target and the position of the drone, the drone lands. Can be controlled.

Description

METHOD AND APPARATUS FOR CONTROLLING DRONE LANDING}

The embodiments below relate to a method, apparatus and system for controlling drone landing.

Drones are used for remote control or autonomous flight along a designated route, and are used in various fields such as camera photography, military use, transportation, security, and personal use. Drones are generally used as small unmanned aerial vehicles manually operated by the operator's radio manipulation signals, and manned drones are also commercialized.

Although the use of drones is expanding gradually, drone landing errors can cause fatal accidents and can cause fatal damage to drone operations. Therefore, research of techniques for controlling the landing of drones more accurately and safely is required.

Embodiments seek to save the time it takes for the drone to recognize the target to land and the load required for target recognition.

Embodiments improve the safety of landing by increasing the recognition accuracy of the target to be drone landing.

According to an embodiment, a drone landing control method includes: recognizing a target and detecting a first position of the target by using an ultrasonic sensor and a GPS sensor of the drone; Acquiring an RGB image and an IR image of the target by using the optical sensor and the IR sensor of the drone; Recognizing the target as the landing target for the drone to land from the RGB image and the IR image using a recognizer trained to recognize targets to be landed by the drone from RGB images and IR images, the landing target being the landing target Detecting a second position of the target; Using the first position and the second position, generating a precise position of the target that is a landing target; And controlling the landing of the drone based on the precise position of the target and the location of the drone as the landing target.

According to an embodiment, the detecting of the second position of the target may include: generating at least one feature corresponding to at least one learner previously learned based on the RGB image and the IR image; Determining whether the target is a landing target based on the at least one output generated by the at least one recognizer from the at least one feature; And detecting a second position corresponding to at least one of the RGB image and the IR image when the target is a landing target.

According to an embodiment, the generating of the feature may further include generating at least one feature corresponding to the recognizer based on the information obtained from the ultrasonic sensor and the GPS sensor.

According to an embodiment, the detecting of the second position of the target may include generating at least one first feature corresponding to the learned at least one first recognizer based on the RGB image and the IR image. step; Determining whether the target is a landing target based on the at least one first output generated by the first recognizer from the first feature; If the target is a landing target, generating at least one second feature corresponding to the at least one second recognizer previously learned based on the first output; And generating, based on the at least one second output generated by the second recognizer from the second feature, a second location of the target that is a landing target, for training the first recognizer. First training features are generated from RGB images and training IR images, and the first recognizer generates the first labels from the first training features and first labels corresponding to the training RGB images and the training IR images. Trained based on first training outputs generated by a first recognizer, for training of the second recognizer, second training features are generated from the first training outputs, and the second recognizer generates the second training Based on second labels corresponding to outputs and second training outputs generated by the second recognizer from the second training features. It can be.

According to one embodiment, controlling the landing of the drone comprises: generating at least one third feature corresponding to the at least one learned third recognizer based on the precise position and the position of the drone; And generating, by the third recognizer, control information for landing of the drone based on at least one third output generated from the third feature, and for learning the third recognizer, the drone Third training features are generated from the training precision locations and locations of the third recognizer, wherein the third recognizer is configured from the third labels and the third training features corresponding to the training precision locations and the locations. Can be learned based on the third training outputs generated by.

According to one embodiment, the step of generating the precise position generates at least one feature corresponding to the at least one recognizer learned to recognize the precise positions of the landing targets, based on the first position and the second position. Doing; And generating, by the recognizer, a precise position of the target based on at least one output generated from the feature.

According to an embodiment, a drone landing control method includes transmitting the generated precision position to a GCS of the target; Receiving an offset generated by the GCS based on the reference position and the precise position of the target; Receiving a landing permit signal generated by the GCS in response to the offset satisfying a predefined range; And correcting the precision position based on the offset, wherein controlling the landing of the drone is based on the corrected precision position and the position of the drone, and controls information for landing of the drone. It may include the step of generating.

According to an embodiment, a drone landing control method includes: receiving a reference position of the target from a GCS; Generating an offset based on the reference position and the precise position; Sending the offset to the GCS; And in response to the offset satisfying a predefined range, receiving a landing permit signal generated by the GCS.

According to an embodiment, a drone landing control method includes transmitting a reference position of the target to a GCS; Receiving a determination result of whether the reference position is changed from the GCS; If there is no change in the reference position, generating an offset based on the reference position and the precise position; And generating an offset based on the reference position and the precision position received from the GCS when there is a change of the reference position.

Drone landing control method according to an embodiment comprises the steps of receiving a precise position of the target generated from the drone; Generating an offset based on the reference position and the precise position of the target; Determining whether the offset meets a predefined range; And transmitting a landing permission signal to the drone landing control device based on the determination result.

The drone landing control method according to an embodiment may include at least one based on at least one of information obtained from an ultrasonic sensor and a GPS sensor of a drone, and at least one of an RGB image and an IR image of a target obtained from the optical and IR sensors of the drone. Creating a feature; And using the recognizer learned to recognize targets to be landed by the drone, recognizing the target as a landing target to be landed by the drone from the feature, and controlling the precise position of the target as the landing target and landing of the drone. Generating at least one of the information.

The apparatus according to one embodiment may be controlled by a computer program stored in a medium in combination with hardware to carry out the method of any one of the aforementioned methods.

Embodiments can save the time it takes for the drone to recognize the target to land and the load required for target recognition.

Embodiments can increase the safety of landing by increasing the recognition accuracy of the target to be drone landing.

1 is a flowchart illustrating a drone landing control method according to an exemplary embodiment.
2 is a conceptual diagram illustrating a drone landing control method according to an embodiment.
3 is a view for explaining a drone landing control method according to an embodiment.
4 is a view for explaining a drone landing control method according to an embodiment.
5 is a diagram for describing a learning method, according to an exemplary embodiment.
6 is a flowchart illustrating a drone landing control method according to an exemplary embodiment.
7 is a flowchart illustrating a drone landing control method according to an embodiment.
8 is an exemplary diagram of a configuration of an apparatus according to an embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for purposes of illustration only, and may be practiced in various forms. Accordingly, the embodiments are not limited to the specific disclosure, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea.

Terms such as first or second may be used to describe various components, but such terms should be interpreted only for the purpose of distinguishing one component from another component. For example, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

When a component is referred to as being "connected" to another component, it should be understood that there may be a direct connection or connection to that other component, but there may be other components in between.

Singular expressions include plural expressions unless the context clearly indicates otherwise. As used herein, the terms "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, but includes one or more other features or numbers, It is to be understood that it does not exclude in advance the possibility of the presence or addition of steps, actions, components, parts or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art, and are not construed in ideal or excessively formal meanings unless expressly defined herein. Do not.

Embodiments may be implemented in various forms of products, such as personal computers, laptop computers, tablet computers, smart phones, televisions, smart home appliances, intelligent cars, kiosks, wearable devices, and the like. For example, embodiments may be applied to recognize a user in a smart phone, mobile device, smart home system, and the like. Embodiments may be applied to a payment service through user recognition. In addition, the embodiments may be applied to an intelligent vehicle system that automatically starts the vehicle by recognizing the user. Hereinafter, exemplary embodiments will be described in detail with reference to the accompanying drawings. Like reference numerals in the drawings denote like elements.

1 is a flowchart illustrating a drone landing control method according to an exemplary embodiment.

Referring to FIG. 1, a drone landing control apparatus may recognize a target and detect a first position of the target by using an ultrasonic sensor and a global positioning system (GPS) sensor of the drone (101). . The drone landing control device is a device for controlling the landing of the drone, and may be implemented as a server or a terminal, for example, may be implemented as a software module, a hardware module, or a combination thereof. The drone landing control device may be mounted on the drone to process signals, commands or information for controlling the landing of the drone. The drone landing control device may process signals, commands or information for controlling the landing of the drone outside the drone to be controlled, and control the drone while communicating with the drone remotely. The drone may perform flight operation, take-off and landing operation, information acquisition operation, drone status information acquisition operation, charging operation automatically according to input or received command or manual adjustment, and embodiments may be performed by the type of drone. It is not limited.

The drone landing control device may recognize the target based on information obtained from the ultrasonic sensor and the GPS sensor of the drone. Here, the target may be a landing target to be landed by the drone. The target may be a landing target of a ground control station (GCS), for example, may be indicated by a marker of a predefined shape. The drone landing control device may recognize a target based on a predefined marker.

According to an embodiment, the GCS may transmit or receive commands, plans, and information for operating the drone through communication with external GCSs or drones. The network of systems for running drones enables wired and wireless communication between various entities in the system. GCSs and drones can communicate with each other via a network, and the network can use standard communication technologies and / or protocols. Drones and GCSs can induce accurate, safe and fast drone landings by exchanging information with each other to control drone landings.

The drone landing control device may recognize a positional relationship between the target and the drone based on information obtained from the ultrasonic sensor and the GPS sensor of the drone. The drone landing control device may recognize the location of the drone based on the information obtained from the GPS sensor. The drone landing control device may measure the distance between the target and the drone based on information obtained from the ultrasonic sensor and the GPS sensor of the drone. The drone landing control device may recognize the first position of the target based on the position of the drone. Here, the first position is the position of the target detected from the information obtained from the ultrasonic sensor and the GPS sensor.

Referring to FIG. 1, the drone landing control apparatus may acquire an RGB image and an IR image of a target using an optical sensor and an infrared ray (IR) sensor of the drone (102). The drone landing control device may acquire an RGB image and an IR image of the target recognized from the ultrasonic sensor and the GPS sensor. The drone landing control device quickly recognizes a target from an ultrasonic sensor and a GPS sensor, captures an image of the recognized target, speeds up the recognition of the target, saves the load required for the target recognition, and operates the target. Can increase the processing efficiency. The drone landing control device may supplement the position detection based on the ultrasonic sensor with the position detection based on the optical sensor and the IR sensor, thereby increasing the recognition accuracy of the target and detecting the precise position of the target.

Referring to FIG. 1, the drone landing control apparatus uses a recognizer trained to recognize targets to be landed by a drone from RGB images and IR images, and the landing target for the drone to land on the target from the obtained RGB image and the IR image. The second position of the target, which is the landing target, may be detected (103). Here, the second position is the position of the target detected based on the RGB image and the IR image. The drone landing control device may identify a target from the acquired RGB image and the IR image, and detect a second position corresponding to the identified RGB image and the IR image.

The drone landing control apparatus may determine whether the target is a landing target using a pre-learned recognizer and detect a second position of the target. An embodiment using the recognizer will be described later with reference to FIGS. 3 and 4. The recognizer may be learned by a deep learning technique, and embodiments related to learning will be described later with reference to FIG. 5.

Referring to FIG. 1, the drone landing control apparatus uses a first position detected based on an ultrasonic sensor and a GPS sensor, and a second position detected based on an optical sensor and an IR sensor to determine a precise position of a target that is a landing target. May be generated (104). According to one embodiment, the drone landing control device may correct the first position using the second position, and generate the precise position of the target based on the corrected result. The accuracy of the second position may be higher than the first position, in which case the drone landing control device may increase the position detection accuracy of the target using the second position.

According to one embodiment, the drone landing control device may generate a precise position of the target from the first position and the second position using a pre-learned recognizer. The recognizer can be learned to recognize the precise positions of the landing targets from the positions detected by the ultrasonic sensor and the GPS sensor and the positions detected by the optical sensor and the IR sensor. The drone landing control device may generate at least one feature corresponding to the at least one pre-learned recognizer based on the first position and the second position. The drone landing control device may generate the precise position of the target based on the at least one output generated by the at least one pre-learned recognizer from the at least one feature. Using a machine learning technique, the precise position is generated from the first position detected based on the ultrasonic sensor and the GPS sensor and the second position detected based on the optical sensor and the IR sensor. In addition, various features can be applied in various ways.

According to one embodiment, the drone landing control device may determine whether the difference between the first position and the second position exceeds a predefined threshold range. The drone landing control device may repeatedly detect at least one of the first position and the second position when the second position is far away from the first position beyond the threshold range. The drone landing control device may correct the first position by using the second position when the difference between the detected first position and the second position is again within a predefined threshold range. The drone landing control device can generate the precise position of the target from the calibration result.

According to an embodiment, the drone landing control apparatus may apply predefined weights to the first position and the second position, respectively, and generate the precise position of the target based on the weighting result. Here, the weights may each be set based on the difference between the first position and the second position. The drone landing control device may use a table that records weights corresponding to the difference between the first position and the second position. The drone landing control device may obtain weights for applying to the first position and the second position, respectively, from a previously databased table.

Referring to FIG. 1, the drone landing control apparatus may control the landing of the drone based on the precise position of the target as the landing target and the position of the drone (105). The drone landing control device may detect the position of the drone using the GPS sensor. The drone landing control device may determine the positional relationship between the drone and the target based on the position of the drone and the precise position of the target. The drone landing control device may generate control information for landing of the drone based on the positional relationship between the drone and the target. The drone landing control device may control the actuator of the drone using the control information.

According to an embodiment, the drone landing control device may generate control information for landing of the drone from the precise position of the target and the position of the drone using a pre-learned recognizer. An operation of generating control information using the recognizer will be described later with reference to FIGS. 3 to 4.

The drone landing control device may utilize the information obtained through communication with the GCS in generating control information. Details related to communication between the drone landing control device and the GCS will be described later with reference to FIG. 2.

The drone landing control device may acquire environmental information related to the flight or landing of the drone. Environmental information is information related to the environment that may affect the flight or landing of the drone, and includes, for example, parameters such as weather information (wind speed, rainfall, temperature and humidity), and information about the possibility of drone flight. The drone landing control device can obtain environmental information from an automatic weather system (AWS). AWS can be installed on drones or GCS. The drone landing control device may obtain environmental information from an external server. The drone landing control device may obtain environmental information through communication with the GCS or an external server. The drone landing control device may generate control information for landing of the drone based on the precise position of the target, the position of the drone, and the environment information. Alternatively, the drone landing control device may correct the previously generated control information based on the environment information. The drone landing control device may generate at least one feature corresponding to the at least one pre-learned recognizer from the precise position of the target, the position of the drone, and the environment information. The drone landing control device may obtain control information generated from the at least one feature by the at least one pre-learned recognizer.

2 is a conceptual diagram illustrating a drone landing control method according to an embodiment.

The drone landing control device (not shown) may recognize the position of the target 203 based on the information obtained from the sensors 202 of the drone 201. The sensors 202 may include an ultrasonic sensor, a GPS sensor, an optical sensor, and an IR sensor. The drone landing control device may control the landing of the drone 201 through communication with the GCS 204.

Referring to FIG. 2A, the drone landing control device may recognize the target 203 using the ultrasonic sensor and the GPS sensor, and detect the first position of the target 203.

Referring to FIG. 2 (b), the drone landing control apparatus may generate a feature from an RGB image and an IR image obtained by using an optical sensor and an IR sensor, and apply the generated feature to a pre-learned recognizer. The drone landing control device may determine whether the target 203 is a landing target to be landed by the drone 201 based on the output of the recognizer. When the target 203 is the landing target, the drone landing control device may detect a second position corresponding to the RGB image and the IR image.

Referring to FIG. 2 (c), the drone landing control device may control the landing of the drone 201 using the precise position of the target generated by using the first position and the second position. The drone landing control device may transmit the generated precision position to the GCS 204 of the target 203. The GCS 204 may generate an offset based on the reference position of the pre-stored target 203 and the received precision position.

The precise position may be represented by (x1, y1, z1), and the previously stored reference position may be represented by (x2, y2, z2). According to one embodiment, the GCS 204 may generate an offset using a value based on the difference or difference between the precise position (x1, y1, z1) and the reference position (x2, y2, z2). According to one embodiment, the GCS 204 preprocesses the precision position (x1, y1, z1) and the reference position (x2, y2, z2), respectively, to generate feature values, and based on the generated feature values, Lp-norm , And an offset may be generated based on the calculated value. Various schemes can be employed in the generation technique of the offset.

The GCS 204 may transmit a landing permission signal to the drone landing control device in response to the generated offset meeting a predefined range. The drone landing control device can continue the landing control of the drone 201 in response to the landing permission signal. The drone landing control device may correct the precise position based on the offset received from the GCS 204. The drone landing control device may generate control information for landing of the drone 201 based on the corrected precision position and the position of the drone 201. The control information may be transmitted to the actuator of the drone 201, and may include, for example, a value of (Pitch, Roll, Yaw), a landing speed, and the like.

According to one embodiment, the drone landing control device may receive a reference position of the target 203 from the GCS 204. The drone landing control device may generate an offset based on the received reference position and the precise position. The drone landing control device may send an offset to the GCS 204. As described above, the GCS 204 may transmit the landing permission signal to the drone landing control device in response to the case where the offset satisfies the predefined range. The drone landing control device may correct the precise position based on the offset in response to the landing hunger signal and transmit control information for landing.

According to one embodiment, the drone landing control device may transmit a pre-stored reference position of the target 203 to the GCS 204. The GCS 204 may determine whether the received reference position has changed. The GCS 204 may transmit a determination result of whether the reference position is changed to the drone landing control device. If there is a change in the reference position, the GCS 204 may transmit the changed reference position to the drone landing control device. The drone landing control device may generate an offset based on the reference position and the precise position when there is no change of the reference position. The drone landing control device may generate an offset based on the reference position and the precision position received from the GCS 204 when there is a change in the reference position.

The drone landing control device may generate control information for landing of the drone from the precise position and the offset using a pre-learned recognizer. Here, the recognizer can be trained to generate control information from the precise positions and offsets. The drone landing control device may generate at least one feature from the precision position and offset. The drone landing control device may apply the generated at least one feature to a previously learned recognizer. The drone landing control device may generate control information for landing of the drone based on at least one output generated by the previously learned recognizer.

The above-described operations, such as generating an offset to the precise position of the drone, generating control information for landing of the drone, and generating a landing permission signal, are performed by the drone landing control device or by the GCS 204. The drone landing control device and the GCS 204 may be performed in conjunction. Subjects of the above-described operations may be variously modified according to system efficiency, design intention, and load burden policy.

Referring to FIG. 2 (d), the drone landing control device may control the landing of the drone 201 based on the generated control information, thereby completing landing of the drone 201.

3 is a view for explaining a drone landing control method according to an embodiment.

Referring to FIG. 3, the drone landing control apparatus uses a first recognizer 301, a second recognizer 302, and a third recognizer 303 to target a landing from an RGB image 311 and an IR image 312. It can determine whether or not, recognize the position (x, y, z) of the target, and generate the control information (Pitch, Roll, Yaw) for landing.

The drone landing control device may generate at least one first feature 321 corresponding to at least one first recognizer 301 previously learned from the RGB image 311 and the IR image 312. The drone landing control device may further utilize the sensor information 313 in generating the first feature 321. The sensor information 313 includes information obtained from the ultrasonic sensor and the GPS sensor.

The drone landing control device may apply the at least one first feature 321 to the at least one first recognizer 301. The drone landing control device may determine whether the target is a landing target based on the at least one first output 322 generated by the at least one first recognizer 301.

According to one embodiment, for training the at least one first recognizer 301, first training features may be generated from training RGB images and training IR images. The at least one first recognizer 301 is a first training output generated by the at least one first recognizer 301 from first labels and first training features corresponding to training RGB images and training IR images. Can be learned based on them. The parameter of the at least one first recognizer 301 may be optimized in a direction to minimize the learning error defined based on the difference between the first labels and the first training outputs.

When the target is a landing target, the drone landing control device may include at least one second feature 323 corresponding to the at least one second recognizer 302 previously learned based on the at least one first output 322. Can be generated.

The drone landing control device may apply the at least one second feature 323 to the at least one second recognizer 302. The drone landing control device may generate a location of a target, which is a landing target, based on the at least one second output 324 generated by the at least one second recognizer 302. Here, the position of the target may be the above-described precision position or the second position.

According to one embodiment, for training the at least one second recognizer 302, second training features may be generated from the first training outputs. The at least one second recognizer 302 is based on second training outputs generated by the at least one second recognizer 302 from second labels and second training features corresponding to the first training outputs. Can be learned. The parameter of the at least one second recognizer 302 may be optimized in a direction that minimizes the defined learning error based on the difference between the second labels and the second training outputs.

The drone landing control device may generate at least one third feature 325 corresponding to the at least one learned third recognizer 303 based on the at least one second output 324. According to one embodiment, the drone landing control device may generate at least one third feature 325 corresponding to the at least one learned third recognizer 303 based on the precise position and the position of the drone. .

The drone landing control device may apply the at least one third feature 325 to the at least one third recognizer 303. The drone landing control device may generate control information for landing of the drone based on at least one third output 326 generated by the at least one third recognizer 303.

According to one embodiment, third training features may be generated from the second training outputs for learning of the at least one third recognizer 303. Alternatively, third training features may be generated from the training precision positions and positions of the drone. The at least one third recognizer 303 is based on third training outputs generated by the at least one third recognizer 303 from third labels and third training features corresponding to the second training outputs. Can be learned. The parameter of the at least one third recognizer 303 may be optimized in a direction that minimizes the defined learning error based on the difference between the third labels and the third training outputs. Various techniques may be employed and applied to the above-described features, a method of defining input / output of recognizers, or a learning method.

4 is a view for explaining a drone landing control method according to an embodiment.

4 (a) to 4 (c), the drone landing control apparatus recognizes a target by using various combinations of at least one recognized learner, detects the position of the target, and provides control information for landing. Can be generated.

Referring to FIG. 4A, the drone landing control device may use the first recognizer 401 and the second recognizer 402. The drone landing control device may generate the feature 403 based on the RGB image and the IR image of the target, and apply the generated feature 403 to the first recognizer 401. The drone landing control device may determine whether the target is a landing target based on the output 404 of the first recognizer 401. The drone landing control device may generate a feature based on the output 404 of the first recognizer 401 and apply the generated feature to the second recognizer 402. The drone landing control device may recognize the position of the target based on the output 405 of the second recognizer 402. The drone landing control device may generate control information for landing of the drone based on the output 406 of the second recognizer 402.

Referring to FIG. 4B, the drone landing control device may use a first recognizer 411 and a second recognizer 412. The drone landing control device may generate a feature 413 based on the RGB image and the IR image of the target, and apply the generated feature 413 to the first recognizer 411. The drone landing control device may determine whether the target is a landing target based on the output 414 of the first recognizer 411. The drone landing control device may recognize the position of the target based on the output 415 of the first recognizer 411. The drone landing control device may generate a feature based on the output 414 and the output 415 of the first recognizer 411, and apply the generated feature to the second recognizer 412. The drone landing control device may generate control information for landing of the drone based on the output 416 of the second recognizer 412.

Referring to FIG. 4C, the drone landing control device may use the first recognizer 421. According to an embodiment, the first recognizer 421 may be pre-learned in an end-to-end form. The drone landing control apparatus may generate a feature 422 based on the RGB image and the IR image of the target, and apply the generated feature 422 to the first recognizer 421. The drone landing control device may determine whether the target is a landing target based on the output 423 of the first recognizer 421. The drone landing control device may recognize the position of the target based on the output 424 of the first recognizer 421. The drone landing control device may generate control information for landing the drone at the output 425 of the first recognizer 421.

The above-described recognizers may be modified in various ways, and various techniques may be applied and learned.

5 is a diagram for describing a learning method, according to an exemplary embodiment.

The learning device may train at least one recognizer 501. The learning device is a device for learning a recognizer for drone landing control, for example, may be implemented as a software module, a hardware module, or a combination thereof. The learning apparatus can learn the above-mentioned recognizers, and its parameters can be optimized to match the characteristics or input / output of the above-described recognizers. Although an embodiment of learning is described with reference to FIG. 5, aspects of learning are not limited thereto, and learning may be performed to conform to the above-described recognizers.

Referring to FIG. 5, the learning apparatus generates a plurality of features 504 from a training RGB image 502 and a training IR image 503, and sends the generated features 504 to the plurality of recognizers 501. Can be authorized. The learning device can obtain the training output 505 generated by the recognizers 501. The learning device may generate a learning error based on the training output 505 and the label 506 corresponding to the training RGB image 502 and the training IR image 503. The learning apparatus may optimize the parameters of the recognizers 501 based on the learning error. For example, the recognizers 501 may be implemented in various ways, such as a convolutional neural network (CNN) and a recurrent neural network (RNN), and various techniques, such as a gradient descent technique, may be employed to optimize the parameter. The learning apparatus may repeat the operation of the parameter optimization until at least one of the learning error, the amount of change in the learning error, and the degree of optimization of the parameters satisfies a predefined condition and completes the learning.

6 is a flowchart illustrating a drone landing control method according to an exemplary embodiment.

Referring to FIG. 6, the GCS may receive 601 a precise location of a target generated from the drone. The GCS may generate an offset based on the reference position and the precise position of the target (602). The GCS may determine whether the generated offset satisfies a predefined range (603). The GCS may transmit a landing permission signal to the drone landing control device based on the determination result (604). Here, since the above description may be applied to the operations of the drone landing control device and the GCS, a detailed description thereof will be omitted.

7 is a flowchart illustrating a drone landing control method according to an embodiment.

Referring to FIG. 7, the drone landing control device may include at least one of at least one of information obtained from an ultrasonic sensor and a GPS sensor of a drone, and at least one of an RGB image and an IR image of a target obtained from an optical sensor and an IR sensor of the drone. A feature may be created (701). The drone landing control device recognizes the target as a landing target to be drone to land from the generated feature by using a recognizer trained to recognize targets to be landed by the drone, and controls the landing position of the drone and the precise position of the target as the landing target. At least one of the information may be generated (702). The drone landing control device may directly generate judgment results from the feature by using a pre-learned recognizer in an end-to-end format. Here, the recognizer previously learned in the form of end-to-end may be implemented similarly to the recognizer described above with reference to FIG. 4 (c).

8 is an exemplary diagram of a configuration of an apparatus according to an embodiment.

Referring to FIG. 8, apparatus 800 includes a processor 801, a memory 802, a sensor / camera 803, a communication interface 804, an actuator 805, and a data bus 806. The memory 802 may record an application, program or software for processing instructions related to drone landing control, and may record instructions for executing the operations described with reference to FIGS. 1 to 7. The device 800 may be any one or a combination of the above-described system, drone landing control device, and GCS. The processor 801 may execute a program and control the apparatus 800. Code of a program executed by the processor 801 may be stored in the memory 802. The device 800 may be connected to an external device (eg, a personal computer or a network) through the communication interface 804 and exchange data. The processor 801 may load and execute a program recorded in the memory 802. Here, since the above-described embodiments are applied to the operations of the processor 801, description of overlapping contents will be omitted.

The embodiments described above may be implemented as hardware components, software components, and / or combinations of hardware components and software components. For example, the devices, methods, and components described in the embodiments may include, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, field programmable gates (FPGAs). It may be implemented using one or more general purpose or special purpose computers, such as an array, a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of explanation, one processing device may be described as being used, but one of ordinary skill in the art will appreciate that the processing device includes a plurality of processing elements and / or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as parallel processors.

The software may include a computer program, code, instructions, or a combination of one or more of the above, and configure the processing device to operate as desired, or process it independently or collectively. You can command the device. Software and / or data may be any type of machine, component, physical device, virtual equipment, computer storage medium or device in order to be interpreted by or to provide instructions or data to the processing device. Or may be permanently or temporarily embodied in a signal wave to be transmitted. The software may be distributed over networked computer systems so that they may be stored or executed in a distributed manner. Software and data may be stored on one or more computer readable recording media.

The method according to the embodiment may be embodied in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

Although the embodiments have been described with reference to the accompanying drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques may be performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different form than the described method, or other components. Or even if replaced or substituted by equivalents, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are within the scope of the claims that follow.

201: Drone

Claims (12)

Recognizing a target and detecting a first position of the target by using an ultrasonic sensor of a drone and a Global Positioning System (GPS) sensor;
Acquiring an RGB image and an IR image of the target using an optical sensor of the drone and an infrared ray (IR) sensor;
Recognizing the target as the landing target for the drone to land from the RGB image and the IR image using a recognizer trained to recognize targets to be landed by the drone from RGB images and IR images, the landing target being the landing target Detecting a second position of the target;
Using the first position and the second position, generating a precise position of the target that is a landing target; And
Controlling the landing of the drone based on the precise position of the target and the location of the drone, which is a landing target
Including,
Detecting the second position of the target
Generating at least one feature corresponding to at least one learner previously learned based on the RGB image and the IR image;
Determining whether the target is a landing target based on the at least one output generated by the at least one recognizer from the at least one feature; And
If the target is a landing target, detecting a second position corresponding to at least one of the RGB image and the IR image
Containing
How to control drone landing.
delete The method of claim 1,
Generating the feature
Generating at least one feature corresponding to the recognizer based on the information obtained from the ultrasonic sensor and the GPS sensor
Including,
How to control drone landing.
Recognizing a target and detecting a first position of the target by using an ultrasonic sensor of a drone and a Global Positioning System (GPS) sensor;
Acquiring an RGB image and an IR image of the target using an optical sensor of the drone and an infrared ray (IR) sensor;
Recognizing the target as the landing target for the drone to land from the RGB image and the IR image using a recognizer trained to recognize targets to be landed by the drone from RGB images and IR images, the landing target being the landing target Detecting a second position of the target;
Using the first position and the second position, generating a precise position of the target that is a landing target; And
Controlling the landing of the drone based on the precise position of the target and the location of the drone, which is a landing target
Including, The detecting of the second position of the target
Generating at least one first feature corresponding to the at least one pre-learned first recognizer based on the RGB image and the IR image;
Determining whether the target is a landing target based on the at least one first output generated by the first recognizer from the first feature;
If the target is a landing target, generating at least one second feature corresponding to the at least one second recognizer previously learned based on the first output; And
Generating a second location of the target, which is a landing target, based on the at least one second output generated by the second recognizer from the second feature.
Including,
For training of the first recognizer, first training features are generated from training RGB images and training IR images,
The first recognizer is trained based on first labels corresponding to the training RGB images and the training IR images and first training outputs generated by the first recognizer from the first training features,
For training of the second recognizer, second training features are generated from the first training outputs,
Wherein the second recognizer is learned based on second labels corresponding to the second training outputs and second training outputs generated by the second recognizer from the second training features;
How to control drone landing.
Recognizing a target and detecting a first position of the target by using an ultrasonic sensor of a drone and a Global Positioning System (GPS) sensor;
Acquiring an RGB image and an IR image of the target using an optical sensor of the drone and an infrared ray (IR) sensor;
Recognizing the target as the landing target for the drone to land from the RGB image and the IR image using a recognizer trained to recognize targets to be landed by the drone from RGB images and IR images, the landing target being the landing target Detecting a second position of the target;
Using the first position and the second position, generating a precise position of the target that is a landing target; And
Controlling the landing of the drone based on the precise position of the target and the location of the drone, which is a landing target
Including, The step of controlling the landing of the drone
Generating at least one third feature corresponding to the at least one learned third recognizer based on the precision position and the position of the drone; And
Generating control information for landing of the drone based on at least one third output generated by the third recognizer from the third feature;
Including,
For training of the third recognizer, third training features are generated from the training precision positions and positions of the drone,
Wherein the third recognizer is trained based on the training precision positions, third labels corresponding to the positions and third training outputs generated by the third recognizer from the third training features;
How to control drone landing.
The method of claim 1,
Generating the precision position
Based on the first position and the second position, generating at least one feature corresponding to at least one recognizer learned to recognize precise positions of landing targets; And
Generating, by the recognizer, a precise position of the target based on at least one output generated from the feature
Including,
How to control drone landing.
Recognizing a target and detecting a first position of the target by using an ultrasonic sensor of a drone and a Global Positioning System (GPS) sensor;
Acquiring an RGB image and an IR image of the target using an optical sensor of the drone and an infrared ray (IR) sensor;
Recognizing the target as the landing target for the drone to land from the RGB image and the IR image using a recognizer trained to recognize targets to be landed by the drone from RGB images and IR images, the landing target being the landing target Detecting a second position of the target;
Using the first position and the second position, generating a precise position of the target that is a landing target;
Controlling the landing of the drone based on the precise position of the target and the location of the drone, which is a landing target;
Transmitting the generated precise position to a GCS (Ground Control Station) of the target;
Receiving an offset generated by the GCS based on the reference position and the precise position of the target;
Receiving a landing permit signal generated by the GCS in response to the offset satisfying a predefined range; And
Correcting the precision position based on the offset
Including,
Controlling the landing of the drone
Generating control information for landing of the drone based on the corrected precision position and the position of the drone;
Including,
How to control drone landing.
Recognizing a target and detecting a first position of the target by using an ultrasonic sensor of a drone and a Global Positioning System (GPS) sensor;
Acquiring an RGB image and an IR image of the target using an optical sensor of the drone and an infrared ray (IR) sensor;
Recognizing the target as the landing target for the drone to land from the RGB image and the IR image using a recognizer trained to recognize targets to be landed by the drone from RGB images and IR images, the landing target being the landing target Detecting a second position of the target;
Using the first position and the second position, generating a precise position of the target that is a landing target;
Controlling the landing of the drone based on the precise position of the target and the location of the drone, which is a landing target,
Receiving a reference position of the target from a GCS;
Generating an offset based on the reference position and the precise position;
Sending the offset to the GCS; And
In response to the offset satisfying a predefined range, receiving a landing permit signal generated by the GCS
Including,
How to control drone landing.
Recognizing a target and detecting a first position of the target by using an ultrasonic sensor of a drone and a Global Positioning System (GPS) sensor;
Acquiring an RGB image and an IR image of the target using an optical sensor of the drone and an infrared ray (IR) sensor;
Recognizing the target as the landing target for the drone to land from the RGB image and the IR image using a recognizer trained to recognize targets to be landed by the drone from RGB images and IR images, the landing target being the landing target Detecting a second position of the target;
Using the first position and the second position, generating a precise position of the target that is a landing target;
Controlling the landing of the drone based on the precise position of the target and the location of the drone, which is a landing target;
Transmitting a reference position of the target to a GCS;
Receiving a determination result of whether the reference position is changed from the GCS;
If there is no change in the reference position, generating an offset based on the reference position and the precise position; And
If there is a change in the reference position, generating an offset based on the reference position and the precision position received from the GCS
Including,
How to control drone landing.
Receiving the precise position of the target generated from the drone;
Generating an offset based on the reference position and the precise position of the target;
Determining whether the offset meets a predefined range; And
Transmitting a landing permission signal to a drone landing control device based on the determination result;
Including,
The precision position may be a first position detected by the ultrasonic sensor and the GPS sensor of the drone, and the first position detected by the recognizer previously learned from the RGB image and the IR image of the target obtained by the optical sensor and the IR sensor of the drone. Generated based on 2 positions,
How to control drone landing.
A computer program stored in a computer readable recording medium in combination with hardware for executing the method of any one of claims 1 and 3 to 10.
Using a drone's ultrasonic sensor and GPS sensor, the target is recognized, the first position of the target is detected,
Using the optical sensor and the IR sensor of the drone, to obtain the RGB image and the IR image of the target,
Recognizing the target as the landing target for the drone to land from the RGB image and the IR image using a recognizer trained to recognize targets to be landed by the drone from RGB images and IR images, the landing target being the landing target Detect a second position of the target,
Using the first position and the second position, generate a precise position of the target that is a landing target,
Control the landing of the drone based on the precise position of the target and the location of the drone, which is a landing target,
Using a drone's ultrasonic sensor and Global Positioning System (GPS) sensor, the target is recognized and the first position of the target is detected.
By using the optical sensor and IR (Infrared Ray) sensor of the drone, to obtain the RGB image and the IR image of the target,
Recognizing the target as the landing target for the drone to land from the RGB image and the IR image using a recognizer trained to recognize targets to be landed by the drone from RGB images and IR images, the landing target being the landing target Detect a second position of the target,
Using the first position and the second position, generate a precise position of the target that is the landing target,
A processor that controls the landing of the drone based on the precise position of the target and the location of the drone, which is a landing target
Containing
Drone landing control device.

Images