KR102267287B1 - LSTM-based future threat prediction method and apparatus - Google Patents

Abstract

Disclosed are a long short term memory (LSTM)-based future threat prediction method which does not rely on visual information and considers influence f radio interference. According to the present invention, an LSTM-based future threat prediction apparatus comprises a processor and a memory connected to the processor. The memory stores executable program instructions allowing the processor to set a first travel path of a drone having a plurality of steps; to input a feature vector of a past step before the current step included in the first travel path to an LSTM model to determine influence from a threat entity in a future step, wherein the feature vector includes information about a threat entity signal reception direction and a threat entity signal type received from the threat entity; and toreset the first travel path to a second travel path in consideration of the influence of the threat entity in the future step.

Description

LSTM-based future threat prediction method and apparatus

The present invention relates to a method and apparatus for predicting future threats based on LSTM, and more particularly, to a method and apparatus for ensuring safe driving of a drone.

Recently, drones have been widely used for personal, commercial and military purposes. As a result, many drone-related accidents have occurred. For example, collisions between objects, such as drones and passenger planes, may result in human casualties, and general users may fall due to frequency interference while operating drones in large cities with high frequency interference.

Recently, research on collision avoidance of unmanned entities such as drones has been actively conducted. Typically, a field of view is secured through a camera so that an object or terrain with a risk of collision from the surrounding environment can be avoided with visual information.

In addition, there is a method of providing an evasion command using an unmanned object or data observed from the ground by a ground station. In recent years, interest in technology research that detects and avoids obstacles by itself is also increasing.

1 illustrates a camera- or ultrasound-based collision avoidance technology.

Referring to FIG. 1 , obstacles can be avoided by mounting a camera on a drone to recognize and avoid obstacles, or by mounting an ultrasonic sensor to analyze information that an ultrasonic wave generated from a drone arrives at an object and returns.

A drone equipped with a camera or ultrasonic sensor is capable of recognizing and detecting surrounding objects, so it has an excellent ability to avoid threats.

However, it is difficult to avoid threats such as relatively fast flying objects or radio wave interference with unknown locations.

Equipment such as a camera cannot avoid threats such as radio wave interference that is invisible to the field of view due to its high visual dependence, and ultrasonic sensors have a relatively short distance to recognize objects, making it difficult to avoid threats approaching at high speed. There is a problem.

Republic of Korea Patent 10-2054713

In order to solve the problems of the prior art, the present invention intends to propose a method and apparatus for predicting future threats based on LSTM that do not depend on visual information and can also consider the effect on radio wave interference.

In order to achieve the above object, according to an embodiment of the present invention, as a future threat element prediction apparatus, a processor; and a memory connected to the processor, wherein the memory sets a first driving path of the drone having a plurality of steps, and before a current step included in the first driving path in a Long Short Term Memory (LSTM) model store program instructions executable by the processor to determine the influence of the threat entity in the future step by inputting the feature vector in the past step of An apparatus for predicting future threats including information on the type of entity signal is provided.

The feature vector may include, based on the current step, the driving direction of the drone, the receiving direction of the threat entity signal, the type of the threat entity signal, the future drone driving direction, the average angle of change of the threat entity signal, and the threat entity. It may include at least one of the number of steps included in the first threat zone by the threat entity and the number of steps included in the second threat zone by the threat entity.

The traveling direction of the drone and the signal receiving direction may be defined as one of a preset number of directions.

The type of the threat entity signal may be defined differently depending on whether the drone is included in the first threat region and the second threat region.

The future drone driving direction may be a drone driving direction in each of a preset number of steps following the current step on the first driving route.

The first threat area is defined by a first radius from the threat entity, the second threat area is defined as a second radius from the threat entity, and the first threat area is an area defined by the first radius may be defined as an area excluding the area defined by the second radius.

The average change angle of the threat entity signal may be an average of the reception angles of the threat entity signal at each step at which the threat entity signal is received up to the current step.

The LSTM model is an encoder-decoder model, in which feature vectors of a preset number of steps before a current step are input to the encoder, and the decoder uses information input from the encoder to a preset number after the current step In each step of , the influence of the threat entity can be determined probabilistically.

According to another aspect of the present invention, there is provided a method of predicting a future threat in a device including a processor and a memory, the method comprising: setting a first driving route of a drone having a plurality of steps; and inputting a feature vector in a past step before the current step included in the first driving path into a Long Short Term Memory (LSTM) model to determine the influence of the threat entity in a future step, wherein the feature vector is provided, a method for predicting future threat elements including information on a threat entity signal reception direction and threat entity signal type from the threat entity.

According to another aspect of the present invention, there is provided a computer-readable program stored in a recording medium for performing the above method.

According to the present invention, there is an advantage in that it is possible to determine whether the driving path of the drone is safe even if visibility is not secured and the location of the threat is not known.

1 illustrates a camera- or ultrasound-based collision avoidance technology.
2 is a diagram illustrating a configuration of an apparatus for predicting future threats according to an embodiment of the present invention.
3 is a diagram illustrating a process of resetting a driving route of a drone through prediction of future threats according to an embodiment of the present invention.
4 is a diagram exemplarily illustrating a traveling route of a drone according to the present embodiment.
5 is a diagram illustrating a feature vector in a specific step of FIG. 4 .
6 is a diagram illustrating an LSTM model according to an embodiment of the present invention.
7 is a diagram illustrating a learning process of an LSTM model according to an exemplary embodiment of the present invention.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to specific embodiments, and it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention.

The present invention predicts future threats while driving a drone, and determines whether a preset driving route is safe in relation to a threat entity.

Here, the threat entity may be an element such as a passenger plane or a detector that generates radio interference and a specific signal.

2 is a diagram illustrating a configuration of an apparatus for predicting future threats according to an embodiment of the present invention.

As shown in FIG. 2 , the apparatus for predicting future threats according to the present embodiment may include a processor 200 and a memory 202 .

The processor 200 may include a central processing unit (CPU) capable of executing a computer program or other virtual machines.

Memory 202 may include a non-volatile storage device such as a fixed hard drive or a removable storage device. The removable storage device may include a compact flash unit, a USB memory stick, and the like. Memory 202 may also include volatile memory, such as various random access memories.

The memory 202 stores program instructions executable by the processor 200 .

The program commands according to an embodiment of the present invention set a first travel path of the drone having a plurality of steps, and in a past step before the current step included in the first travel path in a Long Short Term Memory (LSTM) model. Input the feature vector of to determine the influence of the threat entity in the future step.

3 is a diagram illustrating a process of resetting a driving route of a drone through prediction of future threats according to an embodiment of the present invention.

When the first travel path (path #1) of the drone having a plurality of steps is set as shown in FIG. 3A , the device according to the present embodiment uses the feature vector in step 7 (current step) in advance. Determine the impact of the threat entity.

In the future step, the effect (threat level) by the threat entity may be one of safe, warning, and threat.

As shown in FIG. 3B , the device according to the present embodiment may reset the first travel path to the second travel path path #2 in a direction to minimize the influence of the threat entity.

4 is a diagram illustrating a driving path of a drone according to the present embodiment, and FIG. 5 is a diagram illustrating a feature vector in a specific step of FIG. 4 .

4 to 5 , the feature vector according to the present embodiment may include information on a threat entity signal reception direction and threat entity signal type from the threat entity.

In more detail, the feature vector according to the present embodiment is based on the current step (step 11 of FIG. 4 ), the driving direction of the drone, the threat entity signal reception direction, the threat entity signal type, the future drone driving direction, and the threat entity. It may include at least one of the average change angle of the signal, the number of steps included in the first threat region by the threat entity, and the number of steps included in the second threat region by the threat entity.

The driving direction of the drone and the threat object signal reception direction may be defined as one of a preset number of directions. For example, the preset number of directions may be 8 directions, and the driving direction of the drone and the signal reception direction may be determined as one of them.

The type of the threat entity signal according to the present embodiment may be defined differently depending on whether the drone is included in the first threat region and the second threat region from the threat entity.

4 , the first threat area (green area) may be defined as a first radius from the threat entity, and the second threat area (red area) may be defined as a second radius from the threat entity. In addition, the first threat area may be defined as an area excluding the area defined by the second radius from the area defined by the first radius.

Assuming that the threat object is a radar, the green area is the area where the drone can recognize the threat object, but the threat object is far away from the threat object so the drone can be recognized, and the red area is the drone and the threat object because the distance is close. All objects may be regions in which they can recognize each other.

The type of the threat entity signal according to the present embodiment is information for distinguishing the two threat areas. For example, a green area may have a value of 0 and a red area may have a value of 1.

The future drone traveling direction is the drone traveling direction in each of a preset number of steps following the current step, and for example, as shown in FIG. 5 , may be defined as one of 8 directions for 5 steps.

In addition, the average change angle of the threat entity signal is an average of the reception angles of the threat entity signal at each step at which the threat entity signal is received up to the current step.

As shown in Figure 4, when considering the influence of the threat entity from step 4, the average value of each threat entity signal reception angle from step 4 to step 10 is the threat at the current step (step 11). It is the average angle of change of the individual signal.

Next, the feature vector includes information on the number of steps included in the first threat zone and the number of steps included in the second threat zone up to the current step.

6 is a diagram illustrating an LSTM model according to an embodiment of the present invention.

As shown in FIG. 6 , the LSTM model is an encoder-decoder model, and a feature vector of each preset number of steps before the current step is input to the encoder, and the decoder uses information input from the encoder Thus, the influence of the threat entity is probabilistically determined in each of the preset number of steps after the current step.

As described above, when the effect of the threat entity is divided into one of safety/warning/threat, the decoder probabilistically determines and outputs one of safety/warning/threat for each of the preset number of steps after the current step. can

Although FIG. 6 illustrates that the encoder and the decoder according to the present embodiment are composed of two layers, the present invention is not limited thereto.

Through this, the device for predicting future threats according to the present embodiment determines whether the driving path of the drone is safe in relation to the threat entity.

As described above, the prediction of future threats according to the present embodiment may be performed based on a Long Short Term Memory (LSTM) model, and for this purpose, a process of learning future threats of the LSTM model is preceded.

7 is a diagram illustrating a learning process of an LSTM model according to an exemplary embodiment of the present invention.

7 shows a first driving path of the drone is set from the lower left to the upper right of the grid map (step 700).

Operation 700 may be performed using initial parameters such as a size of the grid map, a plurality of driving routes and direction information of the drone, and the like.

Next, a drone driving environment is randomly generated around the driving path of the drone by using environmental parameters such as clouds and mountains (Cloud and Mt.) (step 702).

Thereafter, the threat entity is randomly placed around the drone driving path (step 704).

LSTM data through the simulation operation is collected according to the first driving route (step 706).

Here, the LSTM data is information sensed by the drone in each step included in the first travel path, and is a feature vector in each step.

Finally, the collection of LSTM data is repeated through a random driving route change (step 708).

A learning process for resetting the driving route in which the influence of the threat entity is minimized may be performed using the data collected from the randomly changed driving route in step 708 .

The LSTM data collected while the drone is driving is time-series sequence data as data in each sequential step.

The sequence data according to the present embodiment is classified into a sequence training dataset and a sequence test dataset.

Learning of the LSTM model is performed using the sequence training dataset, and the LSTM model that has been trained receives the sequence test dataset as an input to predict future threats and resets the drone driving route.

The above-described embodiments of the present invention have been disclosed for the purpose of illustration, and various modifications, changes, and additions will be possible within the spirit and scope of the present invention by those skilled in the art having ordinary knowledge of the present invention, and such modifications, changes and additions should be regarded as belonging to the following claims.

Claims (10)

As a future threat prediction device,
processor; and
a memory coupled to the processor;
The memory is
Setting a first driving route of a drone having a plurality of steps,
input the feature vector in the past step before the current step included in the first travel path into the LSTM (Long Short Term Memory) model to determine the influence of the threat entity in the future step;
store program instructions executable by the processor;
The feature vector includes: a direction of receiving a threat entity signal from the threat entity, information on a threat entity signal type, a driving direction of the drone, a future drone driving direction, an average angle of change of the threat entity signal, and a second by the threat entity. 1 includes the number of steps included in the threat zone and the number of steps included in the second threat zone by the threat entity,
The type of the threat entity signal is defined differently depending on whether the drone is included in the first threat region and the second threat region,
The first threat area is defined by a first radius from the threat entity, the second threat area is defined as a second radius from the threat entity, and the first threat area is an area defined by the first radius A device for predicting future threats defined as an area excluding the area defined by the second radius in the . delete According to claim 1,
A future threat element prediction device in which the driving direction of the drone and the signal reception direction are defined as one of a preset number of directions. delete According to claim 1,
The future drone driving direction is a drone driving direction in each of a preset number of steps following the current step on the first driving path. delete According to claim 1,
The average change angle of the threat entity signal is an average of the reception angles of the threat entity signal at each step at which the threat entity signal is received up to the current step. According to claim 1,
The LSTM model is an encoder-decoder model, a feature vector of each of a preset number of steps before a current step is input to the encoder, and the decoder uses information input from the encoder to a preset number after the current step A device for predicting future threats to probabilistically determine the influence of the threat entity in each step of A method for predicting future threats in a device comprising a processor and memory, the method comprising:
setting a first driving route of the drone having a plurality of steps; and
Comprising the step of inputting a feature vector in a past step before the current step included in the first driving path into an LSTM (Long Short Term Memory) model to determine the influence of the threat entity in the future step,
The feature vector includes information on a direction of receiving a threat entity signal from the threat entity and information on a threat entity signal type, a driving direction of the drone, a future drone driving direction, an average angle of change of the threat entity signal, and a second by the threat entity. 1 includes the number of steps included in the threat zone and the number of steps included in the second threat zone by the threat entity,
The type of the threat entity signal is defined differently depending on whether the drone is included in the first threat region and the second threat region,
The first threat area is defined by a first radius from the threat entity, the second threat area is defined as a second radius from the threat entity, and the first threat area is an area defined by the first radius A method for predicting future threats defined as an area excluding the area defined by the second radius in the . A computer readable program stored in a recording medium for performing the method according to claim 9 .

Images