KR102458270B1 - VR-based monitoring apparatus and method for non-visible flight drones for autonomous flight of drones - Google Patents

Abstract

Provided are a device and method for monitoring based on a VR of a UAV for a non-visible autonomous flight. A grid map generation module generates a 3D three-dimensional grid map for an area where an unmanned aerial vehicle will fly in a virtual reality (VR) space, and enables a flight path calculation module to calculate a flight path based on the grids representing a mission performance point of the unmanned aerial vehicle among the grids constituting the 3D three-dimensional grid map, and a monitoring module may enable a flight of the unmanned aerial vehicle to be visualized in real-time in the VR space based on a photographed photographing image and the status information of the unmanned aerial vehicle while the unmanned aerial vehicle is flying along the flight path. Therefore, the present invention is capable of increasing a monitoring convenience and efficiency.

Description

VR-based monitoring apparatus and method for non-visible flight drones for autonomous flight of drones

The present invention relates to a VR-based monitoring device and method of a UAV for autonomous flight in the invisible area, and more particularly, to provide a route consisting of 3D coordinates for autonomous flight of an unmanned aerial vehicle and to operate an unmanned aerial vehicle based on virtual reality. It relates to a VR-based monitoring device and method of a UAV for non-visible autonomous flight that can be monitored.

In the field of facility maintenance, the utility and necessity of an unmanned aerial vehicle (UAV) such as a doron for the operation and management of infrastructure that is difficult to manage with the naked eye or spans a wide area is increasing.

Currently, the autonomous driving method of the Doron using GCS (Ground Control Station) is that when the drone operator selects the route (coordinate) of the Doron and delivers it to the drone, the drone operates along the promised route and transmits its location to the GCS, and the operator is monitored in two dimensions on the GCS.

As a result, the coordinates for route planning of the Doron are given in 3D, but since the screen of the computer or tablet or GCS used by the drone operator provides a two-dimensional interface, it is difficult to precisely specify the 3D coordinates at which the drone should fly.

In addition, in order for a drone to approach a 3D object and take a picture along a 3D path, it is necessary to calculate flight coordinates by calculating an appropriate distance not to collide with an object (or obstacle), but it is impossible to measure aerial coordinates on the ground.

Whether a drone in operation approaches an obstacle can be checked visually from the position of the drone operator, visually check the camera mounted on the drone through GCS, or use the radar installed in the drone.

However, when the drone is flying out of sight or flying at night, the drone operator cannot visually check from the position of the drone operator.

Domestic Registered Patent No. 10-1815028 (Registered on December 28, 2017)

The technical problem to be achieved by the present invention in order to solve the above problems is to provide a path made of 3D coordinates to the unmanned aerial vehicle so that it can be photographed without colliding with an obstacle, and to monitor the operation of the unmanned aerial vehicle based on virtual reality. To present a VR-based monitoring device and method for UAVs for autonomous flight in line-of-sight.

The problems to be solved of the present invention are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

As a means for solving the above-described technical problem, according to an embodiment of the present invention, an apparatus for providing a flight path for autonomous flight of an unmanned aerial vehicle and monitoring autonomous flight is unmanned in a virtual reality (VR) space. a grid map generation module for generating a 3D three-dimensional grid map for an area in which an aircraft will fly (hereinafter referred to as a 'flying area'); a flight path calculation module for calculating a flight path based on the grids representing the mission performance points of the unmanned aerial vehicle among the grids constituting the 3D three-dimensional grid map; and a monitoring module that visualizes the flight of the unmanned aerial vehicle in real time in a VR space based on a captured image captured while the unmanned aerial vehicle is flying along a flight path and state information of the unmanned aerial vehicle.

It further includes a; VR data construction module for mapping the 3D object modeling data of the flying obstacle located in the flight area and the image data of the flying obstacle to build 3D VR data for the flight area.

The grid map generation module generates a 3D three-dimensional grid map for a flight area including a flight obstacle area made of flight obstacles and a flightable area not including the flight obstacle.

The flight path calculation module, when the mission performance point of the unmanned aerial vehicle is selected, a point of interest designator for designating a center point of a grid to which the selected mission performance point belongs as a point of interest; a flight path calculation unit for calculating an optimal flight path by connecting one or more points of interest designated by the point of interest designation unit; and a flight path transmission unit for processing the flight path calculated by the flight path calculation unit to be transmitted to the unmanned aerial vehicle.

The monitoring module may include: an information receiving unit configured to receive a mission performance result of the unmanned aerial vehicle performing a mission at a mission performance point and status information of the unmanned aerial vehicle; and a VR viewer that maps the current position of the unmanned aerial vehicle to the 3D three-dimensional grid map in real time based on the state information of the unmanned aerial vehicle and visualizes it based on VR; and a 3D monitoring unit for generating a monitoring screen including a plane visualization area for visualizing the flight path and the mission performance point set in the flight area in two dimensions.

The flight path calculated by the flight path calculation module is transmitted to the unmanned aerial vehicle through a GCS (Ground Control Station), and the mission performance results and status information obtained from the unmanned aerial vehicle are transmitted to the monitoring module through the GCS.

On the other hand, the method of providing a flight path for autonomous flight of an unmanned aerial vehicle and monitoring autonomous flight is (A) an electronic device, an area in which an unmanned aerial vehicle will fly in a VR (Virtual Reality) space (hereinafter, 'flying' generating a 3D three-dimensional grid map for 'region'; (B) calculating, by the electronic device, a flight path based on grids indicating a mission performance point of the unmanned aerial vehicle among grids constituting the 3D three-dimensional grid map; and (C) visualizing, by the electronic device, the flight of the unmanned aerial vehicle in real-time in VR space based on a captured image captured while the unmanned aerial vehicle is flying along a flight path and state information of the unmanned aerial vehicle; do.

Before the step (A), (D) the electronic device maps the 3D object modeling data of the flying obstacle located in the flight area and the image data of the flying obstacle to build 3D VR data for the flight area step; further includes.

In the step (A), a 3D three-dimensional grid map is generated for a flight area including a flight obstacle area made of a flight obstacle and a flightable area not including the flight obstacle.

The step (B) may include: (B1) designating a center point of a grid to which the selected mission performance point belongs as a point of interest when the mission performance point of the unmanned aerial vehicle is selected; (B2) calculating an optimal flight path by connecting one or more points of interest designated in step (B1); and (B3) processing the flight path calculated in step (B2) to be transmitted to the unmanned aerial vehicle.

The step (C) includes the steps of: (C1) receiving a mission performance result obtained by the unmanned aerial vehicle performing a mission at a mission performance point, a captured image, and state information of the unmanned aerial vehicle; and (C2) a VR viewer that maps the current location of the unmanned aerial vehicle to the 3D three-dimensional grid map in real time based on the state information of the unmanned aerial vehicle and visualizes it based on VR, and mission visualization for visualizing the mission performance result of the unmanned aerial vehicle and generating a monitoring screen including an area and a plane visualization area for visualizing the flight path and mission performance point set in the flight area in two dimensions.

The flight path calculated in step (B) is transmitted to the unmanned aerial vehicle through the GCS (Ground Control Station), and the mission performance results and status information obtained from the unmanned aerial vehicle are transmitted to the step (C) through the GCS. .

According to the present invention, a path made of 3D coordinates is provided to an unmanned aerial vehicle so that the unmanned aerial vehicle can photograph an obstacle or its surroundings without colliding with the obstacle while flying in an invisible area, and the captured image and the flight situation of the unmanned aerial vehicle are simulated. Monitoring convenience and efficiency can be increased by visualizing it on the VR screen so that it can be monitored based on reality.

Effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a view showing an autonomous flight path provision and VR-based monitoring system of an unmanned aerial vehicle according to an embodiment of the present invention;
2 is a block diagram illustrating an autonomous flight path provision and VR-based monitoring apparatus 100 of an unmanned aerial vehicle 20 according to an embodiment of the present invention;
3 and 4 are exemplary views showing the VR data construction results generated by the VR data construction module 151;
5 is an exemplary view showing the first flight path calculation screen 500 that calls the VR data of the flight area after the VR service application is executed;
6 is an exemplary view showing a second flight path calculation screen 600 that generates a grid in the VR space of the flight area;
7 is a block diagram showing the flight path calculation module 155 shown in FIG. 2 in detail;
8 is an exemplary diagram illustrating a third flight path calculation screen 800 for calculating a flight path in the VR space of the flight area in which the grid is activated;
9 is a block diagram showing the monitoring module 157 shown in FIG. 2 in detail;
10 is an exemplary diagram illustrating a monitoring screen 1000 provided while the unmanned aerial vehicle 20 is flying;
11 is a flowchart schematically illustrating a method for providing an autonomous flight path of an unmanned aerial vehicle and a VR-based monitoring method according to an embodiment of the present invention.

The above objects, other objects, features and advantages of the present invention will be easily understood through the following preferred embodiments in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments described herein and may be embodied in other forms. Rather, the embodiments introduced herein are provided so that the disclosed content may be thorough and complete, and the spirit of the present invention may be sufficiently conveyed to those skilled in the art.

In some cases, it is mentioned in advance that in describing the invention, parts that are commonly known but not largely related to the invention are not described in order to avoid confusion for no reason in explaining the invention.

In this specification, when terms such as first, second, etc. are used to describe components, these components should not be limited by these terms. These terms are only used to distinguish one component from another.

Also, when it is said that a first component is operated or executed on (ON) a second component, the first component is operated or executed in an environment in which the second component is operated or executed, or is combined with the second component It should be understood that the operation or execution is directly or indirectly through interaction.

When it is stated that a component, device, or system includes a component consisting of a program or software, even if not explicitly stated, the component, device, or system is the hardware (or hardware) necessary for the program or software to run or operate. It should be understood to include, for example, memory, CPU, etc.) or other programs or software (eg, drivers required to run an operating system or hardware, etc.).

In addition, it should be understood that, unless otherwise specified in the implementation of a certain component, the component may be implemented in software, hardware, or any form of software and hardware.

In addition, the terminology used herein is for the purpose of describing the embodiments and is not intended to limit the present invention. In this specification, the singular also includes the plural, unless specifically stated otherwise in the phrase. As used herein, 'comprises' and/or 'comprising' does not exclude the presence or addition of one or more other elements.

In addition, in this specification, terms such as 'unit', 'module', 'system', 'platform', 'device' or 'terminal' refer to hardware and functional and structural features of software driven by the corresponding hardware or for driving the hardware. It may be intended to refer to a bond. For example, the hardware herein may be a data processing device including a CPU or other processor. In addition, software driven by hardware may refer to a running process, an object, an executable file, a thread of execution, a program, and the like.

In addition, the above terms may mean a logical unit of a predetermined code and a hardware resource for executing the predetermined code, and does not necessarily mean a physically connected code or a single type of hardware. It can be easily inferred to an average expert in the technical field of

Hereinafter, specific technical contents to be practiced in the present invention will be described in detail with reference to the accompanying drawings.

Each of the components shown in FIGS. 1, 2, 7 and 9 indicates that they can be functionally and logically separated, which means that each configuration is necessarily divided into a separate physical device or written in a separate code. An average expert in the technical field of the present invention can easily infer that this is not the case.

For autonomous flight of unmanned aerial vehicles (UAVs), accurate route planning and flight status monitoring are required. The route planning of a commercial UAV operation app has a limit in establishing an accurate 3D route plan by designating a point of interest on a two-dimensional map and planning a flight route by inputting the height from the ground to the point of interest.

In addition, it is difficult to overcome the limitation of the camera angle of view in the existing peripheral risk recognition method that relies only on the camera of the unmanned aerial vehicle.

Therefore, in an embodiment of the present invention, a system for calculating the flight path of an unmanned aerial vehicle based on virtual reality (VR) and monitoring the flight situation is proposed. In an embodiment of the present invention, first, VR data of an area in which an unmanned aerial vehicle will fly is constructed. For VR data construction, topographic data, 3D object models of buildings, and images of buildings are used. Second, design the flight data linkage module of the unmanned aerial vehicle. Design the structure and main class of the link module that can receive the flight status data of the unmanned aerial vehicle in real time. Third, we plan the flight path based on VR and design the application architecture of the system that monitors the flight status in real time. Fourth, the results of the implementation of the system are shown.

As a result of the implementation, it overcomes the limitation of the first-person view, which relies only on the sensor mounted on the unmanned aerial vehicle, and provides the effect of understanding the positional relationship between the unmanned aerial vehicle and obstacles in real-time in the third-person view, and It is judged that it can be extended to the implementation of a monitoring system.

1 is a diagram illustrating an autonomous flight path provision and VR-based monitoring system of an unmanned aerial vehicle according to an embodiment of the present invention.

1 , the autonomous flight path provision and VR-based monitoring system of an unmanned aerial vehicle according to an embodiment of the present invention is a ground control station (GCS) 10, an unmanned aerial vehicle (UAV) 20 and autonomous It may include a flight path provision and VR-based monitoring device 100 .

The unmanned aerial vehicle 20 is a vehicle such as a Doron flying by manual control or automatic control. In the passive control method, the unmanned aerial vehicle 20 is operated through the ground control device 10 within the pilot's field of view or is operated through an embedded system equipped with a communication device outside the field of view. The automatic piloting method is a method of autonomously flying and performing missions without the control of a pilot. A GPS module is built into the unmanned aerial vehicle 20 to plan a flight path in the form of coordinates that the GPS module can recognize in advance. ) to enable autonomous flight and mission.

To this end, the unmanned aerial vehicle 20 receives and stores the flight route and mission execution command from the ground control device 10 , and the captured image obtained while flying along the flight route and the status information of the unmanned aerial vehicle 20 and mission performance Results may be transmitted periodically or in real time to the ground control unit 10 .

The captured image may be a real-time moving image or a still image or moving image captured at a predetermined point. The status information of the unmanned aerial vehicle 20 includes the flight speed of the unmanned aerial vehicle 20, current flight position (latitude, altitude, longitude) and attitude information, camera 3D position and attitude information (Pitch, Yaw, Roll), battery information (temperature , residual amount, etc.).

The ground control device 10 provides the autonomous flight path and transmits the flight path and mission execution command calculated by the flight path calculation module 155 of the VR-based monitoring device 100 to the unmanned aerial vehicle 20, and the unmanned aerial vehicle 20 ), the result of the mission, the captured image, and status information may be transmitted to the monitoring module 157 of the device 100 . The ground control device 10 and the device 100 may be connected through a USB (Universal Serial Bus) interface or may be connected to enable wired/wireless network communication.

The autonomous flight path provision and VR-based monitoring device 100 calculates the optimal flight path for autonomous flight of the unmanned aerial vehicle 20 and provides it with a mission execution command, and the captured image and status information transmitted by the unmanned aerial vehicle 20 may be provided through a 3D modeling screen in real time or based on virtual reality (VR) to monitor the flight of the unmanned aerial vehicle 20 .

2 is a block diagram illustrating an autonomous flight path provision and VR-based monitoring apparatus 100 of the unmanned aerial vehicle 20 according to an embodiment of the present invention.

Referring to FIG. 2 , the autonomous flight path providing and VR-based monitoring apparatus 100 of the unmanned aerial vehicle 20 according to an embodiment of the present invention includes a user interface unit 110 , a communication interface unit 120 , and a storage 130 . , a memory 140 and a processor 150 may be included.

The user interface unit 110 is a device for interfacing between a user and the device 100 , and may include various input/output devices such as a display panel, a keyboard, and a mouse.

The communication interface unit 120 transmits the flight path and the mission execution command of the unmanned aerial vehicle 20 to the ground control device 10, and the result of the mission performed by the unmanned aerial vehicle 20 from the ground control device 10, photographing It includes a communication circuit for receiving the image and the status information of the unmanned aerial vehicle (20).

The storage 130 stores 3D VR data of the flight area required to create the flight path of the unmanned aerial vehicle 20, the flight path of the unmanned aerial vehicle 20, and information on the mission and mission performance point to be given to the unmanned aerial vehicle 20 It is possible to store and store the mission performance result transmitted from the unmanned aerial vehicle 20, the photographed image, and state information. The flight area means an area in which the unmanned aerial vehicle 20 will fly.

Memory 140 may include volatile memory and/or non-volatile memory. In the memory 140, commands or data related to the components 110 to 150, one or more programs and/or software, an operating system, etc. can be saved.

The program stored in the memory 140 may include a VR building program for building VR data and a VR service application. The VR service application may provide a monitoring service by creating a flight path of the unmanned aerial vehicle 20 from VR data, and visualizing the flight result of the unmanned aerial vehicle 20 flying according to the flight path in 3D.

The processor 150 controls the overall operation of the device 100 by executing one or more programs stored in the memory 140 .

For example, the processor 150 executes the VR building program stored in the memory 140 to build VR data for the flight area, and executes a VR service application to build a VR space from the VR data and create a 3D flight path In addition, the state related to the flight of the unmanned aerial vehicle 20 can be visualized in 3D VR space.

To this end, the processor 150 may include a VR data construction module 151 , a grid map generation module 153 , a flight path calculation module 155 , and a monitoring module 157 .

When the VR building program is executed, according to a user command input through the user interface unit 110, the VR data building module 151 provides topographic data of the flight area in which the unmanned aerial vehicle 20 will fly, and the flight located in the flight area. By mapping the 3D object modeling data of the obstacle and the image data of the flying obstacle, 3D VR data for the flight area can be constructed. Flying obstacles are terrain features that impede flight, such as buildings, power lines, towers, and trees. The VR data building module 157 may be omitted in the device 100 , and in this case, VR data may be built in an external electronic device (not shown) and stored in the storage 130 .

More specifically, the VR data building module 151 may construct 3D VR data for a flight area defined by 3D modeling and texture-mapped data. For example, the VR data building module 151 may construct a realistic model through texture mapping after 3D modeling of buildings located in the flight area, and 3D modeling of surrounding buildings outside the flight area without texture mapping. If the terrain feature is used only as an obstacle checking role when the unmanned aerial vehicle 20 flies, the texture mapping may be omitted. In the embodiment of the present invention, it is assumed that the unmanned aerial vehicle 20 moves through a point of interest, and texture mapping is performed for visual confirmation of the point of interest.

For 3D modeling and texture mapping, topographic data, 3D object models of buildings, and images of buildings are required. The topographic data is produced and utilized by using a 1:5000 numerical map that can be collected free of charge from the National Geographic Information Service, and the VR data construction module 151 is built on top of the produced topographical information. By collecting 25cm-level resolution aerial image data, 3D terrain can be constructed and visualized.

In addition, the VR data building module 151 performs 3D modeling and texture mapping of the building by using the aerial image, and the surrounding buildings other than the texture-mapped building are on the 2D building layer provided by the Ministry of Public Administration and Security's road name address. Multiply the number of floors by 3m and model it in 3D.

3 and 4 are exemplary views showing the VR data construction results generated by the VR data construction module 151 .

Referring to FIG. 3 , a texture is mapped to a specific building among VR data generated by the VR data building module 151 . Referring to FIG. 4 , a VR space implemented with VR data constructed for the entire target area, that is, the entire flight area is shown.

Referring back to FIG. 2 , when the VR service application is executed, the grid map generation module 153 , the flight path calculation module 155 , and the monitoring module 157 respond to a user command input through the user interface unit 110 . The flight path calculation screens for calculating the flight path may be generated to calculate the flight path, and monitoring screens for monitoring the flight state of the unmanned aerial vehicle 20 may be generated and provided to the user interface unit 110 .

5 is an exemplary diagram illustrating the first flight path calculation screen 500 that calls up VR data of the flight area after the VR service application is executed.

Referring to FIG. 5 , the first flight path calculation screen 500 includes a 3D viewer 510 , a map area 520 , a status information area 530 , a captured image area 540 , and a function button area 550 . do.

The 3D viewer 510 shows the 3D VR space using VR data of the target area (ie, the flight area) in which the unmanned aerial vehicle 20 will fly. The 3D viewer 510 shows the flight area as a 3D VR space from a viewpoint in the inclination angle direction.

The map area 520 shows a flight area in the nasolabial direction.

The state information area 530 shows the current state information of the unmanned aerial vehicle 20 transmitted from the unmanned aerial vehicle 20 .

The captured image area 540 may display a captured image transmitted from the unmanned aerial vehicle 20 in real time or may display a mission performance result.

The function button area 550 shows a plurality of buttons for requesting a function related to flight route calculation and monitoring among a plurality of functions provided by the VR service application. Icons displayed in the function button area 500 mean, for example, grid activation, input of points of interest, optimal path calculation, GCS transmission, UAV transmission, mission start, mission stop, return, and network status.

The grid map generation module 153 may generate a 3D three-dimensional grid map for a flight area in which the unmanned aerial vehicle 20 will fly in a VR space constructed with VR data.

6 is an exemplary diagram illustrating the second flight path calculation screen 600 in which a grid is generated in the VR space of the flight area.

Referring to FIG. 6 , when a grid activation button is selected in the function button area 650 , the grid map generation module 153 generates a grid for the flight area according to the grid size set through the user interface unit 110 as the earth coordinates. It is possible to generate a 3D three-dimensional grid map by generating based on the , and display it on the 3D viewer 610 . At this time, the grid map generation module 153 generates a 3D three-dimensional grid map for the flight area including the flight obstacle area including the flight obstacle including the topographic feature and the flight area not including the flight obstacle, and the 3D viewer 610 ) can be displayed.

The grid map generation module 153 may generate a grid of only one of a flight obstacle area or a flightable area according to a user's request.

In addition, the grid map generation module 153 visualizes the grid of the flight obstacle area and the grid of the non-obstacle area in different shapes or colors, or visualizes only one of them, so that the user recognizes the flightable area and the non-flyable area with the naked eye. can make it happen

In addition, the grid map generation module 153 may set the properties of the grid corresponding to the building and the ground surface among the VR data as obstacles so that it cannot be designated as a flight path. Accordingly, as shown in FIG. 6 , when the grids (displayed in light blue) including the flight obstacle area in the 3D three-dimensional grid map for the flight area are activated, the grid map generation module 153 generates the grids whose properties are set as obstacles. Avoid becoming a flight path.

As for the properties of the grid, when the user selects an area or grid corresponding to a building, the grid map generation module 153 automatically recognizes the selected area or a building, a topographical feature, or a ground surface including the grid, and sets it as an obstacle or non-obstacle property. can be set. Alternatively, when the user selects an area or grid corresponding to a building, the grid map generation module 153 may set each grid as an obstacle or a non-obstacle property in conjunction with the properties set in the numerical elevation model.

In addition, the size of the grid set by the operator may be set, for example, to a size of 5 m x 5 m, which is an actual distance. The size of the grid is variable in the system, and is related to the flight error of the unmanned aerial vehicle 20 and the accuracy of VR data. The moving unmanned aerial vehicle 20 may have a certain position error due to wind or device error, and if the position error of VR data is large, the operator may be guided to configure a larger grid in consideration of the safety factor. However, as the size of the grid increases, it may affect detailed flight such as close-up photography.

Referring back to FIG. 2 , the flight path calculation module 155 may calculate the flight path based on the grids indicating the mission performance point of the unmanned aerial vehicle among the grids constituting the 3D three-dimensional grid map.

7 is a block diagram illustrating in detail the flight path calculation module 155 shown in FIG. 2 , and FIG. 8 is a third flight path calculation screen 800 for calculating a flight path in VR space of a flight area in which the grid is activated. is an example diagram showing

Referring to FIG. 7 , the flight path calculation module 155 includes a point of interest designation unit 155a, a flight path calculation unit 155b, and a flight path transmission unit 155c.

When the point of interest designation unit 155a selects the mission performance point of the unmanned aerial vehicle 20 (a shooting point for surrounding shooting, a logistics delivery point, etc.) in the 3D viewer 810, the grid (purple, The center point of 1~4) can be designated as the point of interest. The operator may select a point to perform a task on the 3D three-dimensional grid map displayed on the 3D viewer 810 by operating the user interface unit 110, and the point of interest designator 155a selects the center point of the grid corresponding to the selected point. designate as a point of interest.

Also, when the operator selects a point of interest, the point of interest designation unit 155a may create a task execution command and generate a screen (not shown) for receiving a starting point and an arrival point from the operator. The operator writes a mission to be performed by the unmanned aerial vehicle 20 at a point of interest through a screen (not shown). The missions vary, for example, photographing the surroundings or delivering logistics. The starting point and the ending point may be a grid corresponding to the point of interest or may be additionally designated by the operator.

At this time, since the user interface 110 cannot move to the grids in which properties are set as obstacles, the operator can distinguish the obstacle area from the flightable area while moving, for example, a mouse point of the user interface 110. Alternatively, in the grid map generating module 153, the grid of the obstacle area and the grid of the flightable area are displayed in different colors, or the outline of the obstacle is displayed overlapping with a lower contrast than the grid in the obstacle area, so that the operator can distinguish them. may be

In FIG. 8 , all of the grids in the obstacle area and some of the grids in the flightable area are activated. Accordingly, the operator can select a mission performance point based on some activated grids in the flightable area.

The flight path calculating unit 155b may calculate an optimal flight path by connecting grids corresponding to one or more points of interest designated by the point of interest designation unit 155a. In the case of FIG. 8 , since four points of interest are designated, the flight path calculating unit 155b calculates an optimal flight path passing through the four points of interest grids.

The point of interest can be specified, for example, by clicking the mouse, and the center point of all grids becomes a node, and the center point of the grid is connected by a link (i.e., 6-connected link) that connects up to 6 nodes to the source and destination. , the optimal path between points of interest may be determined by the A* algorithm.

The flight path transmitting unit 155c processes the flight path calculated by the flight path calculating unit 155b to be transmitted to the unmanned aerial vehicle 20 through the communication interface unit 129 . Accordingly, the flight path transmitter 155c transmits the flight path and mission execution command to the ground control device 10 , and the ground control device 10 transmits the received flight path and mission execution command to the unmanned aerial vehicle 20 . send.

The unmanned aerial vehicle 20 stores the received flight route, performs a mission while flying, periodically checks the state information of the unmanned aerial vehicle 20 and the photographed image captured in real time around it, and then transmits it to the ground control unit 10. .

Referring back to FIG. 2 , the monitoring module 157 displays the captured image taken while the unmanned aerial vehicle 20 is flying along the received and stored flight path, the state information of the unmanned aerial vehicle 20 and the mission performance result to the ground control device. When received through (10), information related to the flight of the unmanned aerial vehicle 20 can be visualized in real time in VR space based on the received mission performance result, captured image, and state information.

9 is a block diagram illustrating the monitoring module 157 shown in FIG. 2 in detail, and FIG. 10 is an exemplary diagram illustrating the monitoring screen 1000 provided while the unmanned aerial vehicle 20 is flying.

Referring to FIG. 9 , the monitoring module 157 includes an information receiving unit 157a and a 3D monitoring unit 157b.

The information receiving unit 157a transmits the result of the mission performed by the unmanned aerial vehicle 20 at the mission performance point, the captured image captured by the unmanned aerial vehicle 20, and the state information of the unmanned aerial vehicle 20 to the ground control device 10 and through the communication interface unit 120 .

Referring to FIG. 10 , the 3D monitoring unit 157b maps the current location of the unmanned aerial vehicle 20 to a 3D three-dimensional grid map in real time based on the state information of the unmanned aerial vehicle 20 and visualizes it in a VR-based 3D viewer ( 1010), a map area 1020 that visualizes the flight path and mission performance point set in the flight area in two dimensions, a status information area 1030 showing status information of the unmanned aerial vehicle 20, and the unmanned aerial vehicle 20 A monitoring screen 1000 including a captured image area 1040 and a function button area 1050 is generated, which shows a result of performing a task or shows a captured image captured in real time.

The monitoring screen 1000 shows that as the flight of the unmanned aerial vehicle 20 starts, the flight status information is being monitored under VR. Accordingly, the operator can check the flight state of the unmanned aerial vehicle 20 and the captured image in real time even while the unmanned aerial vehicle 20 flies in the visible region or non-visible region.

11 is a flowchart schematically illustrating a method for providing an autonomous flight path of an unmanned aerial vehicle and a VR-based monitoring method according to an embodiment of the present invention.

Since the autonomous flight path provision and VR-based monitoring device 100, the ground control device 10, and the unmanned aerial vehicle 20 shown in FIG. 11 have been described with reference to FIGS. 1 to 10, detailed descriptions will be omitted.

Referring to FIG. 11 , the autonomous flight path providing and VR-based monitoring device 100 applied as an electronic device maps 3D object modeling data of a flying obstacle located in the flight area and image data of the flying obstacle to 3D VR for the flight area. Build data (S1110).

The device 100 generates a 3D three-dimensional grid map for the flight area in which the unmanned aerial vehicle 20 will fly in the VR space (S1120).

The device 100 determines the flight path based on the grids indicating the mission performance point selected when the mission performance point of the unmanned aerial vehicle 20 and the mission to be performed among the grids constituting the 3D three-dimensional grid map generated in step S1120 are selected by the operator. can be calculated (S1130).

Step S1130 will be described in detail, and when the mission performance point, the flight start point, and the flight end point of the unmanned aerial vehicle 20 are selected by the operator on the 3D three-dimensional grid map, the device 100 selects the center point of the grid to which the selected mission performance point belongs. An optimal flight path can be calculated by designating a point of interest and connecting a number of designated points of interest and a starting point and an ending point.

The device 100 transmits the mission performance information and the flight path to the ground control device 10 , and the ground control device 10 transmits the received mission performance command and the flight path to the unmanned aerial vehicle 20 ( S1140 , S1150 ). ).

The unmanned aerial vehicle 20 performs the mission while flying the flight area according to the flight path after storing the received mission execution command and the flight path (S1160, S1170).

The unmanned aerial vehicle 20 acquires a result of performing a mission while flying, an image of the surrounding area, and flight state information, and periodically transmits it to the ground control device 10 ( S1180 ).

The ground control device 10 transmits the received mission performance result, the captured image and the state information of the unmanned aerial vehicle 20 to the device 100 (S1190).

The device 100 stores the received mission performance result, the captured image and the state information of the unmanned aerial vehicle 20, and visualizes it on the screen as shown in FIG. 10 for VR-based monitoring (S1195).

On the other hand, although described and illustrated in relation to a preferred embodiment for illustrating the technical idea of the present invention above, the present invention is not limited to the configuration and operation as shown and described as such, and deviates from the scope of the technical idea. It will be apparent to those skilled in the art that many changes and modifications to the present invention are possible without the above. Accordingly, all such suitable alterations and modifications and equivalents are to be considered as falling within the scope of the present invention. Accordingly, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

10: ground control device 20: unmanned aerial vehicle
100: Autonomous flight path provision and VR-based monitoring device
110: user interface unit 120: communication interface unit
130: storage 140: memory
150: processor 151: VR data building module
153: grid map generation module 155: flight path estimation module
157: monitoring module

Claims (10)

A device for monitoring autonomous flight and providing a flight path for autonomous flight of an unmanned aerial vehicle,
In a virtual reality (VR) space, topographic data of an area (hereinafter referred to as a 'flying area') where an unmanned aerial vehicle will fly, 3D object modeling data of a flying obstacle located in the flight area, and an image of the flying obstacle a VR data building module that maps texture data, which is data, to build 3D VR data for the flight area;
A 3D three-dimensional grid map is generated for a flight area including a flight obstacle area consisting of a flight obstacle and a flightable area not including the flight obstacle, and the grids of the flight obstacle area and the flightable area are at least in different shapes and colors. a grid map generation module that processes one to be visualized differently;
a flight path calculation module for calculating a flight path based on the grids representing the mission performance points of the unmanned aerial vehicle among the grids constituting the 3D three-dimensional grid map; and
A monitoring module that visualizes the flight of the unmanned aerial vehicle in real-time in VR space based on a photographed image captured while the unmanned aerial vehicle is flying along a flight path and state information of the unmanned aerial vehicle;
The monitoring module,
an information receiving unit configured to receive a mission performance result of the unmanned aerial vehicle performing a mission at a mission performance point and status information of the unmanned aerial vehicle; and
A 3D viewer that maps the current location of the unmanned aerial vehicle to a 3D three-dimensional grid map in real time based on the state information of the unmanned aerial vehicle and visualizes it based on VR, and the flight path and mission performance point set in the flight area are visualized in two dimensions Creating a monitoring screen including a map area, a status information area showing the status information of the unmanned aerial vehicle, a photographed image area showing a captured image taken in real time or showing a mission performance result of the unmanned aerial vehicle, and a function button area Including; 3D monitoring unit;
The VR data building module omits texture data mapping for obstacles located outside the flight area,
The grid map generation module sets the properties of a grid corresponding to a building, a topographical feature, and a ground surface among the 3D VR data as obstacles so as not to be designated as a flight route in the flight route calculation module.
A VR-based monitoring device for UAVs for non-line-of-sight autonomous flight.
delete delete According to claim 1,
The flight path calculation module,
a point of interest designation unit for designating a center point of a grid to which the selected mission performance point belongs as a point of interest when the mission performance point of the unmanned aerial vehicle is selected;
a flight path calculation unit for calculating an optimal flight path by connecting one or more points of interest designated by the point of interest designation unit; and
a flight path transmission unit for processing the flight path calculated by the flight path calculation unit to be transmitted to the unmanned aerial vehicle;
A VR-based monitoring device of a UAV for non-visual autonomous flight, characterized in that it comprises a.
delete A method of providing a flight path for autonomous flight of an unmanned aerial vehicle and monitoring autonomous flight, the method comprising:
(A) Topographic data of an area where an electronic device will fly an unmanned aerial vehicle in VR (Virtual Reality) space (hereinafter referred to as 'flying area') and 3D object modeling data of flying obstacles located in the flight area and mapping texture data, which is image data of the flying obstacle, to construct 3D VR data for the flight area;
(B) the electronic device generates a 3D three-dimensional grid map for a flight area including a flight obstacle area consisting of a flight obstacle and a flightable area not including the flight obstacle, and a grid of the flight obstacle area and the flightable area processing at least one of different shapes and colors to be visualized differently;
(C) calculating, by the electronic device, a flight path based on grids indicating a mission performance point of the unmanned aerial vehicle among grids constituting the 3D three-dimensional grid map; and
(D) visualizing, by the electronic device, the flight of the unmanned aerial vehicle in real-time in VR space based on a photographed image captured while the unmanned aerial vehicle is flying along a flight path and state information of the unmanned aerial vehicle; and ,
The step (D) is,
(D1) receiving a mission performance result obtained by the unmanned aerial vehicle performing a mission at a mission performance point, a captured image, and state information of the unmanned aerial vehicle; and
(D2) A 3D viewer that maps the current location of the unmanned aerial vehicle to a 3D three-dimensional grid map in real time based on the state information of the unmanned aerial vehicle and visualizes it based on VR; A monitoring screen comprising a map area visualized as a map area, a status information area showing the status information of the unmanned aerial vehicle, a captured image area showing the results of mission performance of the unmanned aerial vehicle or showing a captured image taken in real time, and a function button area Including;
The step (A) omits texture data mapping for obstacles located outside the flight area,
In the step (B), in the 3D VR data, properties of a grid corresponding to a building, a topographic feature, and a ground surface are set as obstacles to prevent them from being designated as a flight path in the flight path calculation module. For autonomous flight A VR-based monitoring method for UAVs.
delete delete 7. The method of claim 6,
The step (C) is,
(C1) when a mission performance point of the unmanned aerial vehicle is selected, designating a center point of a grid to which the selected mission performance point belongs as a point of interest;
(C2) calculating an optimal flight path by connecting one or more points of interest designated in step (C1); and
(C3) processing the flight path calculated in step (C2) to be transmitted to the unmanned aerial vehicle;
A VR-based monitoring method of a UAV for non-visible autonomous flight, comprising a. delete

Images