KR102204435B1 - Apparatus for providing an augmented reality using unmanned aerial vehicle, method thereof and computer recordable medium storing program to perform the method - Google Patents

Abstract

The present invention relates to an apparatus for providing augmented reality using an unmanned aerial vehicle, a method therefor, and a recording medium on which the method is recorded. An apparatus for providing augmented reality using an unmanned aerial vehicle according to an embodiment of the present invention includes a display section for displaying a screen, a communication section for communicating with a data server and an unmanned aerial vehicle, and a reference point to the unmanned aerial vehicle through the communication section. Spatial information including an image captured while transmitting a control command including a flight trajectory set based on the reference point, and the unmanned aerial vehicle flying through the flight trajectory set based on the reference point from the unmanned aerial vehicle through the communication unit And a control unit for controlling to display through the display unit by synthesizing a virtual model on the derived reference plane after receiving, setting a 3D coordinate system using spatial information, deriving a reference plane from the set 3D coordinate system do.

Description

Apparatus for providing an augmented reality using unmanned aerial vehicle, method thereof and computer recordable medium storing program to perform the method}

The present invention relates to augmented reality, and more particularly, to an apparatus for providing augmented reality using an unmanned aerial vehicle (UAV), a method therefor, and a recording medium on which the method is recorded.

Augmented reality refers to a technology that combines and complements virtual objects and information made with computer technology in the real world. The virtual world with additional information in real time is added to the real world and displayed as a single image. Since it was first used to explain the assembly process of Boeing's aircraft cables in 1990, technical research and development has been carried out around the United States and Japan. As smartphones appeared and became active after the mid-2000s, they began to gain attention as they applied this technology.

While VR (virtual reality) technology allows users to immerse users in a virtual environment created by computer graphics, so that they cannot see the real environment, augmented reality technology mixes virtual objects in the real environment so that users can see it in the real environment. You can receive realistic additional information. For example, it is applied in a variety of fields, such as when going on the road, when you look at the surroundings with a smartphone camera, information such as the location, phone number, and map of nearby stores is displayed in a 3D image, and when you shine the sky, weather information appears. .

Meanwhile, a technology that combines virtual reality and augmented reality is called mixed reality (MR). MR provides a mixture of real and virtual information in a real background, but it has not been commercialized yet because it requires a technology that can process large amounts of data. VR, AR, and MR all have something in common in that they are technologies that enable people to perceive reality by realizing a reality that does not exist. However, there is a difference in that AR is a method of adding virtual information to real reality, and in VR, all fictional situations are presented.

Korean Patent Publication No. 2018-0109229 published on October 08, 2018 (Name: method and apparatus for providing augmented reality function in electronic device)

An object of the present invention is to provide an apparatus capable of providing augmented reality using an unmanned aerial vehicle, a method therefor, and a computer-readable recording medium in which a program for performing the method is recorded.

An apparatus for providing augmented reality using an unmanned aerial vehicle according to a preferred embodiment of the present invention for achieving the above object includes a display unit for displaying a screen, a data server and a communication unit for communication with the unmanned aerial vehicle, and the A control command including a reference point and a flight trajectory set based on the reference point is transmitted to the unmanned aerial vehicle through a communication unit, and the unmanned aerial vehicle transmits a flight trajectory set based on the reference point from the unmanned aerial vehicle through the communication unit. Receive spatial information including images taken while flying, set a 3D coordinate system using the spatial information, derive a reference plane from the set 3D coordinate system, and synthesize a virtual model on the derived reference plane, It includes a control unit that controls to be displayed through the display unit.

The spatial information corresponds to image information including an image captured by the camera unit of the unmanned aerial vehicle and a distance from a predetermined reference point of the camera unit to an actual object corresponding to each pixel of the image captured, and a time point at which the image is captured. And sensor information including position information of the unmanned aerial vehicle acquired through the sensor unit of the unmanned aerial vehicle at the time point, attitude information of the unmanned aerial vehicle, and attitude information of the camera unit.

The control unit detects a plurality of images in which at least one of the attitude information of the unmanned aerial vehicle and the attitude information of the camera unit are different from the spatial information, and specifies a plurality of feature points in each of the detected plurality of images, and the plurality of feature points A plurality of planes are determined by obtaining a feature point vector, which is a relative distance and angle between the feature points, and the three-dimensional coordinate system is generated based on the specified plurality of planes.

The control unit is characterized in that the control unit specifies, as a plurality of feature points, a plurality of pixels whose change in pixel values is smaller than an average of change rates of pixel values in the entire pixel array of the image.

A method for providing augmented reality using an unmanned aerial vehicle according to a preferred embodiment of the present invention for achieving the above object is a control command including a reference point and a flight trajectory set based on the reference point with an unmanned aerial vehicle. Transmitting and receiving spatial information from the unmanned aerial vehicle, including an image captured by the unmanned aerial vehicle flying in a flight trajectory set based on the reference point, and setting a 3D coordinate system using the spatial information And deriving a reference plane from the set 3D coordinate system, and synthesizing and displaying a virtual model on the derived reference plane.

The spatial information corresponds to image information including an image captured by the camera unit of the unmanned aerial vehicle and a distance from a predetermined reference point of the camera unit to an actual object corresponding to each pixel of the image captured, and a time point at which the image is captured. And sensor information including position information of the unmanned aerial vehicle acquired through the sensor unit of the unmanned aerial vehicle at the time point, attitude information of the unmanned aerial vehicle, and attitude information of the camera unit.

The step of setting a 3D coordinate system using the spatial information and deriving a reference plane from the set 3D coordinate system includes a plurality of images in which at least one of the attitude information of the unmanned aerial vehicle and the attitude information of the camera unit are different from the spatial information. Detecting, specifying a plurality of feature points in each of the detected plurality of images, specifying a plurality of planes by obtaining a feature point vector, which is a relative distance and angle between feature points from the plurality of feature points, and the specified And generating the three-dimensional coordinate system based on a plurality of planes.

The step of specifying a plurality of feature points in each of the plurality of detected images is characterized in that a plurality of pixels having a change in pixel value smaller than an average of a rate of change of pixel values of the entire pixel array of the image are specified as a plurality of feature points. .

According to another aspect of the present invention for achieving the above object, a computer-readable recording medium in which a program for performing the method for providing augmented reality according to the above-described embodiment of the present invention is recorded can be provided. have.

According to the present invention, a user can easily implement an image synthesized with a virtual model even at a high altitude or far away from the user, and the number of equipment used is reduced through the improved augmented reality implementation method, and system resources are also used more efficiently. I can.

1 is a view for explaining a system for providing augmented reality using an unmanned aerial vehicle according to an embodiment of the present invention.
2 is a block diagram illustrating a configuration of a user device for providing augmented reality according to an embodiment of the present invention.
3 is a block diagram illustrating a configuration of a data server according to an embodiment of the present invention.
4 is a block diagram illustrating a configuration of an unmanned aerial vehicle according to an embodiment of the present invention.
5 is a flowchart illustrating a method for providing augmented reality using an unmanned aerial vehicle according to an embodiment of the present invention.
6 is a flowchart illustrating a method of generating a 3D coordinate system according to an embodiment of the present invention.
7 to 9 are diagrams for explaining a method of generating a 3D coordinate system according to an embodiment of the present invention.

Prior to the detailed description of the present invention, terms or words used in the present specification and claims described below should not be construed as being limited to their usual or dictionary meanings, and the inventors shall use their own invention in the best way. For explanation, based on the principle that it can be appropriately defined as a concept of terms, it should be interpreted as a meaning and concept consistent with the technical idea of the present invention. Therefore, the embodiments described in the present specification and the configurations shown in the drawings are only the most preferred embodiments of the present invention, and do not represent all the technical spirit of the present invention, and various equivalents that can replace them at the time of application It should be understood that there may be water and variations.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this case, it should be noted that the same components in the accompanying drawings are indicated by the same reference numerals as possible. In addition, detailed descriptions of known functions and configurations that may obscure the subject matter of the present invention will be omitted. For the same reason, some components in the accompanying drawings are exaggerated, omitted, or schematically illustrated, and the size of each component does not entirely reflect the actual size.

First, a system for providing augmented reality using an unmanned aerial vehicle according to an embodiment of the present invention will be described. 1 is a view for explaining a system for providing augmented reality using an unmanned aerial vehicle according to an embodiment of the present invention. Referring to FIG. 1, a system for providing augmented reality (hereinafter, abbreviated as “augmented reality system”) includes a user device 100, a data server 200, and an unmanned aerial vehicle 300.

The user device 100, the data server 200, and the unmanned aerial vehicle 300 may transmit and receive data through a network. For example, the network may include wireless communication networks such as WLAN (Wireless LAN), Wi-Fi, Wibro, Wimax, and High Speed Downlink Packet Access (HSDPA), and the system implementation method It may include wired communication networks such as Ethernet, xDSL (ADSL, VDSL), HFC (Hybrid Fiber Coaxial Cable), FTTC (Fiber to the Curb), FTTH (Fiber To The Home).

In addition, the network of the present invention may include, for example, a mobile communication network comprising a plurality of access networks (not shown) and a core network (not shown) connecting them. Here, the access network is a network that directly connects to the user device 100 or the unmanned aerial vehicle 300 to perform wireless communication, and, for example, a number of such as BS (Base Station), BTS (Base Transceiver Station), NodeB, eNodeB, etc. It may be implemented with a base station controller of a base station and a base station controller such as a base station controller (BSC) and a radio network controller (RNC). In addition, as described above, the digital signal processing unit and the wireless signal processing unit, which were integrally implemented in the base station, are divided into digital units (hereinafter referred to as DU and hereinafter referred to as RUs). A number of RUs (not shown) can be installed in each area of the network, and a number of RUs can be connected to the centralized DUs, and the core network (not shown) constituting the mobile network together with the access network is It plays a role of connecting other communication networks such as a network, for example, an Internet network.

As described above, this core network is a network system that performs main functions for mobile communication services such as mobility control and switching between access networks, and performs circuit switching or packet switching, and mobile networks It manages and controls the flow of packets within. In addition, the core network may manage mobility between frequencies, and may play a role of interworking with an access network, traffic in the core network, and other networks, such as an Internet network. These core networks will further include SGW (Serving GateWay), PGW (PDN GateWay), MSC (Mobile Switching Center), HLR (Home Location Register), MME (Mobile Mobility Entity) and HSS (Home Subscriber Server). May be.

Further, the network according to the present invention may include an Internet network. The Internet network refers to a common open communication network, that is, a public network, in which information is exchanged according to the TCP/IP protocol. Through this network, the user device 100, the data server 200, and the unmanned aerial vehicle 300 interwork with each other to configure the augmented reality system according to the present invention.

The user device 100 describes a mobile communication terminal as a representative example, but the terminal is not limited to a mobile communication terminal, and is used in various terminals such as all information communication devices, multimedia terminals, wired terminals, fixed terminals, and IP (Internet Protocol) terminals. Can be applied. For example, the terminal is a mobile terminal having various mobile communication specifications such as a mobile phone, a portable multimedia player (PMP), a mobile internet device (MID), a smart phone, a tablet PC, a phablet PC, and an information communication device. It can be used advantageously when As another example, the user device 100 may be a controller dedicated to the unmanned aerial vehicle 300. As another example, the user device 100 may be a notebook computer, a personal computer, or a workstation.

The data server 200 is an entity existing on a network, and serves as a web server, a database server, and an application server.

Unmanned Aerial Vehicle (UAV: 300) refers to an aircraft that does not have a human pilot on board, and is referred to as a so-called drone. The unmanned aerial vehicle 300 may fly by remote piloted or autonomously fly in an automatic or semi-auto-piloted form according to a pre-programmed route.

In an embodiment of the present invention, the unmanned aerial vehicle 300 collects spatial information of a specific place. The spatial information refers to various pieces of information including images collected by the unmanned aerial vehicle 300 for a specific space.

The data server 200 stores a virtual model that can be synthesized with the spatial information obtained by the unmanned aerial vehicle 300. Here, the virtual model refers to shape data that expresses a shape to be implemented in augmented reality as a three-dimensional visual design having a volume. These virtual models include 3D models and computerized mock-ups.

The user device 100 may manipulate the unmanned aerial vehicle 300 by generating a command related to the flight of the unmanned aerial vehicle 300. In addition, the user device 100 may receive spatial information including an image captured by the unmanned aerial vehicle 300. In addition, the user device 300 may receive a virtual model from the data server 200 and synthesize the virtual model with spatial information to construct an augmented reality image.

Next, a configuration of a user device for providing augmented reality according to an embodiment of the present invention will be described. 2 is a block diagram illustrating a configuration of a user device for providing augmented reality according to an embodiment of the present invention. Referring to FIG. 2, the user device 100 includes a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

The communication unit 110 is for communicating with the data server 200 and the unmanned aerial vehicle 300 through a network. The communication unit 110 may include a radio frequency (RF) transmitter Tx for up-converting and amplifying a frequency of a transmitted signal, and an RF receiver Rx for low-noise amplifying and down-converting a received signal. Further, the communication unit 110 may include a modem that modulates a transmitted signal and demodulates a received signal. For example, the communication unit 110 may receive an image captured by the unmanned aerial vehicle 300.

The input unit 120 receives a user's key manipulation for controlling the user device 100, generates an input signal, and transmits the input signal to the controller 150. The input unit 120 may include various keys for controlling the user device 100. When the display unit 130 is formed of a touch screen, the input unit 120 may perform functions of various keys on the display unit 130, and when all functions can be performed only with the touch screen, the input unit 120 will be omitted. May be.

The display unit 130 visually provides a menu, input data, function setting information, and various other information of the user device 100 to the user. The display unit 130 performs a function of outputting a screen such as a boot screen, a standby screen, a menu screen, and the like of the user device 100. In particular, the display unit 130 performs a function of outputting a real space photographed by the camera unit 120, a virtual object selected by a flat user, and the like on a screen according to an embodiment of the present invention. The display unit 130 may be formed of a liquid crystal display (LCD), organic light emitting diodes (OLED), active matrix organic light emitting diodes (AMOLEDs), or the like. Meanwhile, the display unit 130 may be implemented as a touch screen. In this case, the display unit 130 includes a touch sensor. The touch sensor detects a user's touch input. The touch sensor may be composed of a touch sensing sensor such as a capacitive overlay, a pressure type, a resistive overlay, or an infrared beam, or may be composed of a pressure sensor. . In addition to the above sensors, all kinds of sensor devices capable of sensing contact or pressure of an object may be used as the touch sensor of the present invention. The touch sensor detects a user's touch input, generates a detection signal, and transmits it to the controller 150. In particular, when the display unit 130 is formed of a touch screen, some or all of the functions of the input unit 120 may be performed through the display unit 130.

The storage unit 140 stores programs and data necessary for the operation of the user device 100. In particular, the storage unit 140 may store user data generated according to the use of the user device 100. For example, the storage unit 140 may store an image received from the unmanned aerial vehicle 300. Each type of data stored in the storage unit 140 may be deleted, changed, or added according to a user's manipulation.

The controller 150 may control the overall operation of the user device 100 and a signal flow between the internal blocks 110 to 140 of the user device 100, and perform a data processing function of processing data. The control unit 150 may be a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), or the like. The operation of the control unit 150 will be described in more detail below.

Next, the data server 200 according to an embodiment of the present invention will be described in more detail. 3 is a block diagram illustrating the configuration of a data server according to an embodiment of the present invention. Referring to FIG. 3, the data server 200 includes a communication module 210, a storage module 220 and a control module 230.

The communication module 210 is for communicating with the user device 100 and the unmanned aerial vehicle 300 through a network. The communication module 210 may include a modem that modulates a transmitted signal and demodulates a received signal.

The storage module 220 serves to store programs and data required for the operation of the data server 200. In particular, the storage module 220 is an area in which user data generated according to the use of the data server 200 is stored. The storage module 220 stores a virtual model. The virtual model refers to shape data that expresses a shape to be implemented in augmented reality in a three-dimensional visual design with a volume. These virtual models include 3D models and computerized mock-ups. Each type of data stored in the storage module 220 may be deleted, changed, or added according to a user's manipulation.

The control module 230 controls the overall operation of the data server 200 and a signal flow between internal blocks of the data server 200, and may perform a data processing function of processing data. The control module 230 may be a central processing unit (CPU), a digital signal processor (DSP), or the like. The operation of the control module 230 will be described in more detail below.

Next, a description will be given of the unmanned aerial vehicle 300 according to an embodiment of the present invention. 4 is a block diagram illustrating a configuration of an unmanned aerial vehicle according to an embodiment of the present invention. 4, the unmanned aerial vehicle 300 according to an embodiment of the present invention includes an aviation communication unit 310, a flight unit 320, a sensor unit 330, a camera unit 340, an aviation storage unit 350, and It includes an aviation control unit 360.

The aviation communication unit 310 is for communication with the user device 100 and the data server 200. The aviation communication unit 310 may include a radio frequency (RF) transmitter (Tx) for up-converting and amplifying a frequency of a signal to be transmitted, and an RF receiver (Rx) for low-noise amplification and down-converting a received signal. have. In addition, the aviation communication unit 310 may include a modem that modulates the transmitted signal and demodulates the received signal.

The flight unit 320 is for generating lift so that the unmanned aerial vehicle 300 can fly. The flight unit 320 includes a wing, a propeller, a motor, and the like. The flight unit 320 allows the unmanned aerial vehicle 300 to fly in a direction and speed according to the control of the aviation control unit 360. In particular, the flight unit 320 may adjust the direction of the wing and the rotational speed of the propeller so as to fly according to the control of the aviation control unit 360.

The sensor unit 330 is for measuring the current position (latitude, longitude, altitude) and posture (yaw, roll, pitch) of the unmanned aerial vehicle 300. The sensor unit 330 includes a GPS signal receiver for receiving a GPS signal from a GPS satellite, and a plurality of sensors such as a gyro sensor and an acceleration sensor for measuring the attitude of the unmanned aerial vehicle 300. The sensor unit 330 measures the current position and posture using the GPS signal and sensor values measured by the sensors, and provides the measured current position and posture to the aerial control unit 360.

The camera unit 340 is for photographing an image. The camera unit 340 includes a lens, an image sensor, and a converter. In addition, a predetermined filter or the like may be further included as a configuration of the camera unit 340, and mechanically, a lens, an image sensor, and a converter are mounted in a housing including an actuator, and a driver for driving such an actuator And the like may be included in the camera unit 340. The lens causes visible light incident on the camera unit 340 to be focused on the image sensor. The image sensor is an integrated circuit photoelectric conversion device using a semiconductor device manufacturing technology. The image sensor may be, for example, a charge-coupled device (CCD) image sensor or a complementary metal-oxide semiconductor (CMOS) image sensor. Such an image sensor detects visible light and outputs an analog image signal, which is an analog signal constituting an image. Then, the converter converts the analog image signal into a digital image signal, which is a digital signal constituting the image, and transmits it to the control module 280.

In addition, the camera unit 340 may include a depth sensor or a 3D sensor. The 3D sensor is a sensor for obtaining 3D coordinates for each pixel of an image in a non-contact method. The camera unit 340 may generate an image through an image sensor and simultaneously detect location information (3D coordinates) corresponding to each pixel of an image captured through the 3D sensor. do. Accordingly, through the 3D sensor, the distance from the reference point of the camera unit 340 of the pixel of each image, for example, the focus or main point to the object can be identified. The depth sensor or 3D sensor can use various types of sensors using laser, infrared, visible light, and the like. These 3D sensors are laser type 3D scanners using any one of TOP (Time of Flight), phase-shift and Online Waveform Analysis, laser type 3D scanners using optical trigonometry, and optics using white light or modulated light. Method 3D scanner, Handheld Real Time PHOTO, optical 3D scanner, optical method using Pattern Projection or Line Scanning, laser system full body scanner, photo scanner using photogrammetry, Kinect Fusion A real time scanner to be used may be exemplified.

The aviation storage unit 350 is for storing various types of data required for the operation of the unmanned aerial vehicle 300, applications, and various types of data generated according to the operation of the unmanned aerial vehicle 300. The aviation storage unit 350 may be a memory or the like. In particular, the storage module 270 selectively stores an operating system (OS) for booting and operation of the unmanned aerial vehicle 300, and an application for generating a coordinate system according to an embodiment of the present invention. Each type of data stored in the aviation storage unit 350 may be deleted, changed, or added according to a user's manipulation.

The flight control unit 360 may control the overall operation of the unmanned aerial vehicle 300 and signal flow between internal blocks of the unmanned aerial vehicle 300 and perform a data processing function of processing data. The aerial control unit 360 may be a central processing unit (CPU), an application processor, a graphic processing unit (GPU), or the like. The operation of the flight control unit 360 will be described in more detail below.

Next, a method for providing augmented reality using an unmanned aerial vehicle according to an embodiment of the present invention will be described. 5 is a flowchart illustrating a method for providing augmented reality using an unmanned aerial vehicle according to an embodiment of the present invention.

Referring to FIG. 5, the user device 100 and the control unit 150 receive a user's manipulation through the input unit 120 or the display unit 130, and transmit a control command for the unmanned aerial vehicle 300 in step S110 to the communication unit 110. ) Through the unmanned aerial vehicle 300. Here, the control command may include a reference point (RP) and a flight trajectory (FO) set based on the reference point (RP).

The flight control unit 360 of the unmanned aerial vehicle 300 may receive it through the aviation communication unit 310. Then, the aviation control unit 360 controls the flight unit 320 in step S120 according to the received control command to move the unmanned aerial vehicle 300 above the corresponding reference point RP, and then the reference point Fly above the reference point (RP) in the flight trajectory (FO) set according to the control command based on (RP). In this way, the aviation control unit 360 of the unmanned aerial vehicle 300 controls the flight unit 320 so that the unmanned aerial vehicle 300 is flying above the reference point (RP) to the flight orbit (FO) set based on the reference point (RP). When flying, the aerial control unit 360 controls the camera unit 340 to face the reference point RP regardless of the posture of the unmanned aerial vehicle 300.

Then, the flight control unit 360 collects spatial information through the sensor unit 330 and the camera unit 340 in step S130. Here, the spatial information includes image information and sensor information. In addition, the image information is an image captured by the aerial control unit 360 through the camera unit 340 and an actual object corresponding to each pixel of the image captured from a predetermined reference point (eg, focus or main point) of the camera unit 340. Includes the distance to. In addition, the sensor information corresponds to the time point at which each image is captured, and the location information (eg, GPS coordinates) of the unmanned aerial vehicle 300 obtained through the sensor unit 330 at that time point, and the unmanned aerial vehicle 300 and the unmanned aerial vehicle It includes posture information (yaw, roll, pitch) of the camera unit 340 of 300. Here, the posture information of the camera unit 340 may be a photographing angle.

Next, the flight control unit 360 transmits the spatial information to the user device 100 through the flight communication unit 310 in step S140.

Then, the control unit 150 of the user device 100 sets a 3D coordinate system using the spatial information received through the communication unit 110 in step S150, and derives a reference plane based on the 3D coordinate system. Here, the three-dimensional coordinate system means an arbitrary three-dimensional coordinate calculated based on spatial information and sensor information secured by means of photographing, recognition, and measurement. In addition, the reference plane is an x-y plane in the 3D coordinate system, and refers to the base plane at the lowest position in the captured image.

Subsequently, the controller 150 accesses the data server 200 through the communication unit 110 in step S160 and loads a virtual model to be synthesized in the real environment. More specifically, when the control unit 150 connects to the data server 200 through the communication unit 110, the control module 230 of the data server 200 retrieves a list of virtual models stored in the storage module 220. After extraction, the list of virtual models may be provided to the user device 200 in the form of a thumbnail through the communication module 210. The control unit 150 may display a list of virtual models through the display unit 140. Accordingly, when a user who browses the list of virtual models selects any one virtual model, the controller 150 requests the corresponding virtual model through the communication unit 110. Then, the control module 230 of the data server 200 provides the corresponding virtual model to the user device 100 through the communication module 210.

Next, the controller 150 may synthesize a virtual model based on the reference plane derived (S150) prior to the image captured by the unmanned aerial vehicle 300 in step S170 and display it through the display unit 140.

As described above, according to the present invention, a 3D coordinate system is generated using spatial information, a reference plane is derived based on the 3D coordinate system, and a virtual model is synthesized on the derived reference plane. Then, a method of generating a 3D coordinate system using spatial information will be described in more detail. 6 is a flowchart illustrating a method of generating a 3D coordinate system according to an embodiment of the present invention. 7 to 9 are diagrams for explaining a method of generating a 3D coordinate system according to an embodiment of the present invention.

The unmanned aerial vehicle 300 flies from the user to the shooting point that the user wants to implement in augmented reality. In addition, the photographing direction of the camera is directed toward the photographing point, not the flight direction of the unmanned aerial vehicle 300. Therefore, the unmanned aerial vehicle 300 will fly over the surrounding area of the photographing point. As shown in FIG. 7, the unmanned aerial vehicle 300 may fly in a flight orbit (FO) drawing a circle around a tree existing in real space, or may fly according to another flight orbit. In addition, the unmanned aerial vehicle 300 may fly over the surrounding air based on a virtual model implemented in augmented reality rather than an object that exists in a real space. Apart from the posture of the unmanned aerial vehicle 300 while the unmanned aerial vehicle 300 is flying according to a predetermined flight path (FO), the camera unit 340 of the unmanned aerial vehicle 300 always points toward the reference point.

According to the present invention, in order to synthesize a virtual model as described above, a 3D coordinate system must be created, and for this, a plane of real space must be detected. However, it is not possible to estimate all areas of the real space, but artificial structures made by man are mostly flat, and even if they are not structures, the space is reconstructed into a flat environment through the results of civil works. Therefore, the present invention creates a plane and a three-dimensional coordinate system by utilizing these environmental factors. To this end, the controller 150 assumes that a pixel whose pixel value changes within a predetermined range in the captured image among the spatial information collected by the unmanned aerial vehicle 300 is a plane, and extracts a plane of the actual environment. In addition, the control unit 150 generates a three-dimensional coordinate system including a virtual plane parallel to the extracted plane.

Referring to FIG. 6, the control unit 150 includes attitude information of the unmanned aerial vehicle 300 and attitude information of the camera unit 340 for the same area including at least a reference point RP of an actual space from the spatial information in step S210. At least one of them detects a plurality of different images.

For example, referring to FIG. 7, as described above, the unmanned aerial vehicle 300 has a reference point (FO) set based on a reference point RP according to a control command received from the user device 100. RP) fly over the air. In this way, when the unmanned aerial vehicle 300 flies above the reference point (RP) in the flight trajectory (FO) set based on the reference point (RP), the camera unit 340 is irrelevant to the posture of the unmanned aerial vehicle (300). It aims at the reference point (RP). The image captured in this state is also different because the shooting angle is different for the same real object including the reference point RP, as shown in (C), (D) and (E) of FIG. 8. That is, images (c), (d) and (e) of FIG. 8 are at least one of the attitude information of the unmanned aerial vehicle 300 and the attitude information of the camera unit 340 for the same area including the reference point RP. Is different. Therefore, the controller 150 uses the spatial information received from the unmanned aerial vehicle 300 for the same area including the reference point RP as shown in images (c), (d) and (e) of FIG. 8. At least one of the posture information of 300) and the posture information of the camera unit 340 may detect a plurality of images different from each other.

Next, the controller 150 specifies a plurality of feature points corresponding to the same plane in an actual space from each of the plurality of images detected in step S220. In a more detailed description of the step S220, the controller 150 first sets a reference value, which is a criterion for determining a feature point according to a pixel value of an image. This reference value may be an average of the rate of change of the pixel values of the entire pixel array. Among the pixel values of an image, there is a point where the change in pixel values between two adjacent pixels is greater than the average of the rate of change of the pixel values in the entire pixel array. The control unit 150 selects a point where the pixel value change between two adjacent pixels is greater than the average of the rate of change of the pixel value of the entire pixel array, that is, the pixel whose pixel value rapidly changes to a boundary point between separate and separate planes. Can be identified. In addition, among the pixel values of the image, there is a point where the change between two adjacent pixels is smaller than the average of the rate of change of the pixel values of the entire pixel array. The control unit 150 may determine points in which the pixel value change is smaller than the average of the change rate of the pixel values of the entire pixel array, that is, pixels whose pixel value gradually changes, as the same plane. Accordingly, in order to detect a plane, the control unit 150 specifies a plurality of pixels whose pixel values are smaller than the average of the rate of change of the pixel values of the entire pixel array as a plurality of feature points. For example, B, C, B', C', B", C" of Figs. 8(c), (d) and (e) may be characteristic points.

Meanwhile, the controller 150 may correct a reference value, which is a criterion for determining a feature point, according to at least one of a geographic factor and an environmental factor. In other words, in areas where similar colors are mostly present, such as a gray building forest or a blue forest, the average of the pixel value change rate may not be used as a reference value, but the reference value may be corrected. In addition, the control unit 150 uses the distance from a predetermined reference point (eg, focus or main point) of the camera unit 340 to an actual object corresponding to each pixel of the image information, corresponding to the same plane. The feature points of can be extracted.

Next, in step S230, the controller 150 determines a plurality of planes by obtaining a feature point vector, which is a relative distance and angle between the feature points from the plurality of feature points, and generates a 3D coordinate system based on the specified plurality of planes.

That is, as the photographing angle changes, the distance and angle in the actual space do not change, but the feature point vector, which is the relative distance and angle between the feature points, changes on the two-dimensional arrangement of the photographed results. A plane can be configured through the rate of change of two or more feature point vectors existing on the plane. This is to use the point where two straight lines, one of the crystal conditions of the plane, must meet or be parallel to each other.

In this way, the generated 3D coordinate system is not information displayed to real users, but rather sets the reference plane so that the user can install the virtual model, adjusts the size of the virtual model in proportion to the distance, or adjusts the arrangement angle of the virtual model. It is set virtually to judge when adjusting. As described above, the set three-dimensional coordinate system CO is shown in FIG. 9. Accordingly, the control unit 150 of the user device 100 can derive the reference plane PL from the three-dimensional coordinate system CO as shown in FIG. 9, and synthesize a virtual model on this reference plane. have.

Meanwhile, some or all of the configurations and functions of the data server 200 described above may be implemented by being distributed to at least one of the user device 100 and the unmanned aerial vehicle 300. For example, the virtual model stored in the storage module 220 of the data server 200 may be stored in the storage unit 140 of the user device 100 or in the aviation storage unit 350 of the unmanned aerial vehicle 300. have.

Meanwhile, the method according to the embodiment of the present invention described above may be implemented in the form of a program that can be read through various computer means and recorded on a computer-readable recording medium. Here, the recording medium may include a program command, a data file, a data structure, or the like alone or in combination. The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in computer software. For example, the recording medium includes magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic-optical media such as floptical disks ( magneto-optical media), and hardware devices specially configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions may include not only a machine language such as that produced by a compiler but also a high-level language that can be executed by a computer using an interpreter or the like. These hardware devices may be configured to operate as one or more software modules to perform the operation of the present invention, and vice versa.

The present invention has been described above using several preferred embodiments, but these embodiments are illustrative and not limiting. As such, those of ordinary skill in the art to which the present invention pertains will understand that various changes and modifications can be made according to the equivalence theory without departing from the spirit of the present invention and the scope of the rights presented in the appended claims.

100: user device 110: communication unit
120: input unit 130: display unit
140: storage unit 150: control unit
200: data server 210: communication module
220: storage module 230: control module
300: unmanned aerial vehicle 310: Ministry of Aviation and Communication
320: flight unit 330: sensor unit
340: camera unit 350: aviation storage unit
360: flight control unit

Claims (6)

In an apparatus for providing augmented reality using an unmanned aerial vehicle,
A display unit for displaying a screen;
A communication unit for communication with an unmanned aerial vehicle including a camera unit and communication with a data server; And
Transmitting a control command including a reference point and a flight trajectory set based on the reference point to the unmanned aerial vehicle through the communication unit,
Through the communication unit, the unmanned aerial vehicle receives spatial information including an image photographed while the unmanned aerial vehicle flies in a flight trajectory set based on the reference point,
Set up a three-dimensional coordinate system using spatial information,
After deriving the reference plane from the set 3D coordinate system,
By synthesizing a virtual model on the derived reference plane
Includes; a control unit for controlling to be displayed through the display unit,
The control unit
Detecting a plurality of images in which at least one of the attitude information of the unmanned aerial vehicle and the attitude information of the camera unit are different from the spatial information,
Specifying a plurality of feature points in each of the plurality of detected images,
A plurality of planes are specified by obtaining a feature point vector that is a relative distance and angle between feature points from the plurality of feature points,
Characterized in that generating the three-dimensional coordinate system based on a plurality of specified planes
A device for providing augmented reality. The method of claim 1,
The above spatial information is
Image information including an image captured by the camera unit of the unmanned aerial vehicle and a distance from a predetermined reference point of the camera unit to an actual object corresponding to each pixel of the image captured by the camera unit; And
Comprising: sensor information corresponding to a point in time at which the image is captured and including position information of the unmanned aerial vehicle obtained through the sensor unit of the unmanned aerial vehicle at the point in time, attitude information of the unmanned aerial vehicle, and attitude information of the camera unit; Characterized by
A device for providing augmented reality. delete The method of claim 1,
The control unit
Characterized in that a plurality of pixels having a smaller change in pixel value than an average of a rate of change of pixel values of the entire pixel array of the image are specified as a plurality of feature points
A device for providing augmented reality. In a method for providing augmented reality using an unmanned aerial vehicle,
Transmitting a control command including a reference point and a flight trajectory set based on the reference point to an unmanned aerial vehicle including a camera unit;
Receiving spatial information including an image captured by the unmanned aerial vehicle while flying in a flight trajectory set based on the reference point from the unmanned aerial vehicle;
Setting a 3D coordinate system using the spatial information and deriving a reference plane from the set 3D coordinate system; And
Comprising and displaying a virtual model on the derived reference plane; Including,
The step of deriving the reference plane is
Detecting a plurality of images in which at least one of the attitude information of the unmanned aerial vehicle and the attitude information of the camera unit are different from the spatial information;
Specifying a plurality of feature points in each of the plurality of detected images;
Specifying a plurality of planes by obtaining a feature point vector, which is a relative distance and angle between feature points from the plurality of feature points;
And generating the three-dimensional coordinate system based on the specified plurality of planes.
A method for providing augmented reality. A computer-readable recording medium on which a program for performing the method for providing augmented reality according to claim 5 is recorded.

Images