KR101987090B1 - Method for controlling dron and augmented reality sightseeing system therefor - Google Patents

Abstract

The present invention relates to a method of controlling a drones and an augmented reality sightseeing system for the same, and more particularly, to a main control apparatus for displaying an image photographed by a drone on a screen and recognizing and outputting a gesture of a user, A hold button for controlling movement and stop of the drone, a synchronization button for synchronizing movement of the user with the movement of the dron, a rising button for commanding the rise of the dron, A user interface device configured by an auxiliary control device including a descent button and transmitting control information for controlling the dron by combining a gesture of a user recognized by the main control device and a button input of the auxiliary control device; And an experience control device for receiving the control information from the user interface device and transmitting the received control information to the drones to control the flying of the drones.

Description

TECHNICAL FIELD [0001] The present invention relates to a method for controlling a drones,

The present invention relates to a method of controlling a drones, and more particularly, to a method of controlling a dron by synchronizing a movement of a user with a movement of a dron and an augmented reality sightseeing system therefor.

With the development of information and communication technologies, currently available mobile devices are configured to perform various functions as well as a simple call function. Examples of such various functions include web browsing, chatting, watching TV, navigating, playing games, shooting pictures or videos through cameras, audio storage, playing music files through a speaker system, and displaying images or video have.

To this end, mobile device manufacturers are working hard to develop hardware and software to provide more complex and diverse functions.

In addition, recently introduced smart phones are equipped with a variety of complex application software such as mobile banking, mobile coupons, transportation cards, games, and security.

In addition, currently available smart phones are equipped with a high-performance processor and a high-resolution touch screen screen, thereby enabling use of various multimedia functions and applications.

Particularly, in recent years, interest in wearable devices that can be worn on the body has been increasing, and the demands of users have been diversified accordingly.

The wearable device may be a portable information device that provides a variety of computing technologies and sensing technologies while providing comfort and activity in terms of ergonomics as a computing device wearable on the wearer's body.

For example, in recent years, Samsung Electronics introduced a wristwatch-type Galaxy Gear, which provides not only a clock function but also a communication function like a mobile communication terminal.

The Drone has a pronounced "low-pitched sound". Here, a variety of meanings are derived and are used to mean, for example, "a bass" or "a bass produced by a musical instrument". In recent years, however, it has been mainly used to refer to a small unmanned aerial vehicle (UAV).

Recently, a dron has been equipped with a function to keep the balance of the gas using a plurality of small propellers (rotors) mounted and to photograph a surrounding situation by mounting a camera or the like.

Conventional drones have been mainly used for military purposes such as unmanned reconnaissance, combat support, and recently, they have also been used for commercial purposes such as delivery of goods.

However, until now, the use of the drone has been focused on simple image acquisition, and the user has provided a very limited variety of experiences that can be encountered through the drones, so that the utilization of the unmanned aerial vehicle is not maximized.

Therefore, it is required to develop a system capable of maximizing the user's flight experience by using an unmanned aerial vehicle.

Accordingly, a conventional flight experience system and method using an unmanned aerial vehicle are disclosed in Japanese Patent No. 10-1569454.

However, it is quite difficult for a user to steer the unmanned aerial vehicle to move to a specific position. While the spread of small unmanned aerial vehicles is spreading, there is a high possibility that various inconveniences and accidents are caused by the inexperienced user.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problems and it is an object of the present invention to provide a method for controlling a dron which can be easily controlled by a user who is not accustomed to drone control, have.

Also, it is an object of the present invention to provide a drone control device for an augmented reality drone experience system, in which a user who is not familiar with drone control can intuitively and easily control a drone in an augmented reality flight environment.

According to an embodiment of the present invention, the augmented reality sightseeing system using a drone according to the present invention displays an image photographed by a drone on a screen, displays a gesture of a user, A hold button interlocked with the main control device for controlling the movement and stop of the drone, a synchronization button for synchronizing movement of the user with the movement of the dron, And a lowering button for commanding a lowering of the drones, wherein the gantry of the user recognized by the main piloting device is combined with the button input of the auxiliary piloting device to control the drones A user interface device for transmitting control information; And an experience control device for receiving the control information from the user interface device and transmitting the received control information to the drones to control the flying of the drones.

Preferably, the main steering apparatus applies a scale factor between the movement of the user and the movement of the drones.

Preferably, the scale factor is set by double-clicking a hold button of the sub-control device, and the set value is displayed on a screen of the main control device.

Advantageously, said scale factor is limited to the acceleration and deceleration limits of said drones.

Preferably, the main steering apparatus includes an auto scale factor function for automatically setting the scale factor with reference to a topographic image.

Advantageously, the main steering apparatus displays a warning alarm on the screen when the user is close to the compartment in a limited indoor space.

Preferably, the main steering apparatus is an apparatus for measuring a speed, a direction, a gravity, and an acceleration of a moving object, and is any one of a wearable display apparatus and smart phone having an IMU (Inertial Measurement Unit) sensor.

Preferably, the threshold value is set in the IMU sensor of the main steering apparatus, and the drone is stopped for a velocity lower than the set threshold value, and a low-pass (LPF) Filter is applied to prevent the drilling operation of the drone.

Preferably, the auxiliary steering apparatus includes a joystick, a keypad, a dome switch, a touch pad (static / static), a touch screen, a jog wheel, a jog switch, A mouse, a stylus pen, and a touch pen, as shown in FIG.

According to another embodiment of the present invention, a method of controlling a drone according to the present invention includes: a main control device for displaying an image photographed by a drone on a screen, recognizing and outputting a gesture of a user; A hold button for controlling the movement and stop of the drone, a synchronization button for synchronizing movement of the user with the movement of the dron, a rising button for commanding the rise of the dron, And a control unit for controlling the drones by combining the gesture of the user recognized by the main control unit and the button input of the auxiliary control unit ; And controlling the flight of the drones by transmitting the control information received from the experience control device to the drones.

Preferably, in the step of transmitting the control information for controlling the drones, the user interface device releases the hold button while the user sits holding the hold button, presses the hold button three times in succession , When the down button is pressed, control information for instructing the lowering of the drones is transmitted, the user releases his / her hand from the hold button while holding the hold button, presses the hold button twice consecutively, When the user depresses the posture of the auxiliary pilot while keeping the synchronization button depressed, the control unit transmits control information for instructing the upward movement of the drones to be forward, backward, leftward, rightward, And transmits control information indicating either one of them.

Preferably, in the step of transmitting the control information for controlling the drones, the user interface device instructs the hovering (stop flight) of the drones when the user is in a standby state without pressing the hold button And transmits control information for instructing to maintain the hovering state of the drones when the user moves without pressing the hold button. When the user presses the hold button while moving, the movement of the drones And transmits control information indicating a continuous movement of the drones when the user stops in a state in which the hold button is not pressed, and when the user presses and releases the hold button in the stop state The controller transmits control information indicating stop of the drone.

Preferably, in the step of transmitting the control information for controlling the drones, the user interface device controls, when the user advances while pressing the hold button, a control for instructing advancement of the drones in the direction of the user's eyes When the user moves to the right while pressing the hold button, the control information for instructing movement of the drones is transmitted to the right of the direction of the user's eyes, and when the user moves backward while holding the hold button , And transmits control information indicating the movement of the drones behind the direction of the user's gaze.

Preferably, in the step of transmitting the control information for controlling the drones, when the user tilts the base of the auxiliary control apparatus in a state where the synchronization button is pressed by the user, Control information indicating the advancement of the drones is transmitted and when the user tilts the base of the auxiliary control apparatus to the right with the synchronization button depressed, control information for instructing movement of the drones to the right of the direction of the user's eyes And transmits control information indicating movement of the drones to the left of the user's sight line direction when the user tilts the base of the auxiliary control apparatus to the left while pressing the synchronization button.

As described above, according to the present invention, by providing a method for controlling the flight of a dron by combining a gesture of a user recognized by the main pilot apparatus and a button input of the auxiliary pilot apparatus, and an augmented reality tour system for the same, A user who is not familiar with the drones can easily control the drones, thereby reducing various inconveniences and accidents caused by the inexperienced user.

1 is a block diagram of an augmented reality sightseeing system using a drone according to an embodiment of the present invention;
FIG. 2 is a block diagram showing a schematic configuration of a drones according to an embodiment of the present invention;
FIG. 3 is a flowchart illustrating an operation of the augmented reality sightseeing system using the drone according to the embodiment of the present invention,
4 is a diagram illustrating an example of a user gesture used in augmented reality sightseeing system using a drone according to an embodiment of the present invention;
5 is a signal flow diagram illustrating a method of providing an enlarged region of an image photographed by a drone at a high image quality according to an embodiment of the present invention,
6 is a diagram for explaining a method of setting a magnified area by a user in the embodiment of the present invention,
7 is a diagram for explaining a user's enlargement operation and area operation using a mouse in the embodiment of the present invention, and Fig.
FIGS. 8 to 10 are diagrams for explaining a method of controlling the drone by combining a movement of a user and a button input.

It is noted that the technical terms used herein are used only to describe specific embodiments and are not intended to limit the invention. It is also to be understood that the technical terms used herein are to be interpreted in a sense generally understood by a person skilled in the art to which the present invention belongs, Should not be construed to mean, or be interpreted in an excessively reduced sense. Further, when a technical term used herein is an erroneous technical term that does not accurately express the spirit of the present invention, it should be understood that technical terms that can be understood by a person skilled in the art are replaced. In addition, the general terms used in the present invention should be interpreted according to a predefined or prior context, and should not be construed as being excessively reduced.

Also, the singular forms "as used herein include plural referents unless the context clearly dictates otherwise. In the present application, the term "comprising" or "comprising" or the like should not be construed as necessarily including the various elements or steps described in the specification, Or may be further comprised of additional components or steps.

Further, the suffix "module" and "part" for components used in the present specification are given or mixed in consideration of ease of specification, and do not have their own meaning or role.

Furthermore, terms including ordinals such as first, second, etc. used in this specification can be used to describe various elements, but the elements should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

In the augmented reality sightseeing system using the drone according to the embodiment of the present invention, the dron converts the original image of 4K resolution into the image of 1080p resolution in consideration of the transmission capacity of the image modem of the experience control device, i.e., Lt; / RTI > In the following, "360 degree image" means the converted 1080p resolution image.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like reference numerals refer to like or similar elements throughout the several views, and redundant description thereof will be omitted.

In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail. It is to be noted that the accompanying drawings are only for the purpose of facilitating understanding of the present invention, and should not be construed as limiting the scope of the present invention with reference to the accompanying drawings.

1 is a configuration diagram of an augmented reality sightseeing system using a drone according to an embodiment of the present invention.

1, the augmented reality sightseeing system using a dron according to the present invention is an augmented reality sightseeing system using a dron connected to at least one dron 300 through a network 200. The augmented reality sightseeing system includes a curved display device 110 A user interface device 120, a dome projector 130, an experience control device 140, and a storage device 150, and the like.

The curved display device 110 displays images and flight information received from the drones 300.

The user interface device 120 recognizes the gesture of the user and remotely controls the drones 300 through the network 200. For this purpose, the user interface device 120 is a device for receiving at least one of a command, selection, and information of a user. In particular, the user interface device 120 recognizes and outputs a gesture of a user as a camera, And may include a plurality of input keys and function keys for setting.

For example, the user interface device 120 may be a notebook, a tablet PC, a joystick, a key pad, a dome switch, a touch pad (static / static), a touch screen A touch screen, a jog wheel, a jog switch, a jog shuttle, a mouse, a stylus pen, a touch pen, and the like.

As a modification of the user interface device 120, the user interface device 120 may include a main control device for displaying an image photographed by the drones 300 on the screen, recognizing and outputting a gesture of the user, A hold button for controlling the movement and stop of the drones 300 in order to control the drones 300 in a more detailed manner in cooperation with the device, and an auxiliary device including a synchronization button for synchronizing the movement of the user with the movement of the drones 300. [ And a control device. Here, a scale factor is applied between the movement of the user and the movement of the drones 300, so that a large movement of the drones 300 can be reflected even to a small movement of the user. The scale factor is set by double-clicking the hold button, and the set value can be displayed on the screen of the main control unit. Of course, since the acceleration and deceleration limits of the drone 300 are set, rapid start-up operations can be prevented in advance.

In addition, the main steering apparatus includes an auto scale factor function for automatically setting a scale factor by referring to a terrain image combined with image, GPS, Digital Terrain Elevation Data (DTED) .

In addition, the main steering device may display a warning alarm (or mini-map) on the screen when the user approaches a compartment such as a wall in a limited indoor space.

The main steering device is a device for measuring the speed, direction, gravity, and acceleration of a moving object and includes a wearable display device and a smart phone having an IMU (Inertial Measurement Unit) sensor such as an accelerometer, a speedometer, a geomachine, and an altimeter The auxiliary steering device may be a joystick, a key pad, a dome switch, a touch pad (static / static), a touch screen, a jog wheel, a jog switch, A jog shuttle, a mouse, a stylus pen, a touch pen, and the like. Here, the threshold value is set for each sensor of the main control unit, so that the drone 300 is stopped for a slow speed lower than a set threshold value, and a low-pass filter (LPF) ) Can be applied to prevent the drill operation of the drone 300.

The dome projector 130 displays images and flight information received from the drones 300 to the user through a dome screen. The user can designate a specific area of the 360-degree image displayed on the dome screen using the user interface device 120, for example, a mouse, and enlarge the designated specific area. Accordingly, the dome projector 130 can transmit coordinate information (for example, latitude, longitude, magnification, enlargement area size, etc.) of a specific area to be enlarged by the user to the experience control device 140.

Also, the dome projector 130 separates the image photographed by the drones 300 into left and right images, warps the separated left and right images according to the curvature characteristics of the dome screen, By displaying the polarized light components orthogonally to each other on the dome screen, a stereoscopic image can be provided to the user.

The experience control device 140 receives the image and flight information from the drones 300 and stores the received image and flight information in the storage device 150 or transmits the received image and flight information to the curved display device 110 and the dome projector 130, And displays it through the curved display device 110 or the dome projector 130. [ The experience control device 140 transmits control information corresponding to the gesture input through the user interface device 120 to the drones 300.

Meanwhile, when receiving coordinate information of a specific area from the dome projector 130, the experience control device 140 according to the present invention requests the image corresponding to the coordinate information of the specific area to the drones 300, The image corresponding to the coordinate information of the received specific region is overlapped with the 360-degree image to display a high-resolution enlarged image through the dome projector 130 can do.

The experience control device 140 sequentially receives the first image and the second image divided by the drones 300, re-synthesizes the divided first and second images, And can be displayed through the display device 110 or the dome projector 130.

For this purpose, the experience control device 140 may include a wired / wireless communication unit 141, an image processing unit 142, a drone control unit 143, and the like.

The wired and wireless communication unit 141 receives the image and the flight information from the drone 300 and transmits the received image and the flight information to the curved display device 110 or the dome projector 130. The wired / wireless communication unit 141 may transmit control information received from the user interface device 120 to the drones 300.

The wired / wireless communication unit 141 may include a wired / wireless communication module. Herein, the wireless Internet technology may be a wireless LAN (WLAN), a Wi-Fi, a wireless broadband (Wibro), a World Interoperability for Microwave Access (HSDPA), IEEE 802.16, Long Term Evolution (LTE), Wireless Mobile Broadband Service (WMBS), and the like. The Short Range Communication Technology may include Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, and the like. The wired communication technology may include USB (Universal Serial Bus) communication, and the like.

The image processing unit 142 converts the image received from the drones 300 through the wired / wireless communication unit 141 into an image suitable for each of the curved display device 110 and the dome projector 130.

In addition, the image processing unit 142 can generate a high-resolution enlarged image by overlapping the 360-degree image with the image corresponding to the coordinate information of the specific area received from the drones 300. In addition, the image processing unit 142 may re-synthesize the first image and the second image sequentially received from the drones 300.

The drones control unit 143 transmits control information corresponding to the gesture inputted through the user interface device 120 to the drones 300. [

In addition, the drones control unit 143 may have a number of functions including a manual operation mode, an automatic takeoff and landing function, an automatic route point flight, a manual stability enhancement flight, and an emergency return. The manual operation mode is a mode that is operated according to the supervisory command of the manager. The mode can be operated at a limited speed and a tilt angle, and urgent action is possible in an unexpected situation. The automatic takeoff and landing function can be automatic landing and landing with only the control switch or GCS (Ground Control System) screen button without manual operation. In the automatic route point flight, it is possible to plan various scenarios by inputting a plurality of route points, to provide the same tourist scenarios by editing and storing the route, and to reduce the work intensity of the manager and to manage the operation smoothly. Manual Stability Enhancement The flight can differentiate between the flight performance limitations of tourists and managers (speed and attitude limitations). The emergency return is a function of automatically returning to a safe position based on the judgment of the dron controller 143 or the operator's judgment in an emergency mode such as a sudden malfunction or a battery low voltage situation.

In addition, the experience control device 140 can display tourist information or shopping information at a location corresponding to the image together with the image.

In addition, the experience control device 140 controls the drones 300 so that the flight range for each course stored in the storage device 150 does not deviate from the flight range of the selected course based on the set map information and the location information of the drones 300 can do. Then, the experience control device 140 can guide the sightseeing information in textures or voices when the local information is displayed.

The storage device 150 stores route information for each course of a sightseeing spot, map information of a tourist spot, tourist information for each spot of a tourist spot, dron information, dron control information corresponding to a gesture, image information received from the drones 300, do.

To this end, the storage device 150 includes a route information DB 151 for storing route information for each course of sightseeing spots, a map information DB 152 for storing map information of tourist spots, tourist information A shopping information DB 155 for storing information of a shopping mall selling special products or souvenirs, a shopping mall DB 155 for storing information of shopping malls selling special products or souvenirs, An image information DB 156 for storing images, a flight information DB 157 for storing images and flight information received from the drone 300, and a drones control information DB 158 for storing dron control information corresponding to gestures, And the like. Here, when the image of the drones 300 is stored in the memory card of the 360-degree omnidirectional camera, the image information DB 156 storing the image received from the drones 300 may be omitted.

The control server 400 monitors the flight so that the drones 300 are not in a dangerous area and generates control information when the drones 300 are abnormally operated according to the user's operation and transmits them to the drones 300. Thus, the drones 300 may control the flight based on the control information of the control server 400. [ At this time, the control server 400 can receive input information from an operation expert of the drones 300, and can generate control information for operating the drones 300 based on the input information.

FIG. 2 is a block diagram showing a schematic configuration of a drones according to an embodiment of the present invention.

2, the drone 300 includes a 360 degree omnidirectional camera 310, a wireless communication unit 320, a control unit 330, an activation unit 340, a position measurement unit 350, and an orientation measurement unit 360, And the like.

The control unit 330 receives the control information from the experience control device 140 through the wireless communication unit 320 and controls the activation unit 340 that controls the movement of the drones 300 according to the control information, Can fly. At this time, the wireless communication unit 320 may support a communication method according to the network 200. [

The control unit 330 may transmit the image photographed through the 360 degree omnidirectional camera 310 to the experience control device 140 through the wireless communication unit 320 when the drones 300 are flying. At this time, the 360 degree omnidirectional camera 310 is simple because it can capture images of all directions with one camera, and angle adjustment is not necessary. In other words, there is no need to rotate the drone to acquire a specific region image

Meanwhile, when the experience control device 140 requests an image corresponding to coordinate information of a specific region, the controller 330 extracts an image corresponding to the coordinate information of the specific region from the original image , And transmits the image corresponding to the coordinate information of the extracted specific area to the experience control device 140 through the wireless communication unit 320. [

In addition, the control unit 330 divides the photographed image into a first image and a second image using the 360 degree omnidirectional camera 310, sequentially experiences the first image and the second image through the wireless communication unit 320, To the control device 140. FIG.

The control unit 330 includes a gyro sensor, an acceleration sensor, and the like. The control unit 330 determines attitude information based on the sensed values measured through the attitude measuring unit 360 that measures the attitude of the drones 300 according to the control information And transmits it to the experience control device 140 through the silent communication unit 320. [

Further, the controller 330 generates position information on the current position of the drones 300 based on the measured position values measured by the position measuring unit 350 including a Global Positioning System (GPS) unit, 320 to the experience control device 140.

3 is a flowchart illustrating an operation of an augmented reality sightseeing system using a drone according to an embodiment of the present invention, and Fig. 4 is a view illustrating an example of a user gesture used in a augmented reality sightseeing system using a drone according to an embodiment of the present invention .

Referring to FIG. 3, the user selects one of the tour courses he wishes to fly using the user interface device 120. Then, the user interface device 120 transmits information on the selected tour course to the experience control device 140 (S310). These scenic courses are pre-set in a safe area with a lot of scenery and lots of attractions.

The experience control device 140 remotely controls the drones 300 through the network 200 according to previously stored route information corresponding to the selected tour course (S320). The drones 300 are controlled by the drones control unit 143 of the experience control device 140 so that the images and the flight information photographed through the 360 degree omnidirectional camera 310 are controlled by the experience control To the device (140).

The experience control device 140 receives the image and the flight information from the dragon 300 using the 360 degree omnidirectional camera 310 and displays the received image and the flight information through the curved display device 110 S330). The experience control device 140 can store video and flight information in the video information DB 156 and the flight information DB 157, respectively, if necessary.

For example, the curved-surface display device 110 may include a 360-degree 360-degree omnidirectional camera 310 of the drones 300, such as attitude, altitude, heading, latitude, longitude, Additional flight information can be displayed.

In addition, the curved display device 110 processes the original image photographed using the 360-degree omnidirectional camera 310, and outputs a field angle area, converts the MJPG streamer real time image into an MJPG format image It can also be output via a smart device with built-in Ethernet web browser.

In addition, it is possible to provide a separate space for the curved display device 110 so that the user can feel the feeling of proper sightseeing. For example, in the case of an image for Southeast Asia tourism, the atmosphere can be produced around the curved display device 110 by model palm trees, benches, and the like.

In addition, the experience control device 140 can display a hyperlink of a homepage that can expose local information of the region photographed by the drones 300 or purchase special products of the area on the screen of the curved display device 110 . When the user selects a desired hyperlink, the image processing unit 142 of the experience control device 140 refers to the shopping information DB 155 and connects the shop to a shopping mall selling special products or souvenirs, Provide tourist information explaining the area.

In step S330, the experience control device 140 transmits the image and the flight information photographed by the 360 degree omnidirectional camera 310 from the drones 300 to the dome projector 130 as well as the curved display device 110 Can be displayed. The user can designate a specific area of the 360-degree image displayed on the dome screen using the user interface device 120, for example, a mouse, and enlarge the designated specific area. Accordingly, the dome projector 130 can transmit coordinate information of a specific area to be enlarged by the user to the experience control device 140.

The experience control device 140 requests the image corresponding to the coordinate information of the specific region to the drones 300 when receiving the coordinate information of the specific region from the dome projector 130, The image corresponding to the coordinate information of the specific region is received and the image corresponding to the coordinate information of the received specific region is overlapped with the 360 degree image so that a high resolution enlarged image can be displayed through the dome projector 130. [

In addition, the experience control device 140 sequentially receives the first image and the second image, which are divided by the drones 300, and re-synthesizes the received first and second images, It can be displayed through the curved display device 110 or the dome projector 130.

In this state, or before step S320 and step S330, the user can move the drones 300 to his / her desired position according to need. To this end, when the user takes a gesture, the user interface device 120 recognizes the gesture using the built-in camera, and transmits the information about the recognized gesture to the experience control device 140. As described above, the experience control device 140 receives the user gesture through the user interface device 120 (S340).

Thereafter, the experience control device 140 controls the drones 300 through the network 200 remotely according to the control information according to the user gesture (S350). An example of the gesture is shown in Fig.

Referring to FIG. 4, the gesture includes a gesture for controlling the 360-degree omnidirectional camera 310 and a gesture for controlling the dragon 300, which is a flying object.

In the gesture for controlling the 360 degree omnidirectional camera 310, there is a gesture for enlarging, reducing, rotating, and selecting the camera image. Such a gesture is possible using both fingers. For example, when the two fingers are separated from each other, the camera image is enlarged, and when both fingers are closer, the camera image is reduced. When one hand is shaken to the left and right, the line of sight of the 360 degree omnidirectional camera 310 is rotated.

Gestures that manipulate the flying body drone (300) include gestures such as forward / backward, left / right, up / down, and stop. For example, if one hand holds a fist and the other hand is raised, the drones (300) move forward, one hand holds a fist, and the other hand shakes downward.

When one hand holds a fist and the other hand shakes to the left, the drone (300) turns to the left, one hand holds the fist, and the other hand shakes to the right.

When the thumb of both hands is raised, the drones 300 are lifted, and when both hands are lowered, the drones 300 are lowered.

Further, when the both hands are raised, the drones 300 are stopped.

These various gestures can be added, deleted, and modified as needed.

5 is a signal flow diagram illustrating a method of providing an enlarged region of an image photographed by a drone at a high image quality according to an embodiment of the present invention.

Referring to FIG. 5, the dome projector 130 determines whether there is a selection input for a specific area to be enlarged in a 360-degree image displayed on the dome screen from the user through the user interface device 120 (S510) The dome projector 130 calculates an enlargement point of a specific area to be enlarged by the user in operation S520, (E.g., latitude and longitude) of a specific area to be enlarged by the user to the experience control device 140 (S530).

When receiving the coordinate information of the specific area from the dome projector 130, the experience control device 140 requests the drones 300 for the image corresponding to the coordinate information of the specific area (S540).

The drone 300 extracts an image corresponding to the coordinate information of the specific area from the original image in response to the video request of the experience control device 140 at step S550 and displays the image corresponding to the coordinate information of the extracted specific area in the experience control device 140 (S560). That is, the drone 300 cuts out the image corresponding to the coordinate information of the specific region in the original image of 4k resolution and transmits it to the experience control device 140. For example, when the user desires to enlarge a specific area by 2 times, the drone 300 reduces the area ratio of the image cut out from the original image to 1/2 and transmits it to the experience control device 140, The drone 300 transmits the area ratio of the image cut out from the original image to the experience control device 140 as it is.

The experience control device 140 generates an enlarged image of high resolution by overlapping the image corresponding to the coordinate information of the specific area received from the drones 300 with the 360 degrees image in step S570, (Step S580).

Finally, the dome projector 130 displays an enlarged image of high resolution received from the experience control device 140 (S590).

6 is a diagram for explaining a method of setting a magnified area by a user in an embodiment of the present invention.

Referring to FIG. 6, the user can adjust the enlarged area by clicking the wheel of the mouse. For example, if the user clicks the wheel of the mouse once, the first stage region 610 is set. When the user clicks the wheel of the mouse twice, a two-step area 620 is set which is larger than the first-step area 610. When the user clicks the wheel of the mouse three times, a three-step area 630 that is larger than the two-step area 620 is set. When the user clicks the wheel of the mouse four times, the first stage region 610 is set again. As described above, the dome projector 130 according to the embodiment of the present invention has a loop of area setting according to the click of the wheel of the user's mouse. Also, the right mouse button can release all image enlargement functions. Although the embodiment of the present invention has been described by merely dividing into three stages for convenience of explanation, the present invention is not limited thereto, and the loop of area setting may include at least two stages.

FIG. 7 is a diagram for explaining an enlarging operation and a region manipulation of a user using a mouse in the embodiment of the present invention. FIG.

Referring to FIG. 7A, the user can adjust the magnification of the enlarged region by scrolling the wheel of the mouse from a minimum of 1 times to a maximum of 4 times. When the user intends to enlarge the enlarged area by one time, the dragon 300 transmits the image cut out from the original image directly to the experience control device 140. When the user intends to enlarge the enlarged area twice, If the user desires to enlarge the enlarged area by three times, the drones 300 can reduce the area of the image cut out from the original image, If the user wants to enlarge the enlarged area by 4 times, the drones 300 reduce the area ratio of the image cut out from the original image to 1/4 To the experience control device (140).

Referring to FIG. 7 (b), the user can designate an area in three steps by clicking the mouse wheel. For example, when the user clicks the wheel of the mouse once, the area designation 1 corresponding to the first-stage area 610 is designated. When the user clicks the wheel of the mouse twice, When the user clicks the wheel of the mouse 3 times, the area designation 3 corresponding to the 3-step area 630 larger than the 2-step area 620 is designated .

FIGS. 8 to 10 are diagrams for explaining a method of controlling the drone by combining a movement of a user and a button input.

When the user depresses the hold button while holding down the hold button, depresses the hold button three times in succession, and depresses the down button, the drones 300 are lowered. Thereafter, the drones 300 are stopped by pressing the hold button once when the user stops all operations.

When the user holds the hold button and releases the hold button halfway, the hold button is pressed twice continuously, and when the raise button is pressed, the drones 300 are raised. Thereafter, the drones 300 are stopped by pressing the hold button once when the user stops all operations.

When the user tilts the posture of the auxiliary control device while holding the synchronization button, the attitude of the drones 300 and the posture of the assistant control device are synchronized with each other based on the front screen of the drones 300 to move forward, backward, , And a right turn.

More specifically, referring to FIG. 8, if the user is in a standby state without pressing the hold button, the drones 300 perform hovering (stop flight) (S810).

Subsequently, when the user moves without pressing the hold button, the drones 300 maintain the hovering state (S820)

Subsequently, when the user presses the hold button while moving, the drones 300 move (S830).

Subsequently, when the user releases his or her hand from the hold button while moving, the drones 300 continue to move (S840).

Subsequently, when the user stops in a state in which the hold button is not pressed, the drones 300 continue to move (S850). At this time, the user is freely movable, and the movement of the user does not affect the movement of the drones 300.

Finally, when the user presses and releases the hold button in the stopped state, the drone 300 stops because the user's last state is stopped (S860).

Referring to FIG. 9, when the user advances while holding the hold button, the drones 300 advance in the direction of the user's eyes (S910).

When the user holds the hold button and moves to the right, the drones 300 move to the right of the user's sight line (S920).

When the user moves backward while holding the hold button, the drones 300 move backward in the direction of the user's sight (S930).

Referring to FIG. 10, when the user tilts the gas of the auxiliary control apparatus while pushing the synchronization button, the drones 300 advance in the direction of the user's sight (S1010).

When the user tilts the gas of the auxiliary control device to the right while pressing the synchronization button, the drones 300 move to the right of the user's sight line (S1020).

When the user tilts the gas of the auxiliary control device to the left while pressing the synchronization button, the drones 300 move to the left in the direction of the user's eyes (S1030).

The above-described method can be implemented by various means. For example, embodiments of the present invention may be implemented by hardware, firmware, software, or a combination thereof.

In the case of hardware implementation, the method according to embodiments of the present invention may be implemented in one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs) , FPGAs (Field Programmable Gate Arrays), processors, controllers, microcontrollers, and microprocessors.

In the case of an implementation by firmware or software, the method according to embodiments of the present invention may be implemented in the form of a module, a procedure or a function for performing the functions or operations described above. The software code can be stored in a memory unit and driven by the processor. The memory unit may be located inside or outside the processor, and may exchange data with the processor by various well-known means.

As described above, in the embodiment of the present invention, it is not necessary to adjust the angle of the camera in the dron by applying a 360-degree omnidirectional camera capable of shooting a 360-degree image to the dron, so that the weight of the dron can be reduced, It is possible to save the energy because the rotation movement is unnecessary, so that the flight time extension effect is obtained.

Further, in the embodiment of the present invention, tourists can steer the drones with simple gestures.

In addition, in the embodiment of the present invention, tourists can receive images taken while using the drones.

Further, in the embodiment of the present invention, a tourist can view a high-quality image through the curved display device.

In addition, in the embodiment of the present invention, a plurality of tourists can see a tourist area where a person can not go in real time through a dome projector.

Further, in the embodiment of the present invention, the area to be enlarged on the screen of the dome projector can be viewed with high quality.

In addition, in the embodiment of the present invention, it is possible to turn around the place where the individual wants to see in real time using the goggles.

In addition, in the embodiment of the present invention, it is also possible to provide the recorded image and the flight information together so that the extended service (for example, state notification, special product shopping, terrain information description, voice guidance, And screen text display, etc.).

The embodiments disclosed herein have been described with reference to the accompanying drawings. Thus, the embodiments shown in the drawings are not to be construed as limiting, and those skilled in the art will understand that the present invention can be combined with each other, and when combined, some of the components may be omitted.

Here, terms and words used in the present specification and claims should not be construed as limited to ordinary or dictionary meanings, but should be construed as meaning and concept consistent with the technical idea disclosed in the present specification.

Therefore, the embodiments described in the present specification and the drawings are only exemplary embodiments of the present invention, and are not intended to represent all of the technical ideas disclosed in the present specification, so that various equivalents It should be understood that water and variations may be present.

110: Surface display device 120: User interface device
130: Dome projector 140: Experience control device
141: wired / wireless communication unit 142:
143: Drone control unit 150: Storage device
151: Path information DB 152: Map information DB
153: sightseeing DB 154: drone information DB
155 shopping information DB 156 video information DB
157: flight information DB 158: drone control information DB
200: Network 300: Drones
400: control server

Claims (14)

A main control device for displaying an image photographed by the drone on a screen and recognizing and outputting a gesture of a user; a hold button interlocked with the main control device and for controlling the movement and stop of the dragon; An auxiliary control device including a synchronization button for synchronizing the movement of the drones, a rising button for commanding the rising of the drones, and a lowering button for commanding the lowering of the drones, A user interface device for transmitting control information for controlling the drones by combining a gesture of a user and a button input of the auxiliary control device; And
An experience control device connected to the network, receiving the control information from the user interface device, and transmitting the received control information to the drones to control the flying of the drones; Lt; / RTI >
Wherein the main steering apparatus applies a scale factor between the movement of the user and the movement of the drones,
The auxiliary steering device includes a joystick, a key pad, a dome switch, a touch pad (static pressure / static electricity), a touch screen, a jog wheel, a jog switch, a jog shuttle ), A mouse, a stylus pen, and a touch pen. [3] The augmented reality sightseeing system according to claim 1, delete The method according to claim 1,
Wherein the scale factor is set by double-clicking a hold button of the auxiliary control device, and a set value is displayed on a screen of the main control device. The method of claim 3,
Wherein the scale factor is limited to acceleration and deceleration limits of the drones. The method according to claim 1,
Wherein the main control device includes an auto scale factor function for automatically setting the scale factor with reference to a topographic image. The method according to claim 1,
Wherein the main control apparatus displays a warning alarm on the screen when the user approaches the compartment in a limited indoor space. The method according to claim 1,
Wherein the main steering apparatus is any one of a wearable display device and a smart phone equipped with an IMU (Inertial Measurement Unit) sensor for measuring the speed, direction, gravity and acceleration of a moving object, Tourism system. 8. The method of claim 7,
A threshold value is set in the IMU sensor of the main control unit so that the drone stops for a velocity lower than a set threshold value and a low-pass filter (LPF) is applied to a velocity exceeding a set threshold value Thereby preventing the drag operation of the drones. delete A main control device for displaying an image photographed by the drone on a screen and recognizing and outputting a gesture of a user; a hold button interlocked with the main control device and for controlling the movement and stop of the dragon; A user interface device composed of a synchronization button for synchronizing the movement of the drones, an ascending button for instructing the ascending of the drones, and a descending button for instructing descent of the drones, Transmitting control information for controlling the drones by combining a gesture of a user recognized by the control unit and a button input of the auxiliary control unit; And
And controlling the flight of the drones by transmitting the control information received from the experience control device to the drones,
In the step of transmitting the control information for controlling the drones,
Wherein the user interface device comprises:
When the user depresses the hold button while depressing the hold button, releases the hold button, presses the hold button three consecutive times, transmits control information instructing the descent of the dron when depressing the down button,
When the user depresses the hold button while the hold button is pressed, the hold button is released, the hold button is pressed twice continuously, and when the up button is pressed, control information indicating the rise of the dron is transmitted,
When the user tilts the attitude of the auxiliary steering apparatus while holding the synchronization button, transmits control information indicating one of forward, backward, left / right movement, left turn and right turn. Way. delete 11. The method of claim 10, wherein in transmitting the control information for controlling the drones,
Wherein the user interface device comprises:
When the user is in a standby state without pressing the hold button, transmits control information indicating hovering (stop flying) of the drones,
When the user moves without pressing the hold button, transmits control information instructing to maintain the hovering state of the drones,
When the user presses the hold button while moving, transmits control information indicating movement of the drones,
When the user stops in a state in which the hold button is not pressed, transmits control information indicating continuous movement of the drones,
Wherein when the user presses and releases the hold button in a stopped state, control information indicating stop of the drone is transmitted. 11. The method of claim 10, wherein in transmitting the control information for controlling the drones,
Wherein the user interface device comprises:
When the user advances while pressing the hold button, transmits control information instructing advancement of the drones in the direction of the user's eyes,
When the user presses the hold button and moves to the right, transmits control information indicating the movement of the drones to the right of the user's sight line direction,
When the user moves backward while holding the hold button, transmits control information indicating movement of the dron behind the direction of the user's eyes. 11. The method of claim 10, wherein in transmitting the control information for controlling the drones,
Wherein the user interface device comprises:
When the user tilts the base of the auxiliary pilot apparatus in a state in which the synchronization button is depressed, transmits control information instructing advancement of the drone in the direction of the user's sight,
When the user tilts the base of the auxiliary pilot apparatus to the right while holding the synchronization button, transmits control information instructing movement of the drones to the right of the user's sight line direction,
Wherein when the user tilts the gas of the auxiliary pilot apparatus to the left while pressing the synchronization button, control information for instructing movement of the drone to the left of the user's sight line direction is transmitted.

Images