KR102498572B1 - System and Method for providing AR contensts using dron - Google Patents

Abstract

The present invention relates to a content provision system and method based on augmented reality (AR) using drones, which display virtual images together with captured images of swarm drones conducting a drone show. The system comprises: a drone control unit that controls at least one drone according to a preset drone show program; a video receiving unit that receives one or more drone shooting videos controlled by the drone control unit; a virtual image extraction unit that extracts at least one preset virtual image based on an object image including an actual drone generated by a drone shooting video received from the video receiving unit, in the form of an object image including a drone or according to a drone control signal from the drone control unit; and an AR implementation unit that merges the virtual image extracted from the virtual image extraction unit with the object image including the drone generated by the video receiving unit to provide AR content to a user terminal. The system can build realistic content using AR technology, can extend the play time of a performance that combines media art technology, and thus can provide the realistic content by providing performance information and the like to users watching the drone show by utilizing AR technology. The present invention enables performance planning that is well coordinated overall and accurately reflects a planner's intention.

Description

System and method for providing augmented reality-based content using drones {System and Method for providing AR contensts using dron}

The present invention relates to a system and method for providing contents based on augmented reality using drones, and more particularly, to contents based on augmented reality using drones that display virtual images together with images taken by swarm drones performing drone shows. It relates to a provision system and method.

In general, drones have been used for military purposes such as aerial photography or aerial strikes through manual flight by remote control of a ground control system or autonomous flight using a global positioning system (GPS). Due to the expansion of supply and the development of control technology using mobile communication terminals such as smartphones, it is not only in the field of high-altitude video or photo shooting and delivery, weather information collection, etc. It is used in various fields such as agricultural use for spraying pesticides for pest control, fire suppression in high-rise buildings or deep mountains, and performances for various performances in the sky.

Drones are usually divided into quadcopters (4 pieces), hexacopters (6 pieces) and octocopters (8 pieces) according to the number of propellers. It consists of landing legs. Currently, such drones can stay in one place for a long time in the air, and are used for various purposes such as filming, farming, rescue support, etc. because their location is easy to control.

In particular, as disclosed in Korean Laid-Open Patent Publication No. 2014-0174827, drones are equipped with a lighting device to be used as a lighting means using cluster drones, and lighting means are installed on drones to perform light shows in the air with cluster drones. It is also used for demonstration purposes.

In the case of performance drones, a plurality of drones are being introduced to implement various performances required for performances, and while emitting LED beams or laser beams using a beam projector, etc. By generating effect materials, light is diffusely reflected by them to produce various performances.

On the other hand, conventional viewing-type drone content lacks sympathy from viewers, and legal regulatory issues regarding short event time and night flight approval have emerged.

KR 10-2235407 B1 KR 10-2014-0174827 A

The present invention is derived from such a technical background, and builds realistic content using augmented reality (AR) technology and utilizes augmented reality (AR) technology to extend the play time of performances converged with media art technology. Its purpose is to provide an augmented reality-based content providing system and method using drones that can provide realistic content by providing performance information to users watching a drone show.

In addition, it is possible to compose the overall story of the drone show performance more structuredly by using media art, and it is possible to produce its own sound source and video, so it harmonizes well overall and is based on augmented reality using drones that can accurately reflect the intention of the planner. It is intended to provide a content providing system and method.

The present invention for achieving the above object includes the following configuration.

That is, the content providing system based on augmented reality using a drone according to an embodiment of the present invention includes a drone control unit that controls at least one drone according to a preset drone show program, and images captured by one or more drones controlled by the drone control unit. An image receiving unit that receives a drone image received from the image receiving unit, based on an object image including a real drone generated by the drone image, or a virtual image preset according to a drone control signal from the drone control unit. It includes a virtual image extractor for extracting at least one and an augmented reality implementer for merging the virtual image extracted from the virtual image extractor with an object image including a drone generated by the image receiver and providing augmented reality content to a user terminal. .

On the other hand, in the augmented reality-based content providing method using drones performed in an augmented reality-based content providing system using drones, the drone control step of controlling at least one drone according to a preset drone show program, the drone An image reception step of receiving an image captured by one or more drones controlled in the control step, an object image including a drone based on an object image including a real drone generated by the drone captured image received in the image reception step, or the image form of the object including the drone. A virtual image extraction step of extracting at least one of preset virtual images according to a drone control signal from the drone control unit and merging the virtual image extracted in the virtual image extraction step with the object image including the drone generated in the image reception step and an augmented reality implementation step of providing augmented reality content to a user terminal.

According to the present invention, it is possible to build immersive content using augmented reality (AR) technology and extend the play time of a performance converged with media art technology, so that users who watch drone shows using augmented reality (AR) technology The effect of providing an augmented reality-based content providing system and method using a drone capable of providing realistic content by providing performance information to a person is derived.

In addition, it is possible to compose the overall story of the drone show performance more structuredly by using media art, and it is possible to produce its own sound source and video, so it harmonizes well overall and is based on augmented reality using drones that can accurately reflect the intention of the planner. It is possible to provide a content providing system and method of.

1 is an exemplary diagram for explaining the concept of a content providing system based on augmented reality using a drone according to an embodiment of the present invention.
2 is a block diagram showing the configuration of a content providing system based on augmented reality using a drone according to an embodiment of the present invention.
3 is an exemplary view for explaining a drone control technology of an augmented reality-based content providing system according to an embodiment of the present invention.
4 is a flowchart of a method for providing content based on augmented reality using a drone according to an embodiment of the present invention.

It should be noted that technical terms used in the present invention are only used to describe specific embodiments and are not intended to limit the present invention. In addition, technical terms used in the present invention should be interpreted in terms commonly understood by those of ordinary skill in the art to which the present invention belongs, unless specifically defined otherwise in the present invention, and are excessively inclusive. It should not be interpreted in a positive sense or in an excessively reduced sense.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is an exemplary diagram for explaining the concept of a content providing system based on augmented reality using a drone according to an embodiment of the present invention.

An augmented reality-based content providing system using a drone according to an embodiment builds immersive content using augmented reality (AR) technology and can extend the play time of a performance converged with media art technology. That is, it is possible to provide immersive content by providing performance information to users watching a drone show using augmented reality (AR) technology.

In addition to this, it is possible to compose the overall story of the drone show performance more structuredly by utilizing media art. In addition, since it is possible to produce its own sound source and video, it is possible to plan a performance that harmonizes well with the whole and accurately reflects the intention of the planner.

The augmented reality-based content providing system 10 using drones according to an embodiment may include a Wi-Fi router, an RTK antenna, and a telemetry antenna to control swarm drones.

2 is a block diagram showing the configuration of a content providing system based on augmented reality using a drone according to an embodiment of the present invention.

Augmented reality-based content providing system 10 according to an embodiment recognizes clustered drones, converges and augments information about a drone art show with media art technology, and provides augmented reality-based realistic drone content. can

As shown in FIG. 2, the augmented reality-based content providing system according to an embodiment includes a communication unit 110, a drone control unit 120, an image receiving unit 130, a virtual image extraction unit 140, an augmented reality implementation unit 150, It includes an art content providing unit 160, a concept selection unit 170, a virtual image management unit 180, and an art content management unit 190.

The communication unit 110 communicates with any internal component or at least one external terminal through a wired/wireless communication network. Here, wireless Internet technologies include Wireless LAN (WLAN), DLNA (Digital Living Network Alliance), Wireless Broadband (Wibro), WiMAX (World Interoperability for Microwave Access: Wimax), HSDPA (High Speed Downlink Packet Access) ), High Speed Uplink Packet Access (HSUPA), IEEE 802.16, Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), Wireless Mobile Broadband Service (WMBS), etc. , and the communication unit 110 transmits and receives data according to at least one wireless Internet technology within a range including Internet technologies not listed above.

In addition, short-range communication technologies include Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, and Near Field Communication (NFC). , Ultra Sound Communication (USC), Visible Light Communication (VLC), Wi-Fi, Wi-Fi Direct, and the like may be included. In addition, wired communication technologies may include power line communication (PLC), USB communication, Ethernet, serial communication, optical/coaxial cables, and the like.

At this time, at least one arbitrary external terminal may be one of the drone 20, the user terminal 30, and the media art server 40.

The user terminal 30 includes a smart phone, a portable terminal, a mobile terminal, a foldable terminal, a personal digital assistant (PDA), and a portable multimedia device (PMP). Player terminal, telematics terminal, navigation terminal, personal computer, notebook computer, slate PC, tablet PC, ultrabook, wearable device Device, for example, watch type terminal (Smartwatch), glass type terminal (Smart Glass), HMD (Head Mounted Display), etc.), Wibro terminal, IPTV (Internet Protocol Television) terminal, smart TV, digital broadcasting It can be applied to various terminals such as terminals, audio video navigation (AVN) terminals, audio/video (A/V) systems, flexible terminals, digital signage devices, and the like.

In one embodiment, the user terminal 30 is a terminal possessed by a user who wants to use the content provided by the augmented reality-based content providing system 10 .

The drone controller 120 controls at least one drone according to a preset drone show program. In other words, it is possible to control the flight of several drones at the same time. The drone control unit 120 may be implemented to include a mobile ground control station GCS (Ground Control Solution) software. The drone control unit 120 may perform command and control (C2) and control functions of drones and transport equipment.

3 is an exemplary view for explaining a drone control technology of an augmented reality-based content providing system according to an embodiment of the present invention.

In one embodiment, the drone control unit 120 may transmit a control signal to the drones 20 for swarm flight via at least one of a wifi router, an RTK antenna, and a telemetry antenna.

The image receiving unit 130 receives one or more drone captured images controlled by the drone controller 120 .

In one embodiment, the image receiving unit 130 receives an image captured by a camera installed in the user terminal 30 .

The virtual image extractor 140 responds to the form of an object image including a drone or a drone control signal from the drone controller 120 based on an object image including a real drone generated by the drone captured image received from the image receiver 130. Accordingly, at least one of preset virtual images is extracted.

The virtual image extraction unit 140 may receive an image captured by the user terminal 30 possessed by a user watching a drone show, and extract a virtual image according to a shape of a drone object image generated by the captured image.

For example, the virtual image extraction unit 140 may determine the shape of cluster drones put into a drone show according to drone control by the drone controller 120 . That is, the drone control unit 120 can determine the drone show program and determine the form of cluster drones according to the current drone show program in real time.

In the case of applying augmented reality content to drone show footage in real time, the method of detecting drones by receiving drone footage is not limited, but cluster drones at the present time according to the control signal according to the drone show program provided to the drones. form can be accurately identified. However, it is not limited thereto.

As another example, the virtual image extractor 140 may extract a virtual image matched according to the shape of a group of drones based on a drone captured image received from the user terminal 30 to the image receiver 130 . The virtual image extractor 140 may detect drones from a drone-captured image in which the cluster drones are captured, and determine the form of the cluster drones.

Even if augmented reality content is not used by using drone show footage in real time, users can receive augmented reality content by applying cluster drone footage previously filmed or received through various routes.

For example, it is possible to provide realistic content in which a user interacts with a drone art show through an augmented reality (AR) application.

In addition, in one embodiment, drone art show (The star of universe' constellation of the universe (augmented reality (AR) technology, media art technology, swarm flight technology fusion), media art (outer space terrestrial image), star (drone light) It can perform functions such as realization of sky and sky) and augmented reality (realistic constellation information and experience using AR).

In another embodiment, 'The Only contents' may provide only one future content.

In addition, the augmented reality-based content providing system 10 may provide one-of-a-kind realistic content for those with a special story. For example, a dead daughter can be expressed through a drone art show, and an opportunity to communicate through augmented reality technology can be prepared.

The augmented reality implementer 150 merges the virtual image extracted by the virtual image extractor 140 with the object image including the drone generated by the image receiver 130 to provide augmented reality content to the user terminal 30 .

The augmented reality implementation unit 150 may recognize and learn an image using Open Source Computer Vision (Open CV), a programming library for real-time computer vision.

When the augmented reality implementation unit 150 recognizes the shape of drones clustered with Open CV and machine learns, it can provide an augmented reality signal to the user terminal 30 so that an image or video can be output.

In one embodiment, the augmented reality implementer 150 may be implemented on an augmented reality-based content providing app using a drone driven in the user terminal 30 .

Accordingly, while conventional drone shows are only watched passively, augmented reality (AR) technology can be applied to provide augmented reality content in the form of immersive performances that users can participate in.

In one aspect of the present invention, the art content provider 160 provides synchronized media art content to match the drone show program controlled by the drone controller 120. The art content provider 160 may receive media art content matched with various types of drone show programs from the media art server 40 . However, it is not limited thereto.

The art content manager 190 receives media art content provided by the art content provider 160 and edits the uploaded media art content. The art content provider 190 may be implemented to interwork with the media art server 40 . The art content management unit 190 may edit and manage media art in various formats and methods.

The art content provider 160 can configure storytelling in detail using media art, extend the overall story of the drone show performance to extend the time from around 10 minutes to around 30 minutes, and produce its own sound source and video However, since it is possible to edit the information, it is possible to provide well-coordinated performance planning information as a whole.

The concept selector 170 provides at least two or more virtual image concepts to the user terminal 30 through an augmented reality-based content providing application, and selects one of the provided virtual image concepts.

The virtual image extraction unit 140 extracts a virtual image suitable for the virtual image concept selected by the concept selection unit 170 .

The virtual image concept is to provide various types of augmented reality content, and to provide different augmented reality content to different user terminals 30 for the same swarm drone type.

Even if the same type of swarm drone is photographed, when the virtual image concept selected by user a and the virtual image concept selected by user b are different, different augmented reality content may be provided.

For example, a character included in a virtual image is displayed differently, or an image of an animal or object related to a constellation is provided as a virtual image to a user terminal that has taken a swarm drone flying in the form of a constellation according to a selected concept, or it is related to a constellation. Myth-related images may be provided as virtual images.

Alternatively, the color tone of the virtual image may be provided as concept information, and the color tones of the virtual image may be displayed differently to two or more users who simultaneously photograph the same swarm drone. For example, a user who selects a red-based concept is provided with augmented reality content including virtual images made of red-based colors, and a user who selects a blue-based concept is provided with augmented reality content including virtual images made of blue-based colors. Get real content.

The virtual image management unit 180 receives a virtual image suitable for the virtual image concept and edits the uploaded virtual image.

The virtual image management unit 180 matches and uploads virtual images to be used for augmented reality from a manager who operates and manages an augmented reality-based content providing app or a show provider who manages drone shows according to an embodiment, matching the shape of drones for swarming. , it provides a user interface that can be edited.

The virtual image manager 180 may receive and add, edit, or delete concept information for each virtual image. In other words, it is easy to add and edit virtual images, so it is possible to provide augmented reality contents that combine drone shows in various forms.

4 is a flowchart of a method for providing content based on augmented reality using a drone according to an embodiment of the present invention.

In the augmented reality-based content providing method using drones performed in the augmented reality-based content providing system using drones, at least one drone is controlled according to a preset drone show program (S400).

In other words, it is possible to control the flight of several drones at the same time. The drone control step may be implemented to include mobile ground control station GCS (Ground Control Solution) software. The drone control stage can perform command and control (C2) and steering functions of drones and transport equipment.

In addition, an image of one or more drones controlled in the drone control step is received (S410).

In one aspect of the present invention, at least two or more virtual image concepts are provided to a user terminal through an augmented reality-based content providing application, and one of the provided virtual image concepts is selected (S420).

A virtual image that extracts at least one of the form of an object image including a drone based on an object image including a real drone generated by the drone captured image received in the image reception step or a preset virtual image according to a drone control signal from the drone control unit. An image is extracted (S430).

In one aspect of the present invention, the virtual image extraction step extracts a virtual image suitable for the virtual image concept selected in the concept selection step.

For example, in the virtual image extraction step, it is possible to determine the shape of cluster drones used in the drone show according to the drone control in the drone control step. That is, it is possible to identify the drone show program from the drone control step and determine the form of cluster drones according to the current drone show program in real time.

In the case of applying augmented reality content to drone show footage in real time, the method of detecting drones by receiving drone footage is not limited, but cluster drones at the present time according to the control signal according to the drone show program provided to the drones. form can be accurately identified. However, it is not limited thereto.

As another example, in the virtual image extraction step, a virtual image matched according to the shape of a group of drones may be extracted based on a drone captured image received from a user terminal. In the virtual image extraction step, drones may be detected from drone images in which cluster drones are captured, and the form of cluster drones may be identified.

Even if augmented reality content is not used by using drone show footage in real time, users can receive augmented reality content by applying cluster drone footage previously filmed or received through various routes.

In addition, in one embodiment, the drone art show (The star of the universe) Constellation of the Universe (Augmented Reality (AR) technology, media art technology, swarm flight technology fusion), media art (outer space terrestrial image), star (drone light) It can perform functions such as realization of sky and sky) and augmented reality (realistic constellation information and experience using AR).

In another embodiment, 'The Only contents' may provide only one future content.

In addition, augmented reality is implemented by merging the virtual image extracted in the virtual image extraction step with the object image including the drone generated in the image reception step to provide augmented reality content to the user terminal (S440).

Accordingly, while conventional drone shows are only watched passively, augmented reality (AR) technology can be applied to provide augmented reality content in the form of immersive performances that users can participate in.

In one aspect of the present invention, media art content synchronized to match the drone show program controlled in the drone control step is further provided (S450).

In addition, the media art content provided by the media art content providing step may be uploaded, and the uploaded media art content may be edited.

In the stage of providing art contents, storytelling can be configured in detail using media art, and the overall story of the drone show performance can be extended to extend the time from around 10 minutes to around 30 minutes, and self-sound and video production and editing can be performed. It is possible to provide well-coordinated performance planning information as a whole.

In a further aspect of the present invention, a virtual image suitable for a virtual image concept may be uploaded, and the uploaded virtual image may be edited.

The virtual image concept is to provide various types of augmented reality content, and to provide different augmented reality content to different user terminals 30 for the same swarm drone type.

Even if the same type of swarm drone is photographed, when the virtual image concept selected by user a and the virtual image concept selected by user b are different, different augmented reality content may be provided.

For example, a character included in a virtual image is displayed differently, or an image of an animal or object related to a constellation is provided as a virtual image to a user terminal that has taken a swarm drone flying in the form of a constellation according to a selected concept, or it is related to a constellation. Myth-related images may be provided as virtual images.

Alternatively, the color tone of the virtual image may be provided as concept information, and the color tones of the virtual image may be displayed differently to two or more users who simultaneously photograph the same swarm drone. For example, a user who selects a red-based concept is provided with augmented reality content including virtual images made of red-based colors, and a user who selects a blue-based concept is provided with augmented reality content including virtual images made of blue-based colors. Get real content.

A user interface capable of matching, uploading, and editing virtual images to be used in augmented reality from a manager who operates and manages an augmented reality-based content providing app or a show provider who manages drone shows according to an embodiment can provide.

In addition, concept information for each virtual image can be received and added, edited, or deleted. In other words, it is easy to add and edit virtual images, so it is possible to provide augmented reality contents incorporating drone shows in various forms.

The above method may be implemented as an application or implemented in the form of program instructions that can be executed through various computer components and recorded on a computer readable recording medium. The computer readable recording medium may include program instructions, data files, data structures, etc. alone or in combination.

Program instructions recorded on the computer-readable recording medium may be those specially designed and configured for the present invention, or those known and usable to those skilled in the art of computer software.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like.

Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes generated by a compiler. The hardware device may be configured to act as one or more software modules to perform processing according to the present invention and vice versa.

Although the above has been described with reference to embodiments, it will be understood that those skilled in the art can variously modify and change the present invention without departing from the spirit and scope of the present invention described in the claims below. You will be able to.

10: content providing system 110: communication department
120: drone control unit 130: video receiving unit
140: virtual image extraction unit 150: augmented reality implementation unit
160: Art content providing unit 170: Concept selection unit
180: virtual image management unit 190: art content management unit

Claims (10)

a drone control unit controlling one or more drones according to a preset drone show program;
an image receiver for receiving one or more drone-captured images controlled by the drone control unit;
A virtual image that extracts at least one of an object image including a drone based on an object image including a real drone generated by a drone-captured image received from the image receiving unit or a preset virtual image according to a drone control signal from the drone control unit. image extraction unit; and
An augmented reality implementation unit that merges the virtual image extracted by the virtual image extractor with an object image including a drone generated by the image receiver and provides augmented reality content to a user terminal;
A concept selection unit that provides at least two or more virtual image concepts to a user terminal through an augmented reality-based content providing application and selects one of the provided virtual image concepts;
The virtual image extraction unit,
According to the drone control signal, the shape of the group drones put into the drone show is identified, and a virtual image suitable for the virtual image concept selected in the concept selection unit is extracted,
The video receiver,
An augmented reality-based content providing system using drones that receives, from a user terminal, drone-captured images of drones controlled by a drone show program in the drone control unit.
According to claim 1,
An augmented reality-based content providing system using drones, further comprising: an art content providing unit providing synchronized media art content to match the drone show program controlled by the drone control unit.
According to claim 2,
An augmented reality-based content providing system using a drone, further comprising: an art content management unit that receives media art content uploaded by the media art content provider and edits the uploaded media art content.
delete According to claim 1,
A virtual image management unit for uploading a virtual image suitable for a virtual image concept and editing the uploaded virtual image; further comprising a drone-based augmented reality-based content providing system.
In the augmented reality-based content providing method using drones performed in the augmented reality-based content providing system using drones,
A drone control step of controlling one or more drones according to a preset drone show program;
an image reception step of receiving an image of one or more drones controlled in the drone control step;
Extracts at least one of an object image including a drone based on an object image including a real drone generated by the drone-captured image received in the image receiving step or a preset virtual image according to a drone control signal in the drone control step. a virtual image extraction step; and
An augmented reality implementation step of merging the virtual image extracted in the virtual image extraction step with the object image including the drone generated in the image receiving step to provide augmented reality content to a user terminal;
A concept selection step of providing at least two or more virtual image concepts to a user terminal through an augmented reality-based content providing application and selecting one of the provided virtual image concepts;
The virtual image extraction step,
According to the control of the drone, the shape of the group drones put into the drone show is identified, and a virtual image suitable for the virtual image concept selected in the concept selection step is extracted,
The video receiving step,
A method of providing content based on augmented reality using drones, wherein a drone-captured image of drones controlled according to a drone show program is received from a user terminal in the drone control step.
According to claim 6,
An augmented reality-based content providing method using drones, further comprising: providing art content providing synchronized media art content to match the drone show program controlled in the drone control step.
According to claim 7,
An art content management step of uploading media art content provided by the media art content providing step and editing the uploaded media art content; further comprising an augmented reality-based content providing method using drones.
delete According to claim 6,
A virtual image management step of uploading a virtual image suitable for a virtual image concept and editing the uploaded virtual image; further comprising a drone-based augmented reality-based content providing method.

Images