KR20200079857A - System for providing performance contents using augmented reality and Method for providing the same - Google Patents

Abstract

Disclosed are a system for providing performance content using augmented reality to the audience and a method thereof, which are able to provide content by synthesizing an actual performance with an augmented reality image. More specifically, the present invention relates to a system for providing performance content using augmented reality, which comprises a server and a terminal. The server comprises: a user checking unit which checks if a user using the terminal is an audience or not; a user position checking unit which extracts the seat information of the user; and a content storage unit which stores the augmented reality content used for the performance. The terminal includes: a reality image extraction unit which extracts a reality image of the performance venue; a movement information extraction unit which extracts the movement information of the user; an augmented reality synthesizing unit which synthesizes the reality image with the augmented reality content provided from the server, adjusts the size and position of the augmented reality content in accordance with the seat information of the user extracted by the user position checking unit, and adjusts the size and position of the augmented reality content in accordance with the movement of the user extracted by the movement information extraction unit; and a display unit which outputs the synthesized image.

Description

System for providing performance contents using augmented reality and its providing method {System for providing performance contents using augmented reality and Method for providing the same}

The disclosed content relates to a system and a method for providing performance content to the audience using augmented reality, and to provide content by synthesizing a real performance and augmented reality image.

Unless otherwise indicated herein, the content described in this section is not prior art to the claims of this application and is not admitted to be prior art by inclusion in this section.

Recently, with the spread of mobile devices capable of executing and providing various media such as smart phones, methods for providing content to users have been diversified. As these contents lead to an increase in technology demand for interaction between humans and computers from the viewpoint of efficiency and convenience, augmented reality, which is a field of Human Computer Interaction (HCI), has attracted great attention, especially mobile augmented reality ( Interest in Mobile Augmented Reality is growing.

Augmented reality (Augmented Reality), unlike Virtual Reality (Virtual Reality) refers to an environment that is provided to users by fusing images from the real world and images from the virtual world. Generally, the image of the virtual world that appears in augmented reality uses computer graphics, and serves to provide additional information to the image of the real world.

Content that is provided to users using augmented reality in the related art, and extracts user's location information through GPS, GNSS, etc. from a real image projected on the camera of a mobile device, and provides advertisement and navigation information from the extracted location information. In a variety of ways, augmented reality content is utilized to provide services to users.

According to Korean Patent Publication No. 10-1825600, in a mobile terminal, a camera for capturing an image, a screen control means for displaying an image captured through the camera on a screen in real time, and one or more seats from a server via a network. And a wire/wireless communication means receiving seat arrangement information including a number and a location, and location determination means for grasping the current location information of the mobile terminal itself, wherein the location determination means receives the seat number from the user. After determining the position of the seat input from the user on the seat arrangement information, the offline position of the seat corresponding to the seat number entered by the user is determined, and the screen control means displays an image captured through the camera on the screen. On the screen, a predetermined mark is overlaid on the off-line position of the seat corresponding to the seat number entered by the user, and each seat is given a unique tag, and the position determination means is a tag taken by the camera The current position is determined from, and the position determination means determines the relative position from the current position identified from the tag and the position of the seat number input by the user using the matching relationship between the tag and the seat number, and the screen control means Discloses a seat guiding device using augmented reality technology, characterized in that the relative position is displayed overlaid on the screen.

In the Korean Patent Publication No. 10-1825600, augmented reality is used to guide the seat, but augmented reality can be used in more various ways, so the need for research has emerged.

1. Korean Patent Publication No. 10-1825600

We intend to provide augmented reality performance contents that can bring more realism and immersion by integrating real performance and virtual reality.

Disclosed is a system for providing performance content using augmented reality including a server and a terminal, wherein the server comprises: a user identification unit to determine whether a user using the terminal is an audience; a user location identification unit to extract user's seat information; And a content storage unit for storing augmented reality content used for performances, wherein the terminal comprises: a reality image extraction unit for extracting a reality image of a venue, a motion information extraction unit for extracting user motion information, the reality image and the Synthesizing the augmented reality content received from a server, adjusting the size and position of the augmented reality content according to the user's seat information extracted from the user location confirmation unit, and the user's motion extracted from the motion information extraction unit According to an embodiment of the present invention, a system for providing performance content using augmented reality is provided, which includes an augmented reality synthesis unit for adjusting and synthesizing the size and position of augmented reality content and a display unit for outputting the synthesized image.

In addition, the disclosed contents include a first step of confirming whether the user is an audience, a second step of acquiring the seat information of the user, a third step of extracting a reality image of the venue, a fourth step of receiving augmented reality content, Adjusting the location and size of the augmented reality content according to the user's seat information, the fifth step of synthesizing with the real image, the sixth step of outputting the synthesized image to a display, and the augmented reality according to the user's motion information A method for providing performance contents using augmented reality including a seventh step of adjusting the location of the content is presented as an embodiment.

According to the disclosed embodiment, it is possible to improve the immersion of the audience by fusing the actual performance and the virtual reality, and to provide the audience with a large-scale scene that could not be produced in the actual performance.

The effects of the present embodiments are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 is a block diagram of a system for providing performance content using augmented reality according to an embodiment of the disclosed content.
Figure 2 is a flow diagram of a method for providing performance content using augmented reality according to an embodiment of the disclosed content.

Advantages and features of the present invention, and methods for achieving them will be clarified with reference to embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the embodiments allow the disclosure of the present invention to be complete, and common knowledge in the technical field to which the present invention pertains. It is provided to fully inform the holder of the scope of the invention, and the invention is only defined by the scope of the claims. Throughout the specification, the same reference numerals refer to the same components.

In describing embodiments of the present invention, when it is determined that a detailed description of known functions or configurations may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted. In addition, the terms described below are terms defined in consideration of functions in an embodiment of the present invention, which may vary according to a user's or operator's intention or practice. Therefore, the definition should be made based on the contents throughout this specification.

Hereinafter, a preferred embodiment of a system for providing performance content using improved augmented reality will be described with reference to the accompanying FIG. 1.

As shown in FIG. 1, the system for providing performance contents using augmented reality includes a server 10 and a terminal 20, and the server 10 confirms whether a user using the terminal 20 is an audience. It includes a user identification unit 11, a user seat identification unit for extracting the user's seat information, and a content storage unit 13 for storing augmented reality content used in the performance, the terminal 20, the reality of the venue A reality image extracting unit 24 for extracting images, a motion information extracting unit 23 for extracting user's motion information, and the augmented reality contents provided from the server 10 are synthesized, but the user location Adjust the size and position of the augmented reality content according to the user's seat information extracted from the confirmation unit 12, and the size and position of the augmented reality content according to the user's motion extracted from the motion information extraction unit 23 It includes an augmented reality synthesis unit 25 for adjusting and synthesizing, and a display unit 21 for outputting the synthesized image.

In more detail, the server 10 includes a user identification unit 11 for checking user information, a user location identification unit 12 for extracting user's seat information, and content for storing augmented reality content used for performances. It includes a storage unit 13,

The user identification unit 11 checks whether the user of the terminal 20 is an audience of the corresponding performance. By assigning a unique ID to each terminal 20, it is possible to check whether the user is an audience of the corresponding performance, or the user can enter the ID of the user in the terminal 20 to check whether the audience is present. Alternatively, a kind of QR code or a specific marker is assigned to the performance ticket, and the user's terminal 20 recognizes the corresponding QR code or marker, thereby checking whether the audience is present.

The user positioning unit 12 extracts the user's seat information. Extracting the user's seat information is for accurate synthesis of the reality image and augmented reality content, which will be described later, and is used as a variable for correcting a time difference according to the user's location. The size and angle of the venue and actors change according to the user's seat, and the location and size of the augmented reality content can be adjusted according to the changed position. As a method of confirming the location, the user can input his seat information, or by installing a unique NFC chip or beacon for each seat, the user's seat information can be extracted. From an economical point of view, it is desirable for a user to input his/her own seat information, but it would be desirable to use a beacon to provide augmented reality content even when the user has to move a seat or move in a venue.

The content storage unit 13 stores augmented reality content used for performances. The stored content is a virtual reality graphic used for a corresponding performance and may be classified and stored according to a corresponding performance.

The terminal 20 is a separately provided device, which can be distributed to the audience prior to the start of the performance, and installs an application prepared by the organizer of the performance in advance on a smartphone or a device on which a program can be installed. It can be used as (20). In order to add new functions or improve scalability, it would be desirable to install an application on a user's smartphone. The terminal 20 may include a reality image extraction unit 24, a motion information extraction unit 23, an augmented reality synthesis unit 25, and a display unit 21.

The reality image extraction unit 24 extracts the reality image of the venue. In the performance, the appearance of the venue and the actor performing the performance are extracted as images or images. A camera or a camcorder may be used as the reality image extraction unit 24, and preferably, a camera built in a smartphone may be used.

The motion information extraction unit 23 may extract motion information of a user. The terminal 20 may be used in a fixed position, but may be held by a user and used, so it is preferable to adjust the size and position of the augmented reality content according to the user's movement. Therefore, a gyro sensor capable of extracting user's motion information or an acceleration sensor having two or more axes may be used, and it may be desirable to use an acceleration sensor having three or more axes to recognize three-dimensional motion. When using a user's smartphone, a gyro sensor or an acceleration sensor built into the smartphone can be used.

The augmented reality synthesis unit 25 synthesizes the augmented reality content received from the real image and the server 10, but augmented reality according to the user's seat information extracted from the user location confirmation unit 12 The size and position of the content may be adjusted, and the size and position of the augmented reality content may be adjusted according to the user's motion extracted from the motion information extraction unit 23 and synthesized. The synthesis of the real image and the augmented reality content is not limited to one time, but may be continuously performed during the time of providing the augmented reality content. For example, the user's location information and motion information may be measured every 1 ms (milli second), and a real image and augmented reality content may be synthesized every 1 ms according to the measured information.

For accurate synthesis of augmented reality content and reality images, one or more first markers 31 may be included on the stage of the performance. In addition, a marker extraction unit capable of recognizing the first marker 31 may be included in the terminal 20. The first marker 31 may be installed at a fixed position on the stage, and the marker extraction unit recognizes the first marker 31 from the reality image extracted from the reality image extraction unit 24 to identify the location and size. can do.

In addition, a second marker 32 may be installed on at least one of an actor performing a performance and an object used in the performance, and the second marker 32 may be recognized by the marker extraction unit. The first marker 31 and the second marker 32 may be made of a specific pattern that can be identified by the marker extraction unit, and may be made of a beacon or infrared transceiver. As a method of recognizing a marker, when the marker is formed of a specific pattern, the marker is extracted while extracting the real image, and the size and position of the marker can be identified, or, as a beacon, Bluetooth Low Energy (BLE). Using the device, the location of the beacon can be identified with a Bluetooth receiver, and the location of the corresponding marker can be identified using an infrared transceiver. Although only one of the beacon and infrared transceivers may be used, it is preferable to use triangulation by arranging three or more because it is difficult to determine the exact location when only one is used. As described above, if the first marker 31 and the second marker 32 are recognized by the marker extractor 22, when synthesizing the augmented reality content and the real image, it may be synthesized with the correct position and size.

The display unit 21 may output the synthesized image. The display unit 21 may include all screens capable of outputting images, and may use LCDs, LEDs, AMOLEDs, and the like. The display unit 21 may be separately provided in the terminal 20, but when a user's smartphone is used, the smartphone screen may be used as the display unit 21.

The system for providing performance contents using the augmented reality may be continuously provided during the performance time, or may be limitedly used in a specific scene as needed while performing a general performance. For example, in the case of directing a large-scale scene such as a war scene, there is a limitation in the space and manpower of the stage, so it is possible to provide a more immersive and realistic performance content by utilizing the performance content providing system using this augmented reality. .

Hereinafter, a method for providing performance content using augmented reality will be described with reference to FIG. 2.

The method for providing performance content using augmented reality includes: a first step of determining whether a user is an audience; a second step of acquiring seat information of the user; a third step of extracting a reality image of the venue; receiving augmented reality content The fourth step, adjusting the position and size of the augmented reality content according to the user's seat information, the fifth step of synthesizing with the real image, the sixth step of outputting the synthesized image to a display and the user's motion information And adjusting the location of the augmented reality content according to the seventh step.

If you look specifically,

In the first step, it is checked whether the user is an audience. A user who enters the corresponding venue can enter and enter the terminal 20 capable of viewing the augmented reality content, and by assigning a unique ID for each terminal 20, it is possible to check whether the user is an audience of the performance, or The user can check his or her audience by entering his or her ID in the terminal 20. Alternatively, a kind of QR code or a specific marker is assigned to the performance ticket, and the user's terminal 20 recognizes the corresponding QR code or marker, thereby checking whether the audience is present.

In the second step, the user's seat information is acquired. Extracting the user's seat information is for the accurate synthesis of the augmented reality content and the reality image, which will be described later. Since the venue and the actor's size or angle differ depending on the user's seat in the venue, the location and size of the augmented reality content It can be used as a variable to adjust. As a method of checking the location, the user can input his/her seat information into the terminal 20, or by installing a unique NFC chip or beacon for each seat, the user's seat information can be extracted. From an economical point of view, it is desirable for a user to input his/her own seat information, but it would be desirable to use a beacon to provide augmented reality content even when the user has to move a seat or move in a venue.

In the third step, the reality image of the venue is extracted. In the performance, the appearance of the venue and the actor performing the performance are extracted as images or images. As a method of extracting a real image, a camera or a camcorder may be used, and preferably, a camera built in a smartphone may be used.

The 3-1 step of extracting the marker information from the marker 30 installed on at least one of the stage and the actor while extracting the reality image may be further included. The marker is for accurate synthesis of augmented reality content and a real image, and can be made of a specific pattern that can be identified, and can be made of a beacon or an infrared transceiver. As a method of extracting marker information, when the marker 30 is formed of a specific pattern, the marker 30 can be extracted together while extracting the real image to identify the size and position of the marker 30, Alternatively, a Bluetooth low energy (BLE) device may be used as a beacon, and the location of the beacon may be identified using a Bluetooth receiver, and the location of the corresponding marker 30 may be identified using an infrared transceiver. Although only one of the beacon and infrared transceivers may be used, it is preferable to use triangulation by arranging three or more because it is difficult to determine the exact location when only one is used.

The marker 30 may be installed in one or more fixed positions on the stage, or may be installed in at least one of an actor performing a performance and an object used in the performance.

In the fourth step, augmented reality content is received. The received augmented reality content is a virtual reality graphic used for the performance, and can be received separately for each performance.

In the fifth step, the location and size of the augmented reality content are adjusted according to the user's seat information, and the result is synthesized with the real image. Looking in more detail, the real image and the augmented reality content provided from the server 10 are synthesized, but the size and position of the augmented reality content can be adjusted according to the extracted user's seat information, and the above-described marker information Including, it is possible to improve the accuracy of the size and location of the augmented reality content.

In the sixth step, the synthesized image is output to the display. The display may include all screens capable of outputting an image, and LCD, LED, AMOLED, and the like can be used. The display may be included in the separately provided terminal 20, but when a user's smartphone is used, the smartphone screen may be used as the display.

In the seventh step, the location of the augmented reality content is adjusted according to the user's motion information. The terminal 20 that provides the augmented reality content can be installed at a fixed location, and can also be lifted and used by the user, so that the user's motion information is extracted to adjust the augmented reality content to correspond to the user's motion. Needs to be. As a method of extracting the user's motion information, a gyro sensor may be used, or an acceleration sensor of two or more axes may be used, and it is preferable to use an acceleration sensor of three or more axes to recognize 3D motion. When using a user's smartphone, a gyro sensor or an acceleration sensor built into the smartphone can be used.

The method for providing performance content using the augmented reality may be continuously provided during the performance time, or may be limitedly used in a specific scene as needed while performing a general performance. For example, in the case of directing a large-scale scene such as a war scene, there is a limitation in the space and manpower of the stage, so it is possible to provide a more immersive and realistic performance content by utilizing the method for providing performance content using this augmented reality.

The disclosed content is only an example, and can be variously changed by a person having ordinary skill in the art without departing from the gist of the claims claimed in the claims, so the scope of protection of the disclosed content is specified above. It is not limited to the embodiment.

10: Server
11: User identification
12: User location confirmation
13: content storage
20: terminal
21: display unit
22: marker extraction unit
23: Motion information extraction unit
24: reality image extraction unit
25: Augmented Reality Synthesis
30: marker
31: 1st marker
32: second marker

Claims (5)

In the system for providing performance content using augmented reality including a server and a terminal,
The server,
A user identification unit that checks whether a user using the terminal is an audience;
A user location confirmation unit for extracting user's seat information; And
A content storage unit for storing augmented reality content used for performances; Including,
The terminal,
A reality image extraction unit that extracts the reality image of the venue;
A motion information extraction unit for extracting user's motion information;
Synthesizing the real image and the augmented reality content provided from the server, adjusting the size and position of the augmented reality content according to the user's seat information extracted from the user location confirmation unit, and the motion information extraction unit An augmented reality synthesis unit for synthesizing by adjusting the size and position of the augmented reality content according to the extracted user's movement; And
And a display unit for outputting the synthesized image.
The method according to claim 1,
It includes; a first marker installed in a fixed position on the stage,
Marker extraction unit for recognizing the first marker on the terminal; performance content providing system using augmented reality comprising a.
The method according to claim 2,
A system for providing performance contents using augmented reality, further comprising: a second marker that is installed on at least one of an actor performing a performance and an object used in the performance, and is recognizable by the marker extraction unit.
A first step of determining whether the user is an audience;
A second step of obtaining the user's seat information;
A third step of extracting the reality image of the venue;
A fourth step of receiving augmented reality content;
A fifth step of adjusting the location and size of the augmented reality content according to the user's seat information and synthesizing it with the real image;
A sixth step of outputting the synthesized image to a display; And
A seventh step of adjusting the location of the augmented reality content according to the user's motion information; Method for providing performance content using augmented reality comprising a.
The method according to claim 4,
Further comprising a step 3-1 between the third step and the fourth step, extracting the marker information from the marker installed on at least one of the stage and the actor;
The fifth step is a method for providing performance content using augmented reality, characterized in that the position and size of the augmented reality content are adjusted according to the user's seat information and the marker information and synthesized with the real image.

Images