KR102568021B1 - Interactive broadcasting system and method for providing augmented reality broadcasting service - Google Patents

Abstract

The present invention relates to an interactive broadcasting system for providing an augmented reality broadcasting service and a method for providing an augmented reality broadcasting service using the same. Specifically, broadcasting production that collects broadcasting participation data of a photographing device acquiring an image of a space where an object is located and broadcasting viewers who watch the image, and determines a graphic object to be displayed on the image based on the broadcasting participation data. By including a server, it relates to an interactive broadcasting system capable of providing an experience as if the audience directly participates in content production while communicating with the audience.

Description

Interactive broadcasting system and method for providing augmented reality broadcasting service

The present invention relates to an interactive broadcasting system for providing an augmented reality broadcasting service and a method for providing an augmented reality broadcasting service using the same. More specifically, it relates to an apparatus, a system, and a method for providing interactive broadcasting for providing an augmented reality broadcasting service capable of communicating with audiences by graphically combining reactions of viewers watching a performance while relaying a performance in real time.

Internet live broadcasting is a method in which a broadcasting server transmits broadcasting content to a network, and a plurality of clients simultaneously receive and play the content, and is generally referred to as a live streaming service.

Unlike the VOD (Video On Demand) service, which provides the content desired by one client in real time, the live streaming service can provide one content to multiple clients at the same time. It is called a streaming service platform.

Recently, as the number of users (clients) using such a streaming service platform increases exponentially, the broadcasting environment is rapidly changing accordingly. That is, Internet multimedia broadcasting systems (IPTV) using wired/wireless networks are spreading beyond terrestrial, satellite, and cable broadcasting systems that provided only one-way information, and interest in interactive broadcasting services is growing.

Currently, data transmission and reception for interactive broadcasting service is limited to creating a stream of video, audio, function, screen layout information, etc. produced by the broadcasting station server, and simply combining text or images provided to viewers, which can be called practical interactive broadcasting. Therefore, there is a limit to arousing viewers' interest.

In addition, in order to produce various contents that can be applied to interactive broadcasting services such as AR (Augmented Reality) contents and can arouse interest from viewers, expensive equipment is required and the connection between each equipment is complicated, so the existing broadcasting system There is a problem that it is difficult to produce contents of various materials only, and this eventually brings another problem that it cannot satisfy broadcast viewers who want various contents.

Accordingly, the present invention proposes an apparatus, system, and method for providing an interactive broadcasting service in which artists and audiences communicate on a performance stage by combining audience reactions to a performance in real time with virtual graphics in a broadcasting screen. do.

Registered Patent Publication No. 10-1369273 (registered on February 25, 2014)

The present invention collects viewer responses such as the number of real-time broadcast viewers, the number of comments, and the contents of comments while live broadcasting is in progress, and displays graphic objects for viewer reactions on the live broadcast screen based on the collected responses, so that audiences and artists can An object of the present invention is to provide an interactive broadcasting system and method capable of providing an experience as if the audience directly participates in content production while communicating.

At the same time, the present invention is an interactive broadcasting system that can increase the interest of audiences by confirming the motion of an object through a photographing device that shoots a broadcast image, moving a graphic object accordingly, or displaying it according to a preset special effect, and It aims to provide a method.

In addition, the present invention makes it possible to obtain images for broadcasting services using only various portable terminals capable of recognizing objects, thereby minimizing the cost of constructing a broadcasting system and providing images in more diverse fields and corresponding broadcasting services. An object of the present invention is to provide an interactive broadcasting system and method capable of

Meanwhile, the problems to be solved by the present invention are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description to be described later.

In order to solve the above problem, an interactive broadcasting system according to the present invention includes a photographing device for obtaining an image of a space where an object is located; and a broadcast production server that collects broadcast participation data of broadcast viewers who watch the video and determines a graphic object to be displayed on the video based on data obtainable from the video or broadcast participation data of the broadcast viewers; includes

In addition, in the interactive broadcasting system, the broadcast participation data includes the number of broadcast viewers who watch the video, the number of responses from broadcast viewers, the number of favorable impressions, identification data of broadcast viewers, chatting data of broadcast viewers, and graphic objects of broadcast viewers. At least one of purchase data may be included.

In addition, in the interactive broadcasting system, the data obtainable from the video includes at least one of identification data of an object in the video, tracking data for movement of each object, and 3D space data for a space where the object is located. can do.

In addition, in the interactive broadcasting system, the broadcast production server determines the type, color, size, sound, and special properties of graphic objects to be displayed in the video when data obtainable from the video or broadcast participation data satisfy a preset condition. It may be characterized in that output attribute information of any one of an effect and a moving path is determined.

In addition, in the interactive broadcasting system, the broadcast production server obtains voice data of the object through a microphone possessed by the object, and broadcasts collected at the same time as the time at which the voice data is acquired among the broadcast participation data It may be characterized in that participation data is mapped with the object.

In addition, in the interactive broadcasting system, the broadcasting production server may map a display range of a graphic object representing the text to tracking data of the object when text associated with the object exists in the chatting data. can

Also, in the interactive broadcasting system, the broadcasting production server may generate broadcasting content based on data acquired through the video and the graphic object.

In addition, in the interactive broadcasting system, the broadcast production server checks the type of the graphic object arranged around the object, and if the graphic object is intersect with the corresponding object according to the movement of the object, the graphic object It may be characterized by changing output attribute information.

Also, in the interactive broadcasting system, the broadcast production server may create a streaming channel for each object participating in broadcasting, and classify the broadcast participation data based on the channel.

In addition, in the interactive broadcasting system, a video mixer for generating broadcasting content by mixing an image obtained through the photographing device and a graphic object determined by the broadcast production server, and outputting the broadcasting content; may further include.

Meanwhile, a method for providing an interactive broadcasting service according to another embodiment of the present invention includes acquiring an image of a space where an object is located; Collecting broadcast participation data of broadcast viewers who watch the video; and determining a graphic object based on data obtainable from the image or broadcast participation data. can include

In addition, the interactive broadcasting service providing method may include, after the step of determining, generating broadcasting content in which the graphic object is mixed with the video and transmitting the generated broadcasting content to a broadcasting station server or a streaming server; may further include.

In addition, in the interactive broadcasting service providing method, the data obtainable from the video may include at least one of identification data of an object in the video, tracking data for movement of each object, and 3D space data for a space where the object is located. can include

In addition, in the interactive broadcasting service providing method, the broadcasting participation data includes the number of broadcasting viewers who watch the video, the number of responses from broadcasting viewers, the number of favorable impressions, identification data of broadcasting viewers, chatting data of broadcasting viewers, and the number of broadcasting viewers. It may include at least one of graphic object purchase data.

In addition, in the interactive broadcasting service providing method, the determining may include the type, color, size, sound, special effect, and movement path of a graphic object to be displayed on the image when the broadcasting participation data satisfies a preset condition. It may be a step of determining any one of attribute information.

In the interactive broadcast service providing method, the collecting may further include creating a streaming channel for each object participating in broadcasting and classifying the broadcast participation data based on the streaming channel.

According to the present invention, instead of a broadcasting system that provides images unilaterally, by converting viewers' reactions into graphic objects and immediately sending them to live broadcasting, there is an effect that viewers can produce interactive broadcasting in a desired direction. There is also an effect that a performing artist can experience creating a single broadcasting content.

In addition, according to the present invention, the interest of the audience can be increased by moving the graphic object according to the movement of the object during broadcasting and providing special effects according to various motions.

In addition, according to the present invention, in order to implement a series of processes of generating and broadcasting content by inserting a graphic object to which a viewer's reaction is applied to an image including object data or spatial data, expensive equipment is not required, and the entire system is established. It has the effect of minimizing costs.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 illustrates an implementation example of an augmented reality broadcasting service proposed by the present invention.
2 shows a schematic configuration of an interactive broadcasting system according to the present invention.
3 shows a schematic configuration of a broadcast production server according to the present invention.
4 shows a schematic flow for providing an augmented reality broadcasting service in a broadcast production server according to the present invention.
5 and 6 show examples of actual implementation of the interactive broadcasting service according to the present invention.
7 illustrates an actual implementation example of an interface screen for providing an interactive broadcasting service according to the present invention.

Objects and technical configurations of the present invention and details of the operational effects thereof will be more clearly understood by the following detailed description based on the accompanying drawings in the specification of the present invention. An embodiment according to the present invention will be described in detail with reference to the accompanying drawings.

The embodiments disclosed herein should not be construed or used as limiting the scope of the present invention. It goes without saying that the description, including the embodiments herein, has a variety of applications for those skilled in the art. Therefore, any embodiments described in the detailed description of the present invention are illustrative for better explaining the present invention, and the scope of the present invention is not intended to be limited to the examples.

The functional blocks shown in the drawings and described below are only examples of possible implementations. Other functional blocks may be used in other implementations without departing from the spirit and scope of the detailed description. Also, while one or more functional blocks of the present invention are represented as separate blocks, one or more of the functional blocks of the present invention may be a combination of various hardware and software configurations that perform the same function.

In addition, the expression that certain components are included simply refers to the existence of the corresponding components as an expression of “open type”, and should not be understood as excluding additional components.

Furthermore, it should be understood that when a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but other components may exist in the middle. do.

An interactive broadcasting system for providing an augmented reality broadcasting service according to the present invention will be described with reference to FIGS. 1 and 2 .

First, FIG. 1 illustrates an implementation example of an augmented reality broadcasting service proposed by the present invention.

Referring to FIG. 1, the interactive broadcasting system 1000 of the present invention broadcasts a space 1 in which a stage production is performed in real time through a broadcasting station server 20 or a streaming server 30, and viewers 10 ) can be displayed as a graphic object in the broadcasting screen. Specifically, real-time broadcasting is performed based on images captured by the photographing device 100, and the broadcasting production server 200 collects responses from viewers 10 and displays them as multiple graphic objects (a) (b) on the broadcasting screen. ) (c) can be controlled to be displayed as (d). Here, the graphic object is basically in the form of an image, but may be additionally implemented in the form of text and audio. For example, audio effects when (a) a graphic object appears or disappears, (d) a number within a graphic object rises or falls, or an object (A') is intersect with (b) a graphic object. may be output, and when chatting data of the viewer 10 for any one object (B) is obtained, (c) graphic object including text may be output.

In addition, the broadcast production server 200 may classify and collect viewers' reactions for each object (A') (A") (B) disposed in the space (1), where the objects are broadcast performers and performance stage singers. , various people, objects, animals, etc., such as people such as moderators and sculptures that can specify the space 1. However, for convenience of description, it is assumed that the objects described below are people. The server 200 may display a graphic object at a position corresponding to the position of the object by classifying the reaction of viewers for each object, and the display and position of the graphic object will be described in detail below.

2 shows a schematic configuration of an interactive broadcasting system according to the present invention.

Referring to FIG. 2 , an interactive broadcasting system 1000 providing an augmented reality broadcasting service may include a photographing device 100, a broadcast production server 200, a video mixer 300, and a viewer terminal 400.

The photographing device 100 may obtain an image of a space where an object is located. The photographing device 100 may include a Lidar sensor or a Time Of Flight (ToF) sensor for recognizing an object and acquiring positional coordinates of each point in the photographing space, 3D space information, and distance information. ), the broadcast production server 200 may obtain at least one of object identification data, motion tracking data for each object, and 3D space data for a space where the object is located.

In the present invention, the photographing device 100 may include various types of photographing devices capable of obtaining object and space data, and as an example, a broadcasting camera (eg, an ENG camera), as well as a smart phone equipped with a camera, A webcam, a camcorder, or the like may be the photographing device 100 . In addition, the photographing device 100 may transmit an image acquired using a wired/wireless network to the broadcast production server 200 .

The broadcasting production server 200 is a server capable of generating broadcasting content for providing an augmented reality broadcasting service, and may be a single server device or a system composed of several servers in some cases. Specifically, the broadcast production server 200 may collect broadcast participation data of broadcast viewers watching the video, and determine a graphic object to be displayed on the video based on the broadcast participation data. Here, the broadcast participation data is a generic term for all data that can be obtained through broadcast viewers, and the broadcast production server 200 obtains by uploading a video in real time through a broadcast channel or a streaming channel, such as the number of channel users, viewer ratings, and number of comments. , the number of likes, comments, paid item purchase information, etc. may be included.

The broadcast production server 200 may check whether broadcast participation data satisfies a preset condition, determine a graphic object to be displayed in an image when the condition is met, and display it on the broadcast screen in consideration of the location of the object and space. there is. In this way, the broadcast production server 200 goes beyond simply outputting conventional viewers' reactions and displays them using pre-manufactured graphic objects when a preset condition is satisfied, so that the viewers' reactions and intentions are reflected in the broadcast. The difference is that an object participating in the broadcasting can proceed with the broadcasting through a graphic object, so that a true interactive broadcasting service can be achieved.

Meanwhile, the broadcasting production server 200 uses various APIs (Application Programming Interfaces) from a broadcasting channel platform (broadcasting station server 20) or a streaming channel platform (streaming server 30) to collect viewers' reactions to the video. can be provided. For example, the broadcast production server 200 may be provided with an API for setting a broadcast and an API for collecting a viewer's reaction in order to transmit a video.

The video mixer 300 is a device of a broadcast producer that collects all the data received from the photographing device 100 and the broadcast production server 200 to finally create broadcast content to be transmitted to the broadcast. Broadcasting content may be generated by mixing an image acquired through the photographing device 100 and a graphic object determined by the broadcasting production server 200 through, and output. For example, the video mixer 300 receives video and graphic object display data from the individually connected photographing device 100 and the broadcast production server 200, generates one broadcast content, and then outputs it as a broadcast signal. For another example, broadcast content delivered by the broadcast production server 200 may be converted into a broadcast signal using a converter (not shown) and then output. That is, the broadcast producer can check the broadcast content for the augmented reality broadcast service already completed by the broadcast production server 200 through the video mixer 300 and immediately output it to a broadcast channel or a streaming channel, and thus the viewer's reaction. You can omit the cumbersome process of manually checking and filtering inappropriate content.

Meanwhile, the photographing device 100, the broadcast production server 200, and the video mixer 400 may be connected to each other through physical connectors. For example, the connector may include a high definition multimedia interface (HDMI) connector, a universal serial bus (USB) connector, an SD card connector, a serial digital interface (SDI) connector, or an audio connector.

The viewer terminal 400 is a terminal possessed by a viewer using an augmented reality broadcasting service, and may include various electronic devices such as a smart phone, a PC, and a tablet PC. A viewer accesses a streaming channel or a broadcasting channel created by the broadcast production server 200, provides an opinion on the broadcast, expresses interest, or leaves an answer to a question while watching the broadcast. Here, the expression of interest or the operation of leaving a text may be performed by inputting a comment into a comment window activated on a streaming channel or broadcasting channel, or by sending a text message to a number displayed on a broadcasting screen.

So far, the interactive broadcasting system 1000 for providing the augmented reality broadcasting service of the present invention has been briefly reviewed, and the broadcasting production server 200 that determines graphic objects for implementing augmented reality will be described below. .

3 shows a schematic configuration of a broadcast production server according to the present invention.

Referring to FIG. 3 , a broadcast production server 200 may include a communication unit 210, an input unit 220, an output unit 230, a storage unit 240, and a processor 250.

The communication unit 210 may transmit and receive various data from the photographing device 100, the video mixer 300, the viewer terminal 400, the broadcasting station server 20, and the streaming server 30 using a wired/wireless network. For example, the communication unit 210 may receive an image including data such as object identification data, motion tracking data for each object, and 3D space data for a space where the object is located from the photographing device 100. , It is possible to receive broadcasting participation data such as broadcasting viewer's chatting data and graphic object purchase data from the viewer terminal 400, and video and graphic objects to the video mixer 300, broadcasting station server 20 or streaming server 30 can transmit mixed broadcast content.

The input unit 220 may obtain various setting information for providing an augmented reality broadcast service from a broadcast producer. Specifically, the input unit 220 may obtain information about a broadcasting channel or a streaming channel providing an augmented reality broadcasting service, or obtain graphic object output attribute information of a broadcaster. For example, the input unit 220 may obtain information for generating different streaming channels (live channels) for each object participating in broadcasting together with identification information for generating a streaming channel from a broadcast producer, and broadcast transmission speed. information can be obtained. For another example, the input unit 220 may newly acquire information about a broadcast participant (object) appearing while an augmented reality broadcast service is being provided and text related to the object from a broadcast producer. Here, the text may indicate a condition in which a graphic object is to be displayed. For example, the input unit 220 may obtain broadcast performer information appearing in the augmented reality broadcast service to be broadcast on December 12, and text information such as the broadcast performer's name, nickname, buzzword, etc., from a broadcast producer. can be used to determine For another example, the input unit 220 obtains a condition for displaying a graphic object from a broadcast producer or, when one condition is met, output property information of a graphic object that can be visually and audibly confirmed by broadcast viewers. can be obtained In addition, the output property information may include images such as the type, color, and size of graphic objects, sounds, and moving paths within a screen.

The output unit 230 may visually and audibly output an interface for providing an augmented reality broadcasting service and a graphic object to be displayed on the broadcasting service. Through the output unit 230, a broadcast producer can edit a graphic object to be displayed on a broadcast, and can adjust the position of a graphic object to be displayed on an image. In addition, the output unit 230 may recognize a user's touch input, and accordingly, the output unit 230 and the input unit 220 may be formed as one physical component. For reference, interfaces or objects that are output or transmitted through the output unit 230 may be stored in the storage unit 240 to be described later.

The storage unit 240 may store various types of output property information and condition information for determining a graphic object. Specifically, the storage unit 240 may store various displayable graphic object information for each type of broadcast participation data. For example, the storage unit 240 may store an image of a graphic object that changes step by step in the case of broadcast participation data represented by numbers, and in the case of broadcast participation data represented by text, identification information of broadcast viewers and text are inserted. An image of a graphic object to be used may be stored, and in the case of broadcast participation data for which a predetermined amount has been paid, images of different graphic objects may be stored according to the amount. Also, the storage unit 240 may store reference information for displaying or converting graphic objects. For example, the storage unit 240 may include the number of broadcasting viewers for displaying graphic objects, ratings, comments, number of likes, chatting, comment content, number of viewers for sequentially switching images of graphic objects, ratings, and comments. number, number of likes, purchase information for paid items, etc. can be stored.

In addition, the storage unit 240 may store performer information (object information), appearance time information for each performer, etc. in relation to any one broadcast content for which augmented reality broadcasting service is provided, and microphone (not shown) identification information for each performer. can also be stored.

Meanwhile, the storage unit 240 may include a volatile or non-volatile recording medium capable of storing various data, commands, and information. For example, the storage unit 240 is a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg SD or XD memory, etc.), RAM, SRAM, ROM, EEPROM, PROM, network storage It may include at least one type of storage medium among storage and cloud.

On the other hand, commands for operating the broadcast providing server 200 may be recorded in the storage 240, and programs for providing augmented reality broadcasting services may be recorded in the storage 240. When the program is executed by the processor 250, the output unit 230 may provide a user interface and may include a recording medium storing instructions or program codes to output appropriate messages (data) according to user input. .

The processor 250 is operatively connected to the communication unit 210, the input unit 220, the output unit 230, and the storage unit 240, controls the overall operation of the broadcast providing server 200, and determines graphic objects. and perform various commands for generating broadcast content.

The processor 250 may obtain an image of a space in which an object is located through the communication unit 210, upload it to a broadcast channel or a streaming channel, and collect broadcast participation data of broadcast viewers watching the image in real time. Here, the broadcast participation data is a generic term for all data that can be obtained through broadcast viewers, and the broadcast production server 200 obtains by uploading a video in real time through a broadcast channel or a streaming channel, such as the number of channel users, viewer ratings, and number of comments. , the number of likes, comments, paid item purchase information, etc. may be included.

In addition, the processor 250 may create a streaming channel for each object participating in broadcasting, and may classify broadcast participation data based on the channel. For example, the processor 250 can classify broadcast participation data collected in one broadcast by separately generating streaming channels supporting participants 1, 2, and 3, and collected through the corresponding channels. The broadcast participation data can be mapped to each participant (object). In this way, by classifying the broadcasting participation data for each channel, the amount of calculation used to limit the graphic objects for tens of thousands of broadcasting participation data to a specific object or location can be greatly reduced.

Meanwhile, in order to determine a graphic object, which is an essential element for implementing an augmented reality broadcasting service, spatial data about an object and an image are essential. Accordingly, the processor 250 may obtain at least one of object identification data, motion tracking data for each object, and 3D space data for a space where the object is located through the image. For example, if a group including a plurality of members exists in an image, the processor 250 may identify each member, obtain tracking data for each member, coordinates of the center of the stage, coordinates of the edge area, etc. of spatial data can be obtained. On the other hand, as the processor 250, significant calculations are required to individually extract and obtain object identification data, tracking data, and 3D spatial data from the image, which may cause delay. In the interactive broadcasting system according to the present invention, identification data of an object (person) and 3D space data may be simultaneously obtained by further including a depth camera set to interlock the angle of view with the photographing device 100, that is, a broadcasting camera. That is, the processor 250 extracts the object identification data and object identification data from the image captured and recognized by the photographing device 100 and the depth camera without the need to separately extract object identification data or 3D space data from the image. Since it is possible to acquire 3D spatial data while being matched, it is possible to reduce the amount of calculation.

If all data related to the graphic object is obtained, the processor 250 determines whether the broadcast participation data satisfies a preset condition, and determines a graphic object to be displayed on the image if the preset condition is satisfied as a result of the determination. At this time, the pre-set conditions are broadcast producers/hosts requesting viewer chat data, channel accesses of 10,000/20,000/50,000 or more, audience rating of 1.3% or more, number of likes of 100,000 or more, and text "56" as the correct answer to the question. Various conditions may be included, such as when receiving chatting data and purchasing paid graphic objects.

In addition, determining the graphic object by the processor 250 can be understood as determining the image and audio information of the graphic object and determining the movement path, and more specifically, the type and color of the graphic object to be displayed on the image. , it can be understood as determining output attribute information of any one of size, sound, and movement path.

Among them, the processor 250 may utilize broadcast participation data of broadcast viewers to designate a display range, ie, a location, of a graphic object. Specifically, when there is text related to the object in the chatting data, the processor 250 may map the display range of the graphic object representing the text to the tracking data of the object. For example, if there is chatting data related to the name, nickname, and buzzword of the Z object among a plurality of chatting data, the processor 250 displays the corresponding chatting data as a graphic object and displays the movement of the Z object identified in the video. can be mapped with the movement of

In addition to this, the processor 250 obtains voice data of the object through a microphone (not shown) possessed by the object, and holds the broadcast participation data collected at the same time as the time at which the voice data is acquired among the broadcast participation data through the microphone. It can be mapped to an object. For example, the processor 250 maps the location to a location adjacent to the M object in the process of displaying the broadcast participation data collected during the time when the M object unilaterally asks a question as a graphic object, so that the M object and the viewer can interact with each other. You can experience it as if you were broadcasting while taking a quiz.

As the broadcasting participation data satisfies the conditions as described in the above example, the processor 250 determines a graphic object, and may generate broadcasting content by mixing graphic objects suitable for output property information designated for an image. The generated broadcasting content is transmitted to the broadcasting server 20 or the streaming server 30, and viewers can enjoy the created augmented reality broadcasting service through their own viewer terminal 400.

Meanwhile, a graphic object may be determined by the processor 250 and, while the graphic object is arranged at a designated location, the graphic object may be transformed according to a behavior of an object participating in broadcasting and output property information of the graphic object. Here, the output property information is special effect information assigned to the graphic object. As the object in the image moves, if the object and the graphic object are intersect in one video frame, the graphic object disappears or moves up/down/left/right. Various special effects in which another graphic object is overlaid on an object may occur.

So far, the broadcasting production server 200 for implementing the augmented reality broadcasting service of the present invention has been looked at. According to the present invention, the broadcasting production server 200 is not a broadcasting system that provides images unilaterally, but converts viewers' reactions into graphic objects and immediately transmits them to live broadcasting, thereby producing interactive broadcasting in a direction desired by viewers. It can be effective.

4 shows a schematic flow for providing an augmented reality broadcasting service in a broadcast production server according to the present invention.

Referring to FIG. 4 , the broadcast production server 200 obtains an image of a space where an object is located (S110). The broadcasting production server 200 may obtain at least one of object identification data, tracking data for movement of each object, and 3D space data for the space where the object is located through the corresponding video, and based on this, later graphic objects The location and type of can be determined.

After step S110, the broadcast production server 200 collects broadcast participation data of broadcast viewers watching the video (S120). Specifically, the video is uploaded to a broadcast channel or streaming channel, and the broadcast production server 200 may collect broadcast participation data through text messages received through the broadcast channel or comments in a chat window. Here, the broadcast participation data is a generic term for all data that can be obtained through broadcast viewers, and the broadcast production server 200 obtains by uploading a video in real time through a broadcast channel or a streaming channel, such as the number of channel users, viewer ratings, and number of comments. , the number of likes, comments, paid item purchase information, etc. may be included.

On the other hand, since the amount of broadcasting participation data can increase infinitely according to the size of the augmented reality broadcasting service, the broadcasting production server 200 creates a streaming channel for each object participating in broadcasting, classifies broadcasting participation data for each channel, and It is possible to omit the task of classifying broadcast participation data one by one in order to determine the location or movement path of an object.

After step S120, the broadcast production server 200 determines a graphic object based on at least one of data obtainable from the video or broadcast participation data (S130). Preferably, the broadcast production server 200 acquires from the video A graphic object may be determined by combining the data and broadcast participation data. More specifically, the broadcast production server 200 may determine a graphic object to be displayed in the video when the data obtained from the video or the broadcast participation data satisfies a preset condition. It is possible to determine any one attribute information among type, color, size, sound, special effect, and moving path. Here, the preset condition for the data obtained from the image is when the area occupied by the objects (members) in the image is less than the preset value and the area in the screen where the graphic object can be displayed is sufficient, the object When referring to tracking data for objects, cases in which objects perform preset movements, cases in which objects deviate from or stand at a predetermined position in a 3D space, and the like may be included. In addition, preset conditions in broadcasting participation data may include various conditions such as when a target number is achieved, when a specific text is received, when a paid graphic object is purchased, and the like.

After step S130, the broadcasting production server 200 generates broadcasting content in which a graphic object is mixed with an image, and transmits it to the broadcasting station server 20 or the streaming server 30 (S140). That is, finally, broadcast viewers can check how their responses or texts are actually reflected on the broadcast screen, and interest in broadcasting can be doubled accordingly.

If steps S110 to S140 are applied to an actual broadcast service providing environment, step S110 may be a step of obtaining an image by filming a stage performance of an idol group consisting of 5 members, and the broadcast production server 200 From the obtained image, at least one of member-specific identification data (object identification data) for identifying each member of idol group A, tracking data for tracking the movement of each member, or 3-dimensional space data for the space where each member is located. of data can be obtained.

In addition, step S120 is a step of collecting broadcast participation data, for example, related comments, paid item purchase information, number of visitors, number of likes (likes), etc. can be

At this time, the broadcast production server 200 may collect broadcast participation data for each member through a streaming channel created for each member, and by utilizing the channel for each object in this way, a large amount of broadcast participation data is relatively easy. be able to process On the other hand, even if there is no streaming channel for each member, the broadcast production server 200 provides information that can identify that the broadcast participation data is directed to a specific member, for example, the name/nickname of the specific member included in the comment contents. / By extracting a tag (hashtag) including a nickname or a specific member name, etc., the broadcasting participation data can be implemented to be classified for each member.

In step S130, the broadcast production server 200 determines a graphic object based on at least one of data obtainable from a video or broadcast participation data. % or less, the broadcast production server 200 may determine a graphic object so that the three-dimensionally expressed group name is output in an area other than the area occupied by the A idol group members, or any member of the A idol group members. When a cheering message for is received, a graphic object may be determined so that the message is displayed on the side of the corresponding member on the screen.

5 and 6 show examples of actual implementation of the interactive broadcasting service according to the present invention.

Referring to FIG. 5A , a broadcast viewer may view an image in which a graphic object 51 exists except for a dotted line through a viewer terminal 400 . The graphic object 51 in FIG. 5A is the group name of the group currently appearing in the video, but it can be changed and displayed as other content. Also, here, the graphic object 51 may be newly switched and displayed according to the number of channel accesses, viewer ratings, number of comments, number of likes, paid item purchase information, and the like.

Meanwhile, the graphic object 51 may also be transformed according to the movement of the object disposed in the center of the broadcasting screen. For example, when an object enters area 1 where the graphic object 51 is arranged and the two objects cross each other, the graphic object may move to area 2, 3, or 4 or disappear.

FIG. 5B also shows an actual implementation example in which group names are output in the video similarly to FIG. 5A, but FIG. 5B shows a text message cheering for the group in the video as the amount of broadcast participation data received from broadcast viewers accumulates. , chatting messages, likes, etc. are piled up to show an implementation example in which the group name 52 is completed. As such, the interactive broadcasting service according to the present invention can output various graphic objects in connection with broadcasting participation data.

Next, referring to FIG. 6 , a broadcast viewer can view a video in which graphic objects 63 and 64 for chatting data created by the broadcast viewer are present through the viewer terminal 400 . The graphic objects 63 and 64 for the chatting data are placed in the area around the specific object (C') (C"), which contains information about the specific object (C') (C") in the chatting data and If there is, the location can be assigned automatically.

Meanwhile, another graphic object (61, 62) disposed on the object (C') (C") may be inserted by a broadcast producer.

7 illustrates an actual implementation example of an interface screen for providing an interactive broadcasting service according to the present invention.

A broadcast producer can first select a menu 70 for providing an augmented reality broadcast service through the interface of the broadcast production server 200, and in FIG. 7, one of the augmented reality broadcast services is a video chat service (Super Chat service). It shows how the menu is selected to provide. After the video chat service menu is selected, a specific streaming platform 71 (eg, YouTube) may be selected, and information (eg, login information, etc.) on the selected streaming platform may be input through the information input field 72. In this case, the streaming platform may be a channel for collecting broadcast participation data from broadcast viewers. After the streaming platform is selected, sources required to provide a video chat service, that is, broadcast participation data including an ID and contents may be received from broadcast viewers. At this time, the received broadcast participation data may be implemented to be output within an arbitrary region 73 of the interface 200 . For reference, it is understood that an arbitrary area 73 in the interface 200 may be displayed in a different way according to the interface designer's intention. Meanwhile, after the broadcast participation data are received, the graphic object 74 created in advance may be controlled to be output in association with the received broadcast participation data. For example, a graphic object that receives OX votes as comments and displays comments on the side according to the movement of idol group members, a graphic object in the form of filling empty spaces as much as the number of accumulated comments, etc. Output in conjunction with the broadcast participation data can be controlled as much as possible.

An interactive broadcasting system that provides an augmented reality broadcasting service was examined. The present invention is not limited to the specific embodiments and application examples described above, and various modifications can be made by those skilled in the art without departing from the gist of the present invention claimed in the claims. Of course, these modified implementations should not be understood separately from the technical spirit or perspective of the present invention.

1000: interactive broadcasting system
10: viewers
20: broadcasting station server
30: streaming server
100: shooting device
200: broadcast production server
210: communication department
220: input unit
230: output unit
240: storage unit
250: processor
300: video mixer
400: viewer terminal

Claims (16)

a photographing device that acquires an image of a space where a plurality of objects are located; and
a broadcast production server that collects broadcast participation data of broadcast viewers who watch the video and determines a graphic object to be displayed on the video based on data obtainable from the video and broadcast participation data of the broadcast viewers; Including,
The data obtainable from the image includes identification data of objects in the image and tracking data for motion of each object,
The broadcast production server, if there is broadcast participation data related to a specific object among the broadcast participation data - the broadcast participation data related to the specific object includes the name, alias, nickname, or hashtag of the specific object -, the corresponding Characterized in that the broadcast participation data is mapped with the movement of the specific object and displayed as a graphic object in the image,
Two-way broadcasting system.
According to claim 1,
The broadcast participation data,
Including at least one of the number of broadcast viewers who watch the video, the number of responses from broadcast viewers, the number of like marks, identification data of broadcast viewers, chat data of broadcast viewers, and graphic object purchase data of broadcast viewers,
Two-way broadcasting system.
According to claim 1,
The data obtainable from the image are,
Further comprising three-dimensional space data for the space in which the objects are located,
Two-way broadcasting system.
According to claim 2 or 3,
The broadcast production server,
When data obtainable from the image or broadcast participation data satisfies a preset condition, determining output property information of any one of the type, color, size, sound, special effect, and movement path of the graphic object to be displayed on the image ,
Two-way broadcasting system.
According to claim 4,
The broadcast production server,
Further acquiring voice data of an arbitrary object through a microphone possessed by the object,
Mapping broadcast participation data collected at the same time as the time at which the voice data is obtained among the broadcast participation data with the object;
Two-way broadcasting system.
According to claim 2,
The broadcast production server,
If there is text associated with an arbitrary object in the chat data, mapping the display range of the graphic object representing the text with the tracking data of the object,
Two-way broadcasting system.
According to claim 1,
The broadcast production server,
Generating broadcast content based on data obtained through the image and the graphic object;
Two-way broadcasting system.
According to claim 1,
The broadcast production server,
Checking the type of graphic object arranged around the object, and changing the output property information of the graphic object when the graphic object intersects with the corresponding object according to the movement of the object,
Two-way broadcasting system.
According to claim 1,
The broadcast production server,
Creating a streaming channel for each object participating in broadcasting, and classifying the broadcast participation data based on the channel.
Two-way broadcasting system.
According to claim 1,
an image mixer generating broadcast content by mixing an image acquired through the photographing device and a graphic object determined by the broadcast production server, and outputting the broadcast content; Including more,
Two-way broadcasting system.
As a method of providing interactive broadcasting service performed by a broadcasting production server,
obtaining an image of a space where a plurality of objects are located;
Collecting broadcast participation data of broadcast viewers who watch the video; and
determining a graphic object based on data obtainable from the image and broadcast participation data; Including,
The data obtainable from the image includes identification data of objects in the image and tracking data for motion of each object,
The broadcast production server, if broadcast participation data related to a specific object among the broadcast participation data - the broadcast participation data related to the specific object includes the name, alias, nickname, or hashtag of the specific object - exists, the corresponding Characterized in that the broadcast participation data is mapped with the movement of the specific object and displayed as a graphic object in the image,
How to provide interactive broadcasting service.
According to claim 11,
After the determining step,
generating broadcasting content in which the graphic object is mixed with the image and transmitting the broadcasting content to a broadcasting station server or a streaming server; Including more,
How to provide interactive broadcasting service.
According to claim 11,
The data obtainable from the image are,
Further comprising three-dimensional space data for the space where the object is located,
How to provide interactive broadcasting service.
According to claim 11,
The broadcast participation data,
Including at least one of the number of broadcast viewers who watch the video, the number of responses from broadcast viewers, the number of like marks, identification data of broadcast viewers, chat data of broadcast viewers, and graphic object purchase data of broadcast viewers,
How to provide interactive broadcasting service.
According to claim 13 or 14,
The determining step is
When the broadcast participation data satisfies a preset condition, determining any one of attribute information among the type, color, size, sound, special effect, and movement path of the graphic object to be displayed on the image,
How to provide interactive broadcasting service.
According to claim 11,
The collecting step is
Creating a streaming channel for each object participating in broadcasting, and classifying the broadcast participation data based on the streaming channel.
How to provide interactive broadcasting service.