KR102615377B1 - Method of providing a service to experience broadcasting - Google Patents

Abstract

A broadcast experience service is provided that allows users to directly experience broadcast videos of various genres in their lives and share the experience.
In this method of providing a broadcast experience service, a service server that provides a broadcast experience service using a plurality of broadcast videos (each broadcast video includes broadcast video and broadcast audio): detects a person object that appears in each broadcast video; Extracting a first boundary line of a human object; When the user terminal receives a selection of one of a plurality of broadcast videos, playing the selected broadcast video and its script on the display of the user terminal; Receiving, from a user terminal, a acting video generated by filming the participant's acting experience in accordance with the playback of the broadcast video (the acting video includes acting video and acting audio); detecting a participant object appearing in the performance video and extracting a second boundary line of the participant object; From the broadcast video, image segmentation is performed based on the first boundary line to extract the background excluding the person object. From the acting video, the image is segmented based on the second boundary line to extract participant objects, and then the background and participant objects are combined. generating a composite video; and the service server generating a composite video combining the composite video and the acting audio.

Description

{Method of providing a service to experience broadcasting}

The present invention relates to a method of providing a broadcast experience service, and more specifically, to a method of providing a broadcast experience service in which a composite video combining an actual broadcast video with a participant's performance video can be shared through SNS or the like.

Broadcasting companies create broadcast video content of various genres, including dramas, movies, entertainment, and news reports, and deliver it through various media. To date, no other methods have been explored to utilize these broadcast video archive assets, except for re-watching services through IPTV, OTT, etc.

Accordingly, the present inventor completed the present invention after putting a lot of research and effort into a way for users to directly experience broadcast videos and share the experience.

The problem that the present invention seeks to solve is to provide a broadcast experience service that allows users to directly experience broadcast videos of various genres in their lives and share the experience.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood by those skilled in the art from the description below. will be.

A method of providing a broadcast experience service according to an embodiment of the present invention for achieving the above problem involves a service server providing a broadcast experience service using a plurality of broadcast videos (each broadcast video includes broadcast video and broadcast audio). ): detecting a person object appearing in each broadcast video and extracting a first boundary line of the person object; When the user terminal receives a selection of one of the plurality of broadcast videos, playing the selected broadcast video and its script on the display of the user terminal; Receiving, from the user terminal, a acting video generated by filming the participant's acting experience in accordance with the playback of the broadcast video (the acting video includes acting video and acting audio); detecting a participant object appearing in the performance video and extracting a second boundary line of the participant object; After extracting the background excluding the person object by performing image segmentation based on the first boundary line in the broadcast video, and extracting the participant object by segmenting the image based on the second boundary line in the acting video, generating a composite video by compositing a background and the participant object; and generating, by the service server, a composite video combining the composite video and the acting audio.

The first borderline may be a facial edge of the human object. The second boundary line may be a facial edge of the participant object.

When the user terminal receives service participation from multiple participants, the service server detects a plurality of person objects appearing in each broadcast video and extracts a first boundary line of each person object; the service server detects a plurality of participant objects appearing in the performance video and extracts a second boundary line of each participant object; The service server matches the plurality of person objects with the plurality of participant objects according to predetermined matching criteria; The service server extracts a background excluding the plurality of person objects from the broadcast video, extracts the plurality of participant objects from the acting video, and then synthesizes the background and the plurality of participant objects by reflecting the matching criteria. Composite video can be created.

The method may further include sharing the composite video through at least one of SNS, a portal site, a video sharing service platform, an Internet media broadcasting platform, and an email server at the request of the user terminal.

The method may further include generating an animated GIF (animated GIF) by combining a plurality of still images extracted from the composite video.

Details of other embodiments are included in the detailed description and drawings.

As described above, according to the method of providing a broadcasting experience service according to the present invention, the user accesses the service server through his/her user terminal and selects the desired broadcasting video among various broadcasting videos including dramas, movies, entertainment, news reports, etc. After selecting , you can perform an acting experience by referring to the broadcast video and script while it is played through the user terminal. The user terminal captures the acting experience and creates a acting video, and the service server can synthesize the selected broadcast video and the acting video to create a composite video.

Users can use the user terminal to share composite videos through SNS, portal sites, video sharing service platforms, Internet media broadcasting platforms, email servers, etc. Additionally, the service server can create a moving GIF (animated GIF) that combines multiple still images extracted from the synthetic video, and the user can use the user terminal to upload the animated GIF to SNS, portal sites, video sharing service platforms, and internet media. It can be shared through broadcasting platforms, email servers, etc.

1 is a configuration diagram showing a broadcast experience service system according to an embodiment of the present invention.
FIG. 2 is a schematic configuration diagram of the user terminal of FIG. 1.
Figure 3 is a flowchart sequentially showing a method of providing a broadcast experience service according to an embodiment of the present invention.
Figure 4 is a diagram illustrating an exemplary user interface of a user terminal on which a broadcast video and its script are played.
Figure 5 is a diagram illustrating an exemplary user interface of a user terminal on which an acting video is played.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and will be implemented in various different forms. The present embodiments only serve to ensure that the disclosure of the present invention is complete and that common knowledge in the technical field to which the present invention pertains is not limited. It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

In this specification, the term 'video' is used to include video and audio. For example, a broadcast video includes broadcast video and broadcast audio, and a smoke video includes a smoke video and smoke audio.

Hereinafter, a broadcast experience service system according to an embodiment of the present invention will be described in detail with reference to FIGS. 1 and 2. 1 is a configuration diagram showing a broadcast experience service system according to an embodiment of the present invention. FIG. 2 is a schematic configuration diagram of the user terminal of FIG. 1. The broadcast experience service system of the present invention includes a plurality of user terminals 200 and a service server 100 connected to them through a wireless network.

The service server 100 is a device that provides a broadcast experience service using a plurality of broadcast videos. The service server 100 may be comprised of one or more server devices related to service management, data communication and processing, database construction, security and authentication, etc., and may include hardware and/or software equipment. The service server 100 may provide mobile application software or broadcast experience service mobile application software, and may also provide a web-based platform such as a website. The service server 100 builds and manages multiple databases. The database may store broadcast video selection information uploaded from the user terminal 200, service participation information of multiple participants, information on the number of participants, performance videos, content information, etc. Additionally, if necessary, a database that records user information, message information, advertisement information, billing information, log information, etc. necessary for providing broadcasting experience services may be added.

The user terminal 200 is a terminal device including a processor, memory, input means, and output means, and may have a wireless communication function that the user can carry while moving. A broadcast experience service mobile application that implements a broadcast experience service is installed in the user terminal 200. The broadcast experience service mobile application creates an acting video by filming the participant's acting experience through input means such as a camera while playing the broadcast video and its script that have already been broadcast and stored through other media, and synthesizes the broadcast video and the acting video. As a mobile application that can play or share synthetic videos, it can communicate with the service server 100 or maintain a communication state using the rules and settings set by the OS of the user terminal 200. The broadcast experience service mobile application is application software that the user downloads to the user terminal 200 using an external platform or an external link, or is linked to the operating system software of the user terminal 200 and is installed by default on the user terminal 200. It may be software installed on . The broadcast experience mobile application may be linked to a basic application (eg, a text, voice signal, or data transmission/reception application) installed by default on the user terminal 200, or may be run independently without any separate linkage.

The input means of the user terminal 200 may include a software or hardware input device, and the output means of the user terminal 200 may include a speaker and a display. When the user terminal 200 is a mobile device, the display may include a user interface as a means of detecting the user's touch input in the UI/UX of the operating system software and the UI/UX of the application software. The display may be a touch screen that functions as both a means of outputting a screen and an input means of detecting a user's touch event. An input means of the user terminal 200 may include a camera.

The memory of the user terminal 200 generally provides a place to store computer code and data used in the device. For example, the memory of the user terminal 200 includes not only the broadcast experience service mobile application and the resources required to run/manage the same, but also the basic input/output system, operating system, various programs, applications, or user interface functions running on the device. , firmware for any device including processor functions, etc. may be stored.

The processor of the user terminal 200 executes computer code together with the operating system and performs operations to generate and use data. Additionally, a processor can use a series of instructions to receive and process input and output data between components of an electronic device. In addition, the processor serves as a control unit that executes the functions of the operating system software and various application software (including broadcast experience service mobile applications) installed on the electronic device.

The user terminal 200 may include a terminal device capable of transmitting and receiving various data via a wired or wireless network according to a user's key operation, touch input, drag input, or command. For example, the user terminal 200 may be a personal computer (PC), a tablet computer, a laptop computer, a PDA, or a mobile device including a smartphone. Preferably, the user terminal 200 may be a smartphone that the user can carry while moving.

Hereinafter, a method for providing a broadcast experience service according to an embodiment of the present invention will be described in detail with reference to FIGS. 3 to 5. Figure 3 is a flowchart sequentially showing a method of providing a broadcast experience service according to an embodiment of the present invention. Figure 4 is a diagram illustrating an exemplary user interface of a user terminal on which a broadcast video and its script are played. Figure 5 is a diagram illustrating an exemplary user interface of a user terminal on which an acting video is played.

First, the service server 100 prepares a plurality of broadcast videos to be used in the broadcast experience service (S10). Each broadcast video includes broadcast video and broadcast audio. Broadcast videos may belong to various genres such as dramas, movies, entertainment, and news reports, and may include, for example, 10 to 20 seconds of famous lines or news reports to encourage active participation from users.

The service server 100 detects the human object appearing in each broadcast video and extracts the first boundary lines 222 and 224 of the human object. Here, the first border lines 222 and 224 may be facial edges of the human object. Referring to the user interface 210 of the user terminal 200 where the broadcast video is played in FIG. 4, the service server 100 monitors each frame of the broadcast video to search for a human object. Specifically, the service server 100 extracts broadcast video frame by frame and analyzes the frame to detect edges. The service server 100 determines that pixel areas including pixels arranged consecutively and having similar pixel values among pixel areas divided by an edge are one person object. The service server 100 determines the pixel areas corresponding to the face color as the face area of the person object, defines the edge of the face area as the first boundary line 222, 224, and defines the first boundary line 222, 224 of the person object. Extract. The service server 100 may determine that the person is one object including the face area and the body area connected thereto. Accordingly, once the service server 100 detects a person object, it can continuously track the detected person object even if its size is reduced or moves. Additionally, even if the character turns his or her face or changes the shape of the face, the service server 100 can accurately determine the character object and its facial area.

When a plurality of characters appear in a broadcast video, the service server 100 may detect the plurality of character objects and extract the first boundary lines 222 and 224 of each character object. For example, in FIG. 4, two human objects may be detected, and the service server 100 may extract the first boundary line 222 of the face area of the first human object located on the left. The second person object located on the right can be detected, but the face is not visible, so the service server 100 continuously monitors the broadcast video until the face area of the second person object is determined and later determines the face area of the second person object. The first boundary line 224 can be extracted.

Next, the service server 100 receives service setting information from the user terminal 200 (S12). Service setting information may include, for example, broadcast video selection information, service participation information of multiple participants, information on the number of participants, special effects, etc. Specifically, it can proceed as follows. The user terminal receives a selection of one of a plurality of broadcast videos from an input means. In addition, the user terminal can receive information about service participation of multiple participants and/or the number of participants from the input means. Upon receiving special effect settings from the user terminal 200, the service server 100 later creates a composite video and converts it into, for example, black and white movie style, webtoon style, pop art style, vintage film style, funny cartoon style, etc. It can be provided.

Next, the service server 100 plays the previously selected broadcast video and its script on the display of the user terminal 200 (S14). Referring to FIG. 4, the user interface 210 of the user terminal 200 includes a video area 220 where a broadcast video is displayed and a script area 230 where a script is displayed.

The user terminal 200 creates an acting video by filming the participant's acting experience in accordance with the playback of the broadcast video (S16). Here, the acting video includes acting video and acting audio. The service server 100 receives the acting video from the user terminal 200.

The service server 100 detects the participant object appearing in the performance video and extracts the second boundary lines 242 and 244 of the participant object. Here, the second boundary lines 242 and 244 may be facial edges of the participant object. Referring to the user interface of the user terminal in which the acting video is played in FIG. 5, the service server 100 monitors each frame of the acting video to search for participant objects. Specifically, the service server 100 extracts the acting video frame by frame and analyzes the frame to detect edges. The service server 100 determines that pixel areas including pixels arranged consecutively and having similar pixel values among pixel areas divided by an edge are one participant object. The service server 100 determines the pixel areas corresponding to the face color as the face area of the participant object, defines the edge of the face area as the second boundary line 242, 244, and defines the second boundary line 242, 244 of the participant object. Extract. The service server 100 may determine one participant object including the face area and the body area connected thereto. Accordingly, once the service server 100 detects a participant object, it can continuously track the detected participant object even if its size is reduced or moved. Additionally, even if the participant turns his or her face or changes the shape of the face, the service server 100 can accurately determine the participant object and the facial area.

When a plurality of participants appear in the acting video, the service server 100 may detect the plurality of participant objects and extract the second boundary lines 242 and 244 of each participant object. For example, in Figure 5, two participant objects may be detected, and the service server 100 extracts the second boundary line 244 of the face area of the first participant object located on the left and the second participant object located on the right. The second boundary line 242 of the face area can be extracted. If the face area is not detected, the service server 100 may continuously monitor the performance video until the face area of the participant object is determined and extract the second boundary line of the participant object's face area.

If, among the service setting information, there is service participation information of multiple participants or the number of participants is multiple, the service server 100 matches a plurality of human objects detected in the broadcast video and a plurality of human objects detected in the acting video according to a predetermined matching standard. Match participant objects. The matching criteria can be set in advance by the user through the user terminal 200. For example, the user matches the first person object (the man on the left in Figure 4) and the second participant object (the man on the right in Figure 5) in advance, and the second person object (the woman on the right in Figure 4) and the first participant object. (the woman on the left in Figure 5) can be matched in advance.

Next, the service server 100 generates a composite video using the broadcast video and the acting video (S18). Specifically, the service server 100 performs image segmentation based on the first boundary lines 222 and 224 from the broadcast video to extract the background excluding the human object. The service server 100 extracts participant objects from the acting video by dividing the video based on the second boundary lines 242 and 244. The service server 100 generates a composite video by combining the extracted background and the extracted participant object. Next, the service server 100 generates a composite video by combining the composite video and acting audio.

If there is service participation information of multiple participants in the service setting information or the number of participants is multiple, the service server 100 extracts the background excluding the multiple human objects from the broadcast video, extracts the multiple participant objects from the acting video, and then extracts the multiple participant objects from the acting video. , A composite video is created by combining the background and multiple participant objects, reflecting preset matching criteria. For example, in Figures 4 and 5, the first person object (the man on the left in Figure 4) matches the second participant object (the man on the right in Figure 5), and the second person object (the woman on the right in Figure 4) matches the first person object (the man on the left in Figure 4). If the participant object (the woman on the left in Figure 5) is matched, the service server 100 reflects the preset matching criteria and inserts the second participant object into the area from which the first person object was removed from the background, and the second participant object from the background The first participant object is inserted into the area where the person object was removed. Next, the service server 100 generates a composite video by combining the composite video and acting audio.

Subsequently, the user terminal 200 may request the service server 100 to share the composite video. Specifically, the service server 100 sends the composite video to SNS (Instagram, Facebook, TikTok, Kakao Page, Twitter, Meta, etc.) and portal sites (Naver, Daum, Google, etc.) at the request of the user terminal 200. , it can be shared through at least one of a video sharing service platform (YouTube, etc.), an Internet media broadcasting platform (Afreeca TV, Kakao TV, etc.), and an email server.

Additionally, the user terminal 200 may request the service server 100 to share a moving GIF (animated GIF) generated based on a synthetic video. Specifically, the service server 100 can generate a moving GIF by combining a plurality of still images extracted from the synthetic video at the request of the user terminal 200, and can generate a moving GIF through SNS, portal sites, video sharing service platforms, and internet media. It can be shared through at least one of a broadcasting platform or an email server. Additionally, moving GIFs can be used as self-emoticons through an online emoticon store linked to the user terminal, and self-emoticons can include broadcast dialogue phrases. An animated GIF (animated GIF) is a GIF file that contains multiple images in a certain order in one file. The images may run in an infinite loop in turns or show several sequences and then stop the animation. You can.

The above-mentioned systems, means, modules, etc. are computer programs, for example, plural implemented by one or more processors stored or loaded in a computer readable storage medium. It may be composed of program modules. Additionally, each step mentioned in the flowchart may be implemented as a computer program, program module, or computer-executable instruction.

Although embodiments of the present invention have been described above with reference to the attached drawings, those skilled in the art will understand that the present invention can be implemented in other specific forms without changing the technical idea or essential features. You will be able to understand it. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive.

100: service server
200: user terminal
210: User interface
220: Video area
222, 224: first border
230: Script area
242, 244: Second border

Claims (3)

A service server that provides a broadcast experience service using multiple broadcast videos (each broadcast video includes broadcast video and broadcast audio):
Detecting a person object appearing in each broadcast video and extracting a first boundary line of the person object;
When the user terminal receives a selection of one of the plurality of broadcast videos, playing the selected broadcast video and its script on the display of the user terminal;
Receiving, from the user terminal, a acting video generated by filming the participant's acting experience in accordance with the playback of the broadcast video (the acting video includes acting video and acting audio);
detecting a participant object appearing in the performance video and extracting a second boundary line of the participant object;
After extracting the background excluding the person object by performing image segmentation based on the first boundary line in the broadcast video, and extracting the participant object by segmenting the image based on the second boundary line in the acting video, generating a composite video by compositing a background and the participant object; and
Including the step of the service server generating a composite video combining the composite video and the acting audio,
The service server extracts a plurality of still images from the composite video at the request of the user terminal and combines them to create a moving GIF, and the self emoticon used in conjunction with the user terminal is added to the moving GIF and the composite video. A method of providing a broadcast experience service including a related script. According to paragraph 1,
The first boundary line is a face edge of the person object, and the second boundary line is a face edge of the participant object. According to paragraph 1,
When the user terminal receives service participation from multiple participants,
The service server detects a plurality of human objects appearing in each broadcast video and extracts a first boundary line of each human object;
the service server detects a plurality of participant objects appearing in the performance video and extracts a second boundary line of each participant object;
The service server matches the plurality of person objects with the plurality of participant objects according to predetermined matching criteria;
The service server extracts a background excluding the plurality of person objects from the broadcast video, extracts the plurality of participant objects from the acting video, and then synthesizes the background and the plurality of participant objects by reflecting the matching criteria. A method of providing a broadcast experience service characterized by generating a composite video.

Images