KR102053128B1 - Live streaming image generating method and apparatus, live streaming service providing method and apparatus, live streaming system - Google Patents

Abstract

One aspect of the present invention discloses a live streaming video generation method in a client terminal. The method may include obtaining broadcast platform information associated with a live broadcast, generating a first streaming video based on the broadcast platform information, and displaying the first streaming video. Selecting, by user input, at least one of a sticker and an effect for decorating at least a portion of the displayed first streaming image and at least one of the selected sticker and effect in the first streaming image And generating a second streaming image by applying in real time, wherein at least one of the sticker and the effect is applied by paying a cost to a live broadcast server from a user account of the client terminal.

Description

LIVE STREAMING IMAGE GENERATING METHOD AND APPARATUS, LIVE STREAMING SERVICE PROVIDING METHOD AND APPARATUS, LIVE STREAMING SYSTEM}

The present invention relates to an image processing and service providing method based on a picture taken through a camera or a sticker added to a screen.

In recent years, broadcasts through the Internet as well as TV and cable broadcasts through the broadcast network has been activated so that viewers can experience broadcasting of various contents through the IP (Internet Protocol) network. In particular, in the case of Broadcast Jackey (BJ), which deals with interesting contents, many viewers are watching, but it is difficult to consider all of these parts in general personal broadcasting, although it is useful for advertisement effects. There is a problem that is not well utilized. In addition, there is a problem that it is difficult to utilize all of the various broadcasting special effects available in broadcasting stations in general personal broadcasting.

An object according to an aspect of the present invention is a live streaming video generation method and apparatus for enabling a broadcast jockey of a live broadcast service to perform rich rich media broadcasting using various types of real-time animation stickers and effects, and live services It provides a method and apparatus, and a live streaming system.

According to an aspect of the present invention, there is provided a method for generating a live streaming video in a client terminal, the method comprising: acquiring broadcast platform information associated with a live broadcast, based on the broadcast platform information; Generating a streaming video, displaying the first streaming video, and, by user input, at least one of a sticker and an effect for decorating at least a portion of the displayed first streaming video. Selecting and generating a second streaming image by applying at least one of the selected sticker and effect to the first streaming image in real time, wherein at least one of the sticker and the effect is applied to the client. Cost support from the user account on the terminal to the live broadcast server Can be made by fire.

The generating of the first streaming image may include photographing a subject through a camera and generating a first streaming image based on the photographed subject.

At least one of the stickers and effects may be at least one of (i) at least one of stickers and effects for decorating the subject, or (ii) at least one of stickers and effects for decorating at least one area of the first streaming image unrelated to the subject. It may include one.

The live streaming video generation method may further include transmitting the generated second streaming video to the live broadcast server, wherein the second streaming video transmitted to the live broadcast server is a client of a viewer who views the live broadcast. It may be transmitted to the terminal.

The second streaming video transmitted to the live broadcast server may be simultaneously transmitted to client terminals of a plurality of viewers in real time.

Processing of the sticker may be applied to the second streaming image by an input at the client terminal of the viewer, wherein the processing of the sticker may be made by payment from a user account associated with the client terminal of the viewer.

The sticker may include a moving sticker.

The effect may include at least one of audio, subtitles, text, graphics, 3D content, and video.

Live streaming image generating apparatus according to an aspect of the present invention for achieving the above object, the communication unit for communicating with a server, the first streaming generated based on the platform information obtained from the server through the communication unit a display unit for displaying a streaming image (the platform information includes broadcast platform information associated with a live broadcast) and a sticker for decorating at least a portion of the displayed first streaming image by a user input; A control unit for selecting at least one of effects and applying at least one of the selected sticker and effects to the first streaming image in real time, wherein the controller generates a second streaming image, wherein at least one of the sticker and the effect includes: Application is by paying the cost to the live broadcast server from the user account. It may be the lure.

According to another aspect of the present invention, there is provided a live streaming service providing method in a server device for providing a live streaming service, the method comprising: providing platform information associated with live streaming to a first client terminal; Receiving a second streaming video from a client terminal (The second streaming video is an image to which at least one of stickers and effects for decorating at least a portion of the first streaming video generated based on the platform information in real time And providing a second streaming image obtained from the first client terminal to a second client terminal, wherein application of at least one of the sticker and the effect is (i) a user account associated with the first client terminal. And (ii) associate with the second client terminal. At least one user account may be made from by the payment to the server apparatus.

The live streaming service may include a live broadcast service based on the first client terminal, and the second client terminal may be a terminal associated with a viewer entering a broadcasting station established by the first client terminal.

An apparatus for providing a live streaming service according to another aspect of the present invention for achieving the above object, the live streaming to the first client terminal through the communication unit and the communication unit for communicating with the first client terminal and the second client terminal; And transmitting the associated platform information and receiving a second streaming image from the first client terminal and providing the second streaming image to the second client terminal, wherein the second streaming image is generated based on the platform information. At least one of a sticker and an effect for decorating at least a portion of the streaming video, wherein at least one of the sticker and the effect is applied to (i) a user account associated with the first client terminal and (ii) the second client. Non at least one of the user accounts associated with the terminal Can be made by payment.

A system for providing a live streaming service according to another aspect of the present invention for achieving the above object, the first client terminal for generating a second streaming video based on the platform information associated with the live streaming service to provide to the server device A server device for providing the platform information to a first client terminal, receiving the second streaming image from the first client terminal, and providing the second streaming image to a second client terminal; and a second client terminal receiving the second streaming image. Includes, The second streaming video is an image to which at least one of the stickers and effects for decorating at least a portion of the first streaming video generated based on the platform information is applied, at least one of the stickers and effects applied (I) is associated with the first client terminal Charges from at least one of the user account and (ii) the user account associated with the second client terminal.

According to the live streaming video generation method and apparatus of the present invention, a live streaming service providing method and device, and a live streaming system, it is possible to generate additional revenue by utilizing a brand affiliate sticker through live broadcasting.

In addition, there is an effect of efficiently distributing sales by sharing the advertising sales from various brands and advertisers who want to promote advertising through live broadcasting by the live broadcast provider, BJ and sticker providers.

1 is a conceptual diagram schematically showing a live streaming system according to an embodiment of the present invention;
2 is a diagram illustrating a broadcast screen including a streaming video to which a sticker and / or effect is applied according to a live streaming video generating method according to an embodiment of the present invention;
3 is a flowchart schematically illustrating a live streaming video generating method and a live streaming service providing method according to an embodiment of the present invention;
4 is a conceptual diagram for describing in detail broadcasting platform information provided from a server to a client terminal;
5 is a detailed flowchart illustrating a method of generating a streaming video based on tracking information;
6 is a flow chart illustrating a method of applying a sticker associated with an advertisement and sharing advertising revenue based on the applied sticker;
7A and 7B are conceptual views illustrating examples of various kinds of stickers and effects;
8 is a block diagram showing a live streaming video generating device according to an embodiment of the present invention;
9 is a block diagram showing an apparatus for providing a live streaming service according to an embodiment of the present invention;
10 is a block diagram showing a live streaming system according to another embodiment of the present invention;
11 is a flowchart schematically illustrating a method of generating a streaming image based on a sticker directly created by a user in a live streaming image generating apparatus according to an embodiment of the present invention;
12A is a view showing a screen of an actual application for implementing a sticker authored by a user;
12B is a conceptual diagram illustrating a first embodiment in which a sticker design element displayed on the screen of FIG. 12A is associated with a subject;
FIG. 12C is a conceptual diagram illustrating a second embodiment in which a sticker design element displayed on the screen of FIG. 12A is associated with a subject.

As the present invention allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description.

However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component. The term and / or includes a combination of a plurality of related items or any item of a plurality of related items.

When a component is referred to as being "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but it may be understood that other components may be present in between. Should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that there is no other component in between.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art, and shall not be construed in ideal or excessively formal meanings unless expressly defined in this application. Do not.

Hereinafter, with reference to the accompanying drawings, it will be described in detail a preferred embodiment of the present invention. In the following description of the present invention, the same reference numerals are used for the same elements in the drawings and redundant descriptions of the same elements will be omitted.

Throughout this specification, sticker / effect means both information in a digitized form for decorating a display image, and when only the term sticker is used, effects can also be considered together with one of ordinary skill in the art. Will be self explanatory.

The live streaming video generating device may be referred to as a client terminal and a live broadcast streaming video generating device, and the live streaming service providing device may be referred to as a server device and a live broadcast service providing device. In addition, providing a selection menu including a sticker candidate and an effect candidate for applying a sticker / effect, applying a sticker / effect according to the selection in the selection menu, and generating a streaming image based on the sticker / effect application is performed at the client terminal. It may be performed in the server device.

In an embodiment of the present specification, the sticker / effect is based on being used for the broadcast service, but is not necessarily used only for the broadcast service. It is also applicable to general communication methods. In the broadcast service or other service, the sticker / effect may also be utilized in the form of a virtual advertisement inserting a virtual image into computer graphics.

1 is a conceptual diagram schematically showing a live streaming system according to an embodiment of the present invention. As shown in FIG. 1, a live streaming system according to an embodiment of the present invention may include a first client terminal 110, a server device 120, and a second client terminal 130.

Referring to FIG. 1, the first client terminal 110 is a terminal of a broadcast jockey leading the streaming in the IP network based broadcasting system described above. That is, the platform information is received from the server device 120 to perform a live personal broadcast, and a streaming image associated with the live personal broadcast is generated and transmitted to the server device 120. The platform information may basically include basic information for generating a streaming video. The platform information may include broadcast platform information. The first client terminal 110 may be called a broadcast jockey (BJ) terminal. The first client terminal 110 creates an environment for personal broadcasting based on the broadcasting platform. The generated environment is basically a video-based broadcasting environment, and may include a form in which a broadcast jockey is used as the host. That is, there may be a moving subject such as a person, an animal, or a car. However, it is not necessarily limited to this form. That is, the first client terminal 110 generates a first streaming image (referred to as a basic streaming image) by photographing a stationary or moving subject based on a photographing means such as a camera linked with the terminal 110. Alternatively, the first streaming image may be generated based on a previously stored image and / or an image received in real time or non-real time from another terminal without any special photographing means.

The first client terminal 110 may further enrich the broadcast content of the broadcast jockey using various broadcast related control means existing in the broadcast platform. Means for enriching broadcast content may include stickers and / or effects. The sticker may include elements associated with the design and design for decorating a subject photographed by the camera and displayed on the display means, a display image including the subject, and / or at least a portion of the first streaming image generated based on the platform information. Inclusive means. For example, it may include designed objects such as a fur hat shape, a crown shape, glasses shape, a headband shape, various shapes, and an animated figure that are overlaid on a subject. The sticker may exist at a specific position in the image without moving with respect to the subject, or in the form of moving with the movement of the subject in association with the subject. In addition, the sticker may include a moving sticker and a non-moving stop sticker, such as animation, and may also include a sticker implemented in 3D. When the sticker / effect is selected, the sticker may be used in a form that is continuously applied or may be used in a form of expressing and disappearing a specific emotion. The effect may include at least one of a sound-based audio effect, a subtitle, text, a figure, a sound, a functional figure capable of calling a specific animation, and a video. Stickers and effects may be applied simultaneously, and a plurality of stickers and effects may be applied.

The first client terminal 110 selects a sticker and / or an effect to be applied to the first streaming image among the plurality of sticker candidates and / or effect candidates. Here, the sticker candidate and the effect candidate may refer to self-selectable stickers / effects stored in the sticker storage (or inventory) before being selected as the sticker / effect that is actually applied. The sticker candidate and effect candidate are displayed on the sticker / effect selection menu screen, and at least one of the sticker / effect sets including the displayed sticker candidate and effect candidate may be selected and applied to at least a portion of the display image (first streaming image). have. A second streaming image is generated based on the applied sticker and / or effect. The generated second streaming image is provided to the server device 120. The first client terminal 110 may transmit a basic streaming image (first streaming image) to which the sticker / effect is not applied and a second streaming image to which the sticker / effect is applied, to the server device 120.

The server device 120 is a terminal that manages a live broadcast service. The server device 120 receives a streaming video from the first client terminal 110 that executes personal broadcasting and broadcasts it to the unspecified second client terminals 130. The server device 120 may manage not only the broadcast data but also basic broadcast platform information for broadcasting. The broadcast may be carried out through an application or other software program, wherein the broadcast platform information includes installation and management information of the application or software program. The server device 120 occupies a specific Uniform Resource Locator (URL), allocates a part of the URL occupied in response to a request from the first client terminal 110 to perform a personal broadcast, and broadcasts the broadcast station through the corresponding channel. To be able to open. In addition, by allowing other users (viewers) to enter the established broadcasting station, a smooth broadcasting between the broadcast jockey and the viewer is supported.

According to an embodiment of the present invention, the broadcast platform information includes information related to stickers and / or effects. It may also include policy information related to the calculation of advertising revenue associated with stickers and / or effects and the distribution of the calculated advertising revenue.

The second client terminal 130 connects to the server device 120, enters a number of personal broadcasting stations managed by the server device 120 (including a broadcast station established by the first client terminal 110), and enjoys the broadcast. It is a terminal. The second client terminal 130 may enter a URL-based personal broadcasting station provided by the server device 120 and watch the streaming video provided by the first client terminal 110. In this case, data may be exchanged between a plurality of viewers and broadcast jockeys using a chat window provided by the server device 120. In addition, it is possible to watch a variety of advertising content derived from it based on the stickers and / or effects applied to the streaming video during the broadcast, and also serves to generate advertising-related revenue, such as purchasing related products.

In an embodiment of the present invention, at least one of the first client terminal 110 and the second client terminal 120 may include a mobile station (MS), a user equipment (UE), a user terminal (UT), A wireless terminal, an access terminal (AT), a terminal, a fixed or mobile subscriber unit, a subscriber station (SS), a cellular telephone, a wireless device, a wireless communication device, a wireless transmit / receive unit (WTRU); Wireless Transmit / Receive Unit, mobile node, mobile, mobile station, personal digital assistant (PDA), smartphone, laptop, netbook, personal computer, wireless sensor, consumer electronics (CE) or other terms. May be referred to. Various embodiments of the terminal may be photographed such as a cellular telephone, a smart phone having a wireless communication function, a personal digital assistant (PDA) having a wireless communication function, a wireless modem, a portable computer having a wireless communication function, or a digital camera having a wireless communication function. Devices, gaming devices with wireless communications capabilities, music storage and playback appliances with wireless communications capabilities, internet appliances with wireless Internet access and browsing, as well as portable units or terminals incorporating combinations of such functions. However, the present invention is not limited thereto.

According to another embodiment of the present invention, the first client terminal 110 may not be the source image of the subject currently being photographed through the camera, but may be a sticker using the existing streaming or recorded video as a source. You can apply effects. For example, a sticker may be applied even when another terminal relays a streaming video.

In addition, as in the second client terminal 130, the viewer's terminal may be provided to another client terminal by applying a sticker / effect.

2 is a diagram illustrating a broadcast screen including a streaming image to which stickers and / or effects are applied according to a live streaming image generating method according to an embodiment of the present invention.

Referring to FIG. 2, the broadcast jockey may perform personal broadcasting through the first client terminal. The broadcast may be performed based on the image, and may be performed on both a mobile device and a desktop device such as a PC. Looking at the broadcast screen, the broadcast screen includes a portion where a streaming video is displayed, a chat window portion, and a portion for creating a chat message. In this case, the broadcast jockey may apply stickers and / or effects to the streaming video.

The left figure of FIG. 2 is a screen before applying stickers and / or effects, and the right figure of FIG. 2 is a screen to which stickers and / or effects are applied. When the user enters the sticker / effect application menu to apply stickers and / or effects, the sticker / effect related menu 210 is displayed. In this case, the selection menu 210 for applying the sticker and the icon for entering the selection menu 210 may be displayed only on the first client terminal interface separately from the viewer's screen. Here, various kinds of stickers / effects obtained from the server device are displayed. Numbers may be displayed around the sticker / effect. This is related to the amount that can be earned from advertising revenue when the sticker / effect is used. That is, when a sticker associated with a particular brand marked with the number "10" is used, a user (ie, broadcast jockey) of the first client terminal may obtain 10 points of advertising revenue from the brand. In this case, the 10 points may be 1,000 won according to a predetermined point-won exchange standard, or may be converted into a value of 10 won, 100 won, or the like. According to an embodiment of the present invention, there is a policy that the advertisement revenue-related points are obtained only by using the sticker / effect, or the sticker / effect is applied to the live broadcast streaming video through the use of the sticker / effect, and the applied sticker There may be a policy for the user of the first client terminal to acquire the point when there is a secondary action using the sticker / effect applied at the second client terminal (viewer) who saw the effect. To this end, monitoring of secondary actions in the second client terminal may be performed.

Referring to the right side of FIG. 2, the broadcast jockey may apply a fur hat-shaped general sticker 220-1: a sticker differentiated from an advertisement sticker. In this case, for example, an effect including snow and a snowman may be simultaneously applied to the sticker 220-1 as if winter is displayed at the lower end of the displayed image (first streaming image). Here, the lipstick sticker 220-2 of a specific brand may be selected and applied to be displayed on a specific position of the screen. The lipstick sticker 220-2 of a specific brand may be used as an advertisement sticker to trigger the display of related advertisement contents and / or related advertisement links in a circle 230 on the upper right side of the screen. The broadcast viewer may view advertisement contents and / or links related to the advertisement sticker or sticker displayed on the live broadcast streaming image of the broadcast jockey, and click around to go to a page guiding products related to the advertisement. At this time, the server device detects a secondary action on the viewer's sticker and distributes a point corresponding to each action to the broadcast jockey. The secondary operation may be classified into a shopping activity through the web such as a product purchase, a product tour, a product shopping cart, and the like, and the classified action may have points matching each action. The server recognizes a product associated with the advertisement and a viewer's action on the product so that the broadcast jockey can share the reward. The reward may be shared not only with the broadcast jockey but also with viewers entering the station.

According to an embodiment of the present invention, the particular sticker / effect included in the sticker / effect menu provided through the server device may include a plurality of sticker elements. That is, when one sticker is clicked and applied to a broadcast streaming image, 10 or more lip shapes may appear throughout the screen. That is, one sticker includes a plurality of sticker elements, but a sticker in which sticker elements are disposed over the entire screen may be utilized, such as disposing a first sticker element at a first position and a second sticker element at a second position. have. The information related to the sticker / effect may be retained by the server apparatus in the broadcast platform information, and may also be retained by the first client terminal that acquires the broadcast platform information and performs the broadcast.

According to an embodiment of the present invention, the broadcast streaming video is generated in a first client terminal. In some cases, in relation to the application of the sticker / effect, the server device may generate a streaming image to which the sticker / effect is applied. That is, the first client terminal generates a basic streaming image generated by photographing a subject with a camera or generated based on an image stored in advance or received from another terminal, and then transmits the generated basic streaming image to a server device and selects a sticker / effect. When a signal related to the first client terminal is provided to the server device, the server device parses a signal related to the sticker / effect selection, applies the selected sticker / effect to the basic streaming image, and applies the applied streaming image again to the first client terminal. And / or broadcast in the form of transmitting to the second client terminal.

According to another embodiment of the present invention, the sticker / effect information included in the sticker / effect selection menu included in the broadcast platform information provided to the first client terminal may be identification information of a user (ie, broadcast jockey) of the first client terminal. (Eg, a user account of the broadcast jockey) may be provided differently. That is, when the first user seeks personal broadcasting related to music, a sticker / effect set including a plurality of stickers / effects related to music may be provided as a selection menu. Or, in the case of pursuing personal broadcasting related to food, a menu including a plurality of stickers / effect sets associated with affiliates related to various food / desserts and dividends may be provided. That is, the sticker / effect information may be differently generated in consideration of the broadcasting characteristics of the user, and the sticker / effect information generated differently may be included in the broadcasting platform information and provided to the first client terminal. In addition, different stickers / effect sets may be provided for each IP and service type of the first client. In order to provide such a personalized sticker / effect, the data related to the user's broadcast or the history related to the use of the sticker / effect is stored in the server device, and the server device generates a customized sticker / effect through various statistical algorithms based on the stored data. It may be provided to the first client terminal.

According to another embodiment of the present invention, a screen for selecting a sticker / effect corresponding to the analyzed text and / or comment by analyzing text used in the chat window and / or the announcement used by the broadcast jockey is automatically displayed. You can also add the ability to be displayed. This may be achieved by pre-stored specific texts and comments on the server device, and automatically popping up a menu for selecting a sticker / effect corresponding to the stored texts and comments.

3 is a flowchart schematically illustrating a live streaming video generation method and a live streaming service providing method according to an embodiment of the present invention.

Referring to FIG. 3, in generating a live broadcast streaming image, the first client terminal 310 first acquires broadcast platform information from the server device 320 (S310). At this time, the second client terminal 330 participating in the broadcast as a viewer can also obtain the broadcast platform information from the server device 320 (S310). The broadcast platform information includes various types of information for broadcasting. This will be described in more detail with reference to FIG. 4.

4 is a conceptual diagram for describing in detail platform information provided from a server to a client terminal.

Referring to FIG. 4, platform information is provided from the server 410 to the client terminal 420. The client terminal 420 can include a first client and / or a second client. The platform information includes information for installing an application or program associated with streaming on the terminal. The platform information may include broadcast platform information. The broadcast platform information includes information for installing an application or program related to a broadcast in a terminal. Accordingly, the terminal installs an application / program that enables the terminal to execute a broadcast based on the information.

The platform information includes a tracking algorithm. The tracking algorithm includes a face tracking algorithm. That is, the tracking algorithm is an algorithm for acquiring information for following a movement of a moving subject, and specifies at least one feature point of a specific subject (for example, a face, a mouth, a nose, etc.) to which a specific sticker is applied, and specifies An algorithm for obtaining information (eg, movement of coordinates) related to the movement of at least one feature point. Based on the tracking algorithm, the movement of the moving object is detected, the moving distance of the sticker corresponding to the detected moving distance is calculated, and the sticker / effect is moved according to the calculated moving distance, so that the sticker / effect can be moved along the movement of the subject. You can make it move. However, since the tracking may be necessary or not depending on the characteristics of each sticker, the movement of the sticker according to the tracking information is preferably applied corresponding to the characteristics of the sticker.

Platform information includes various sticker / effect information. This includes not only all sticker / effect information stored by the server device but also sticker / effect set information customized for each user and service. This sticker / effect information is continuously updated as new information associated with the sticker / effect is generated. In addition, it is continuously updated as statistical analysis, such as the user's usage history, is performed. Platform information including updated sticker / effect information may be periodically provided to each terminal. In addition, the sticker / effect information includes information related to the activation of the sticker / effect selection menu corresponding to the detection of a specific text or a specific comment according to voice recognition. Further, according to another embodiment of the present invention, in relation to broadcast platform information, the sticker / effect set is configured based on the advertisement sticker / effect associated with or corresponding to the corresponding position based on the position information of the first client terminal. Can be. That is, the broadcast jockey at a specific position may transmit the broadcast platform information including the advertisement sticker / effect set corresponding to the specific position preset in the server to the broadcast jockey so that the location-based sticker / effect may be applied to the streaming video. have.

Alternatively, according to another embodiment of the present invention, based on the position of the viewer, that is, the second client terminal, the specific advertisement sticker may be applied to the preset position. The region-based sticker may be directly applied to the broadcast streaming image received by the second client terminal. Alternatively, the present invention may be applied to a streaming video relayed from the second client terminal to the third client terminal.

The broadcast platform information includes various functional information for performing a broadcast. This may include information such as chat, screen layout, management of viewers of each broadcasting station, and the like.

The broadcast platform information includes link detection function information, which indicates information related to a function of detecting, by a second client terminal, a viewer in response to a sticker / effect and using an associated advertisement product, advertisement content, or an advertisement link. . In other words, the secondary behavior of the viewer can be used as an important factor in the distribution of advertising revenue, the algorithm for detecting this can be important.

The broadcast platform information includes policy information related to cost sharing. This indicates information related to a policy relating to the in-broadcast use of the sticker / effect, such as whether points can be earned only by using a particular sticker, if there is an activity of a viewer associated with the particular sticker, or the like. Detailed policy information may be included.

Various information related to streaming and / or broadcast as described above, and information related to sticker / effect use in streaming and / or broadcast are included in platform information and provided to the client terminal 420.

3 again, the first and second client terminals 310 and 330 having acquired the platform information install the streaming and / or broadcast related application / program (S312). Then run the installed application / program.

First, in an embodiment of streaming associated with a broadcast, the first client terminal 310 opens a broadcast station to start personal broadcasting (S314). At this time, together with the establishment of the broadcast, the broadcast establishment report to inform the establishment of the broadcast is transmitted to the server (320). The broadcast personalization bulletin may include information such as a user account associated with the first client terminal 310, a total number of viewers who can be admitted, and a content type of what content is based. The broadcast establishment information may be provided to a terminal of viewers who want to enter a broadcasting station, that is, the second client terminal 330 (S318). This may be done in response to a request from the second client terminal 330. In response to requesting the information of the currently established broadcasting station, the information of the established broadcasting station is provided. The second client terminal 330 may enter a broadcasting station selected by the selected broadcasting stations (S330).

In operation S320, the first client terminal 310 that establishes a broadcast may photograph a subject using a camera interoperating with the terminal or a camera included in the terminal. In this case, the subject may be anything other than the user himself. Then, the captured image may be displayed through the display means of the terminal (S322). The display screen of the photographed subject is streamed and broadcasted. During the broadcast, if you want to apply stickers and / or effects, click on the icon that brings up a menu window for the selection of stickers and / or effects via the user interface. Then, various stickers / effects included in the broadcast platform information are displayed on the screen. In this case, a sticker / effect set corresponding to the user's account, the type of the terminal, and information related to the content type of the established broadcasting station may be generated and customized. The first client terminal 310 selects at least one from the provided sticker / effect set (S324). The selected sticker / effect is applied to at least a part of the display image including the moving subject to generate a streaming image (S326). The streaming video to which the sticker / effect is applied is transmitted to the server 320 (S328), and the server 320 provides the streaming video to all viewer terminals 330 entering the corresponding broadcasting station (S334).

5 is a detailed flowchart illustrating a method of generating a streaming image based on tracking information.

Referring to FIG. 5, when the first client terminal selects and applies a sticker / effect, it may be applied in a stationary state, but may be applied in a moving form according to a moving subject. To this end, first, when a sticker / effect to be applied in the first client terminal is selected (S510), the first client terminal executes a tracking algorithm included in the broadcast platform information to track the movement of the subject (S520). In order to track, a tracking target must first be determined, which can be specified based on the target site to which each individual sticker is applied. For example, since the hat-shaped sticker is applied to the head of the human body, the top head of the face of the person is determined as the tracking subject through face tracking. In this manner, when the tracking target subject associated with the application portion of each sticker is determined, the feature points of the determined target subject are extracted. The extraction feature point may be preset in the tracking algorithm. For example, in a mesh-based face tracking, a point at a predetermined specific position of an outline based on a mesh of an upper portion is extracted as a feature point. Then, the sticker / effect is applied based on the corresponding feature point, and since the movement of the subject means the movement of the feature point, tracking information is obtained by calculating a movement path of the feature point (S530). In operation S540, the sticker / effect may move naturally according to the movement of the subject when the selected sticker is applied while the selected sticker is moved along the movement path of the feature point of the target subject based on the obtained tracking information. In this way, the streaming image may be generated based on the sticker applied while tracking the movement (S550).

6 is a flowchart illustrating a method of applying a sticker associated with an advertisement and sharing advertisement revenue based on the applied sticker.

Referring to FIG. 6, the first client may select a sticker / effect to be applied to the broadcast streaming image (S610) and apply it (S620).

In this case, it may be checked whether the applied sticker / effect is related to the advertisement (S630). Relevant to the advertisement means that the sticker may be a sticker created in partnership with a specific business (eg, a brand). As such, the sticker may include information about it in association with the product of a particular business. In other words, when a sticker is clicked, a linking operation such as a product of a business, a business site, or a related advertisement video is played may occur.

If there is no connection with the advertisement, the application of the sticker / effect is performed to entertain viewers, thereby achieving the purpose of applying the sticker / effect. However, if it is related to the advertisement, the revenue share according to the advertisement effect may be additionally considered. The association with the advertisement can be verified by parsing the individual feature information of each sticker. In some cases, a flag can be assigned to display an advertisement-based sticker / effect as "1" and a sticker / effect as "0". have.

If the applied sticker / effect is related to the advertisement, the policy information related to the revenue distribution of the advertisement is parsed. In this case, when the revenue is distributed along with the use, the broadcast jockey may obtain a partial fee only by using the advertisement-based sticker in the first client terminal. In this case, since the more advertising viewers enter the corresponding broadcasting station, the advertising effect is maximized, and the fee may be determined in consideration of the number of viewers entering the broadcasting station at the time of use of the sticker. For example, a sticker of "10" points may be set at 100 points if there are 10 viewers and 1000 points if there are 100 viewers and provided to the broadcast jockey as a fee.

However, when the fee is set based on the second act of the viewer of the corresponding broadcasting station without using the fee at the same time as the use, the act of detecting the terminal associated with the applied sticker / effect is detected (S640). This can be done in the server device. When an advertisement video is played by clicking on a sticker applied by a viewer's terminal, a product purchase, a product tour, or a shopping cart is performed through a site connected with the brand of the sticker, the corresponding action is performed according to the form of subsequent operation. Rewards can be made to the broadcast jockey. Alternatively, a fee may be set in consideration of the number of click operations. That is, the amount of at least a portion of the advertising revenue determined through various methods as described above is allocated to the account associated with the broadcast jockey account (S650). In other words, Broadcast Jockey considers the allocation of these fees as incentives, and uses advertisement-based stickers as much as possible at the appropriate time, thereby allowing businesses associated with advertisements to enjoy various advertising effects.

7A and 7B are conceptual views illustrating examples of various kinds of stickers and effects.

Referring to FIG. 7A, a sticker 710 covering a specific portion of a streaming image may be used. For example, in the case of a broadcast of a broadcast jockey that gives pleasure through makeup, a sticker 710 covering the face of the face may be provided. In this case, the first client terminal may determine a position to cover while directly moving the sticker 710, the coordinate information of the sticker 710 determined through the first client terminal may be transmitted to the server. In this case, basic streaming video information without applying the sticker 710 may also be delivered to the server.

Referring to FIG. 7B, the sticker 710 covering the specific position is basically for erasing the hidden sticker area. However, at this time, since the sticker area is erased to all viewers when the sticker area is erased, the sticker 710 may be used in a form in which a certain area of the sticker 710 disappears according to the click of the viewer. That is, when a particular viewer pays three times, a part of the sticker 710 may be erased by clicking on a desired portion. That is, the three erased regions 720 may be secured by clicking the eye and nose portions 720 three times. Since the server device also holds the basic streaming image, it can be achieved by replacing only the sticker area of the corresponding portion with the basic streaming image. Alternatively, the sticker 710 may be generated by the server device on the basic streaming image, placed on the subject, and the partial area of the uploaded sticker 710 may be erased. In this case, the size of the erased area and the click cost for the erased area may be determined by the broadcast jockey and / or the server. In addition, for the part covered by the sticker, it is possible to apply a sticker in the form of changing to another person's face. When the face change sticker is applied to the face portion of the subject of the first client terminal, a menu for selecting a face of another person to be changed is displayed, and through the menu, the user of the first client terminal makes a picture of another person or An image can be selected, and the face part of another person is extracted from the selected picture or image. Then, the face part of the position where the face change sticker exists may be deleted, and the extracted face of another person may be changed. Alternatively, a sticker in the form of another person's face may be provided in advance in the menu to directly change the face of the sticker.

Based on this function, each viewer can obtain different streaming video through different click counts. This can induce viewers to spend and allow the spent expenses to flow into the broadcast jockey (it is possible for the service provider to earn some fees on the server device), giving broadcast jockey incentives to make the broadcast more fun. It can give an effect.

8 is a block diagram showing an apparatus for generating live streaming video according to an embodiment of the present invention. As shown in FIG. 8, the apparatus for generating a live streaming image according to an exemplary embodiment of the present invention includes a communication unit 810, a camera 820, a display unit 830, a controller 840, an input unit 850, and a memory ( 860 may be included.

Referring to FIG. 8, the communication unit 810 has a function of performing communication with a server device and another device. The communication unit 810 includes equipment for wireless or wired communication. For example, it may include an antenna and / or a communication processor.

The camera 820 is a device for photographing a subject, and the controller 840 processes an image captured by the camera 820. The camera 820 may exist inside the live broadcast streaming image generating device or may be a device connected to the outside through an interface such as a high-definition multimedia interface (HDMI).

The display unit 830 outputs an image processed through the control unit 830. The display unit 830 may be implemented as a monitor, a touch screen, and a TV receiver.

The controller 840 may include a broadcast app execution unit 842, a tracking information generator 844, an application unit 846, and a streaming image generator 848.

The broadcast app execution unit 842 installs and executes an application / program associated with streaming based on platform information received through the communication unit 810. In the case of broadcast streaming, the broadcast app execution unit 842 installs a broadcast application / program based on the broadcast platform information, and executes the broadcast application / program by input through the input unit 850. In this case, a program entry operation such as login may be performed.

The tracking information generator 844 generates tracking information according to a tracking algorithm included in the platform information and provides the tracking information to the application unit 846.

The applying unit 846 performs a function of applying a sticker / effect. The application unit 846 displays a suitable sticker / effect set based on at least one of an account of a user of the first client terminal entering the broadcast application / program, an IP of the first client terminal, and broadcast service content and type. 830). Then, the input associated with the user's selection is input through the input unit 850 and the selected sticker / effect is applied to one area of the target subject or the display screen (first streaming image). At this time, the sticker / effect may be applied according to the movement of the subject based on the tracking information.

The streaming image generator 848 processes the image photographed by the camera 820 to generate a basic streaming image. Here, encoding and decoding can be performed. When the sticker / effect is applied, the second streaming video to which the sticker / effect is applied may also be processed.

The input unit 850 is an interface for processing user input, such as a keyboard, a mouse, and a touch pad.

The memory 860 stores instructions (or instructions) related to each function performed by the controller 840. The memory 860 includes instructions based on broadcast platform information, and information received from a server device or another device through the communication unit 810 may be stored.

9 is a block diagram illustrating an apparatus for providing a live streaming service according to an embodiment of the present invention. As shown in FIG. 9, the apparatus for providing a live streaming service according to an embodiment of the present invention may include a communication unit 910, a controller 920, an input unit 930, and a memory 940.

The communication unit 910 is a device that is responsible for communication with the first client terminal and the second client terminal.

The controller 920 is a component for controlling an application, and includes an account manager 921, a broadcast platform manager 923, a streaming image processor 925, an application determiner 927, and an advertisement revenue processor 929. It may include.

The account manager 921 manages information related to an account for logging in to an application (eg, a broadcast application). In addition to the personal information of each account, it is possible to manage the broadcast-related history performed in each account. For example, information such as opening time of a station opened through a specific user account, opening format, contents of handling, type of service, information related to entry into a station opened by another user, and / or a history of stickers / effects selected and applied. Can manage.

The broadcast platform manager 923 manages platform information (including broadcast platform information) of FIG. 4. This includes periodic upgrades to the software as well as the creation of custom stickers / effect sets based on the user's broadcast usage history.

The streaming image processor 925 processes the first streaming image and / or the second streaming image received from the first client terminal. The streaming video processor 925 may receive basic streaming information (first streaming video) that is not applied to the sticker / effect and streaming video (second streaming video) to which the sticker / effect is applied, from the first client terminal. Can be processed independently For example, in the state of receiving the basic streaming video information, by receiving a signal related to the request to apply a specific sticker and parsing the received signal, the requested sticker can be directly applied to the basic streaming video to directly generate a streaming video to which the sticker is applied. have.

In addition, by processing the request associated with the sticker in the second client terminal separately, the image processing is performed so that different streaming images can be viewed in relation to the area of the sticker in different terminals.

The applicability determining unit 927 determines whether the sticker for advertisement is applied as well as whether the sticker is applied. In the case of advertising stickers, it is determined which business (or brand) is associated with.

The advertisement revenue processor 929 distributes the advertisement revenue associated with the sticker. In particular, in the case of advertising stickers, viewers who have seen the advertising stickers (viewers entering the corresponding broadcasting station) detect an arbitrary operation linked with the corresponding brand, and then convert the revenue according to the advertising revenue into the broadcast jockey and the live broadcasting service providing server (and And / or a sticker providing server (see FIG. 10)). At this time, the distribution of advertising revenue includes cost per click (CPC) method, pay per click (PPC) method, cost per time (CPT) method, cost per mile (CPM) method, and CPM Per-Millenium (CPI), Cost per Impression (CPI), Cost Per Conversion (CPCon) may be used.

The input unit 930 is an interface for processing user input for changing various user settings associated with a broadcast application.

The memory 940 stores instructions associated with each function performed by the controller 920. For example, user history information associated with the account, usage history information of the sticker, the sticker itself information (such as whether it is an advertisement sticker, what operator and which link is associated with the advertisement sticker), and policy information related to the distribution of advertisement revenue. It may include.

10 is a block diagram illustrating a live streaming system according to another embodiment of the present invention. As shown in FIG. 10, the live streaming system according to another embodiment of the present invention includes a first client terminal 1010, a server 1020, a second client terminal 1030, a sticker providing server 1040, and an advertiser server. 1050 may be included.

Referring to FIG. 10, operations of the first client terminal 1010, the server 1020, and the second client terminal 1030 are similar to those described above.

The sticker providing server 1040 may provide a face tracking software development kit (SDK) to the live broadcast server 1020 and apply it to the web or app clients 1010 and 1030 of the live broadcast service. The first client terminal 1010 opens a sticker inventory on a broadcast service used by the broadcast jockey so that at least one of them may be selectively used. For various live broadcast services, various stickers stored in a sticker archive of the sticker providing server 1040 may be provided at the same time.

The sticker providing server 1040 may provide and operate a sticker on a medium desired by advertisers, such as an Ad network. In addition to advertisement stickers (brand affiliated stickers), the server 1030 may provide contents such as a background sound source, a professional effect, a subtitle function, and an intermediate advertisement, to be used in real time. The server 1030 may provide the information provided in this way to the client terminals 1010 and 1030.

In order to implement such an action, the sticker providing server 1040 provides the face tracking SDK and the user use tracker to the live broadcast server 1030, and the live broadcast server 1030 provides information on the user usage data and information related to the function promotion. It may be provided to the providing server 1040. The two can then share advertising revenue together.

As described above, the sticker can be applied in both web and mobile environments, and broadcast jockey can maximize the fun of broadcasting by showing a sticker-applied screen to viewers who are watching the broadcast. In particular, in the case of utilizing the brand affiliate sticker, viewers who saw the brand affiliate sticker detect an arbitrary action linked to the brand, and then convert the revenue according to the advertising revenue into the broadcast jockey, the live broadcast service provider (ie, the server 1020) and Sticker providing server 1040 may be shared together. At this time, the distribution of advertising revenue includes Cost Per Click (CPC) method, Pay Per Click (PPC) method, Cost Per Time (CPT) method, Cost Per Mile (CPM) method, CPM (Cost) Per-Millenium (CPI), Cost per Impression (CPI), Cost Per Conversion (CPCon), etc. may be considered.

According to another embodiment of the present invention, when distributing advertisement revenue, the server apparatus may distribute the revenue to an account related to the broadcast performer appearing on the broadcast through the first client terminal and / or to an account of broadcast viewers. That is, when an advertisement sticker is used for a broadcast performer, information about the broadcast performer to which the sticker is applied may be obtained and the portion of the advertising revenue may be distributed to an account associated with the obtained information. In addition, at the time when the viewer watches the advertisement sticker of the streaming video, it may leave a log and share some of the advertisement revenue.

FIG. 11 is a flowchart schematically illustrating a method of generating a streaming image based on a sticker directly created by a user in the apparatus for generating a live streaming image, according to an exemplary embodiment.

Referring to FIG. 11, the client terminal displays an object photographed by a camera by executing an application related to broadcasting (S1110). In operation S1120, the sticker / effect application menu is started and a sticker authoring tool is executed from the menu. The sticker authoring tool may be provided as a function of the sticker / effect application menu of the broadcast application. When the user selects the sticker authoring tool, the terminal displays a menu for sticker authoring on the display screen. According to another embodiment of the present invention, the sticker may be authored using a photo or a video already stored in the terminal rather than being photographed.

Then, the mode for sticker authoring is selected (S1130). Here, the mode for authoring may include two modes, a first mode and a second mode.

The first mode is a mode for recognizing a subject such as a face, and is a mode for associating a sticker design element generated by a user with at least one of the displayed subjects. In this case, the outline of the subject may be considered as an element for determining the relationship with the sticker design element. The outline of the subject may be identified using a previously stored outline template. For example, when the subject is a face, a contour template including overall layout information of a plurality of components such as eyes, nose, and mouth in the face is stored in advance, and the outline of the face and the components in the face is displayed on the displayed screen. Matching with a pre-stored contour template, it recognizes that the face first, and identifies and stores the contour information of the recognized face. In this case, information such as brightness, motion, color, and eye position estimation may be used to distinguish the face from the background. In this case, it may be efficient to analyze the main part of the face and discard the fine error and compare only the large feature with the stored face template. In embodiments of the present invention, the contour template is not necessarily related to the face, but may also be related to other parts of the human body such as arms, legs, hands, and the like.

In the case of the first mode, the sticker design element generated by the user may also move and change in size in response to the movement of the subject and / or the change in the size of the subject.

The second mode is a mode of designing the display to be displayed at a specific position in the display screen, that is, with respect to the displayed screen without considering the association with the subject. In the second mode, the sticker design element may be displayed at a fixed size in a predetermined position on the display screen without being affected by the movement of the subject. In some cases, even in the second mode, a moving effect such as a change in size may be provided by the user's setting.

When the mode selection is completed, the user receives a user input for sticker design through a user interface (eg, a touch screen, a keyboard, a mouse, etc.) (S1140). The user input for sticker design may be text input and may be input via a drawing mode. The drawing mode may be called a doodling mode. The sticker design element may be generated by inputting text through typing using the keyboard, and the sticker design element may be generated by inputting through a mouse or a touch screen based on a line or a figure selected by the user in the drawing mode.

According to an embodiment of the present invention, in addition to the text and drawing, a figure and an image may be called. The figure may comprise a still figure, an animation figure, or a 3D object, and the image may be imported from a camera or album and used as a sticker design element. The figures may include solid lines, triangles, rectangles, arrows, broken lines, half moons, clouds, hearts, equation-related figures, flowchart-related figures, and the like. Such figures may be stored in a template in advance.

According to another embodiment of the present invention, other effects, such as a sound and a functional figure capable of calling a specific animation, may also be utilized as a sticker design element.

The client terminal identifies in-screen coordinate information of each sticker design element in order to store the sticker design element thus input as a user authoring sticker (S1150). Coordinate information is recognized by identifying the coordinates of the pixel associated with the sticker design element on the displayed screen. The identified coordinate information may be utilized as relationship information indicating association with the display screen or the subject according to the mode selected by the user.

After identifying the input coordinates of the sticker design element, it is determined whether the first mode or the second mode (S1160). In the case of the first mode (subject recognition mode), the image may be associated with the outline of the subject, and the second mode may be associated with the display screen. Switching to the first mode and the second mode is possible at any time before the sticker is stored, and thus, upon detecting the mode switch, all sticker design elements interpreted as the second mode can be interpreted as the first mode. That is, the sticker design elements interpreted through step S1165 may be interpreted through steps S1170 to S1190 and vice versa.

When the first mode is selected in step S1160, first, the outline of the subject is identified using the outline template (S1170). The terminal stores a figure (eg, a rectangle) including all the sticker design elements as an image and recognizes the subject in the stored image to identify the outline of the identified subject. In this case, a plurality of subjects may be present in the image. For example, the face may be the subject, or another part of the person, such as a hand or an arm, may be the subject. The client terminal recognizes the subject based on the previously stored outline template of the subject and identifies the outline of the subject. At this time, the coordinate information of the pixel related to the contour may be checked.

Then, based on the coordinate information of the sticker design element identified in step S1150, the relationship information relating the contour of the identified subject and the sticker design element is generated (S1180). According to an embodiment of the present invention, the center point of the subject is found based on the outline of the subject included in the stored image. Then, based on the distance from the center point to the outline of the subject, the relationship between the sticker design element and the subject may be determined in consideration of the outline of the subject or the distance from the center point to the sticker design element. Then, relationship information may be generated based on the identified relationship.

According to another embodiment, in step S1180, a point corresponding to the sync point is found on a previously stored contour template. The sync point is a point for synchronizing the outline of the currently displayed real subject, the sticker design element, and the previously stored outline template, and may be specified as a specific point on the outline of the subject. For example, a specific point on the outline of the subject having an association with the end of the sticker design element becomes a synchronization point. Alternatively, the starting point and / or the end point of the sticker design element and the closest point on the contour may be the synchronization point. In some cases, the point where the sticker design element meets the contour may be a synchronization point. As described above, there are various algorithms for determining the synchronization point, which may be included in the broadcast platform information received from the server device. The synchronization point on the contour of the actual subject corresponds to a specific point on the contour template matching the subject and may be appropriately applied to another subject to which the contour template is applied. The synchronization point thus generated may be considered as relationship information.

When generating the relationship information, the client terminal stores the user authoring sticker in the local area based on the relationship information and the sticker design element (S290). At this time, the user may instruct the user to press the save icon to perform the save. In other words, while generating a user authoring sticker as a fun factor through the process from step S210 to step S1180, if it is determined that the user has created a sticker so that the user wants to register it as a sticker, press the save icon. You can register and save the current sticker. That is, the storage of the sticker may also be selectively determined by the preference of the broadcast jockey. Then, the stored user authoring sticker information and the streaming video to which the user authoring sticker is applied are transmitted to the server device. The server device receiving this may broadcast a streaming image to which the user authoring sticker is applied to the viewer client.

At this time, the sticker-related information stored and transmitted may include the relationship information, the outline of the subject and coordinate information of each sticker design element, and the sticker design element itself. Or related contour template information may be further stored. In this case, the relationship information may include sync point information (eg, a specific point on a subject or a point on a contour template corresponding thereto), and information (eg, distance information, etc.) indicating a relationship between the sync point and a sticker design element. . The self-information of the sticker design element may include information such as the type (the text or the drawing) of the sticker design element, and the color, the thickness, and the shape of the line constituting the sticker design element.

If the second mode is selected in step S1160, the user enters the second mode, and creates and stores a user authoring sticker in consideration of the position in the display screen based on the coordinate information of the sticker design element (S1165). In this case, the information related to the subject may not be stored, and only coordinate information in the display screen may be stored for association with the display screen.

The use of a pre-stored picture is irrelevant, but the point of identifying the outline of the subject and the pixel coordinates of the sticker design element can be important because the subject can continue to move and change size during shooting. According to an embodiment of the present invention, the time point for identifying the outline of the subject and the coordinates of the sticker design element may include a moment when a user input for generating the sticker design element starts or a time when the user input is completed. The projectile and the sticker design element are identified according to the corresponding time point, and the sticker design element also changes in response to the change of the subject after the corresponding time point.

12A illustrates a screen of an actual application for implementing a sticker authored by a user.

Referring to FIG. 12A, a tool for generating a user authoring sticker includes an icon 1202 for selecting a first mode and a second mode and an icon 1204 for selecting a type of a sticker design element at the top of a broadcast screen. Display. When the icon 1202 is pressed, the sticker moves in association with the outline of the subject, and when the icon 1202 is not pressed, the sticker does not recognize the outline of the subject because there is no associated operation with the subject. The embodiment of FIG. 12A assumes a situation in which the first mode is selected.

In addition, when the icon 1204 is selected, text input is possible, and when the icon 1206 is selected, user input is possible in the drawing mode. The embodiment of FIG. 12A assumes a situation in which a drawing mode is selected.

In the drawing mode, the user may select the thickness and shape of the line through the menu 1220 for selecting a basic line for drawing on the left side of the screen. The user can arbitrarily change the thickness and shape of the line using the menu 1220 while drawing. For example, in relation to the thickness of the line, a menu for selecting a plurality of thicknesses from the thickest line at the top to the thinnest line at the bottom may be provided. In addition, the menu for selecting the shape of a line (menu for selecting a pen type or a brush type) includes a triangular line, a square line, a circular line, a heart line, or a mixture of two colors, and a three-dimensional line. You can include options such as stereoscopic lines and shadowed lines.

In addition, a menu 1230 for selecting a line color may be provided at the bottom of the screen. The user may select one of the plurality of colors provided to perform the drawing.

In the embodiment of Figure 12a, the user is photographing the upper body torso including the face, the client terminal displays it. The broadcast jockey generates a plurality of sticker design elements 1210-1-1210-7 by drawing with a thickness, a shape, and a color of the selected line through the menu 1220 and the menu 1230 while being photographed in real time. Whenever there is an input for each sticker design element 1210-1-1210-7, different lines are available through the menu 1220 and the menu 1230. Overall, sticker design elements 1210-1 and 1210-2 are formed on the top of the face to form a rabbit ear shape. The sticker design elements 1210-3 have a heart shape and are formed near the nose, and the sticker design elements 1210-4-1210-7 have a beard shape in the form of a divergence around the nose.

In detail, the client terminal recognizes the sticker design elements 1210-1 to 1210-7 and stores the sticker design elements 1210-1 to 1210-7 as a whole user-authored sticker. Recognized individually, each sticker design element is associated with a subject. That is, the subjects associated with the sticker design elements 1210-4 to 1210-7 may be different from each other. The sticker design element may be recognized as a figure composed of one connected line and typed at a time.

12B is a conceptual diagram illustrating a first embodiment in which a sticker design element displayed on the screen of FIG. 12A is associated with a subject.

Referring to FIG. 12B, when a sticker design element is generated by a user input, the terminal stores the figure 1240 including all the sticker design elements in an image form. Then, the subject included in the figure 1240 is identified. In the embodiment of FIG. 12B, a face may be identified by a subject in the image. In this case, the subject may be recognized as a face by matching the outline of the subject and a pre-stored face template, and may obtain face outline information based on the subject. When the face outline is obtained, the coordinates of the center point 1242 of the face may be secured based on the face outline information. Then, based on the relationship between the coordinates of the face center point 1242 and the outline of the face, relationship information between the subject (face) and each sticker design element 1210-1 to 1210-7 is generated. For example, the relationship between the distance between the face center point 1242 and the face contour and the distance between the face center point 1242 and each sticker design element 1210-1-1210-7 (including d1, d2, d3, and d4) is related. Can be generated as information. Alternatively, the distance between the face contour and each sticker design element 1210-1 to 1210-7 may be considered as relation information. It may be desirable to consider the start and end points of each sticker design element 1210-1-1210-7 when calculating the distance to each sticker design element 1210-1-1210-7.

If the face contour is changed based on the generated relationship information (for example, when the shape of the face changes or the size of the face changes), the distance between the center point and the face contour also changes, so each sticker design element is proportional to the changed distance. The distance from 1210-1 to 1210-7 may also change.

FIG. 12C is a conceptual diagram illustrating a second embodiment in which a sticker design element displayed on the screen of FIG. 12A is associated with a subject.

Referring to FIG. 12C, the sticker design element 1210-1 begins inputting from the starting point 1250-1 and ends at the end point 1250-2. The client terminal may recognize the start point 1250-1 and the end point 1250-2 to identify a subject to be associated. In this embodiment, the "face" closest to the sticker design element 1210-1 and having a high degree of matching is selected as the object to be associated. In this case, the subject is recognized as a face by matching the outline of the subject with a pre-stored face template, and obtains face outline 1212 information based on this. The client terminal looks for sync points to clarify the relationship with the subject, ie, the facial contour 1212, that is associated with the sticker design element 1210-1. The sync point is a point on the face contour 1212 and preferably has a specific relationship with the start point 1250-1 and the end point 1250-2. For example, a point closest to the start point 1250-1 and the end point 1250-2 may be preferable. Alternatively, a point on the contour 1212 that meets the sticker design element 1210-1 may be a synchronization point. The information related to the sync point includes coordinate information indicating which point on the outline of the subject, and the sync point related information is also information stored in the terminal when the user author sticker is stored.

In this embodiment, the points 1260-1 and 1260-2, which are points on the face contour 1212 closest to the start point 1250-1 and the end point 1250-2, may be detected as sync points. .

After detecting the synchronization points 1260-1 and 1260-2, the client terminal calculates distances d1 ′ and d2 ′ between the face contour 1212, the start point 1250-1, and the end point 1250-2. This can be stored as relationship information. That is, even if applied to the other face, it is possible to reproduce the authored stickers in such a way that the sticker design element is generated at a position separated by distances d1 'and d2' based on the synchronization point. In this case, when the size of the face changes, the distances d1 'and d2' may also be reduced or expanded in proportion to the changed size. That is, even if the shape or scale of the face is changed, the sticker design element may be appropriately applied by finding a coordinate corresponding to the coordinate of the synchronization point of the current face.

In addition, by identifying the coordinates of all the pixels displayed on the sticker design element to recognize and remember the shape, and to resize the size and shape of the sticker design element appropriately in response to changes in the size and shape (tilt, etc.) of the subject in the future. can do. For example, if the size of the subject is different, the size may be changed to correspond to the size of the subject not only while maintaining the shape of the sticker design element as well as the coordinates of the sync point.

Next, in the case of the sticker design element 1210-3, the relation with the facial contour 1212 may also be considered, but the relation with the nose contour 1214 is also considered. Therefore, in this case, a plurality of sync points may be detected with respect to the contours 1212 and 1214 of the plurality of subjects, and the relation information with the detected plurality of sync points may be generated and stored.

According to another embodiment of the present invention, a sticker design element that covers the entire part of the face can also be created. In this case, considering the relationship with the face contour 1212, if the face is displayed small, the size of the sticker design element is also small enough to cover the small face, and if the face is displayed large, it can cover the larger face This will increase the size of the sticker design element. The change in the size, shape, and position of the sync point of the sticker design element may be automatically made in response to the change in the size and shape of the subject photographed in real time.

The tracking method based on the relationship information between the subject and the sticker design element according to the exemplary embodiment of the present invention described with reference to FIGS. 12A to 12C is applicable to tracking between the pre-stored sticker and the subject. In addition, according to the above embodiment, the information related to the created and stored user authoring sticker may be provided to the server device together with the streaming image.

According to another embodiment of the present invention, in the method for providing a live broadcast service, the server stores a hash tag for a sticker applied to a streaming video in a database, and automatically applies the hash tag stored in the database at a time when the video is shared. In addition, it is possible to easily search for an image to which the advertisement sticker is applied, and through this, it is possible to estimate the number of times of use or exposure of the advertisement sticker. Of course, the watermark for the sticker may be inserted into the sticker according to an embodiment of the present invention, and the number of times of use of the sticker may be estimated using the inserted watermark code.

Although described above with reference to the drawings and embodiments, it does not mean that the scope of protection of the present invention is limited by the above drawings or embodiments, and those skilled in the art to the spirit of the present invention described in the claims It will be understood that various modifications and variations can be made in the present invention without departing from the scope of the invention.

Claims (13)

In the live streaming video generation method in the client terminal,
Obtaining broadcast platform information associated with the live broadcast;
Generating a first streaming video based on the broadcasting platform information;
Displaying the first streaming video;
Selecting, by user input, at least one of a sticker and an effect for decorating at least a portion of the displayed first streaming video; And
Generating a second streaming image by applying at least one of the selected stickers and effects to the first streaming image in real time,
The application of at least one of the sticker and the effect is made by paying the cost to the live broadcast server from the user account of the client terminal, live streaming video generation method. The method of claim 1, wherein the generating of the first streaming video comprises:
Photographing a subject through a camera; And
And generating a first streaming image based on the photographed subject. The method of claim 2,
At least one of the stickers and effects may be at least one of (i) at least one of stickers and effects for decorating the subject, or (ii) at least one of stickers and effects for decorating at least one area of the first streaming image unrelated to the subject. Live streaming video generation method comprising one. The method of claim 1,
The method may further include transmitting the generated second streaming video to the live broadcast server.
The second streaming video transmitted to the live broadcast server is a live streaming video generating method is transmitted to the client terminal of the viewer watching the live broadcast. The method of claim 4, wherein
The second streaming video transmitted to the live broadcast server is a live streaming video generation method which is simultaneously transmitted in real time to the client terminal of a plurality of viewers. The method of claim 5,
The processing for the sticker is applied to the second streaming image by an input from the client terminal of the viewer,
The processing of the sticker is made by paying a fee from a user account associated with the viewer's client terminal. The method of claim 1,
The sticker is a live streaming video generating method comprising a moving sticker. The method of claim 1,
And the effect comprises at least one of audio, subtitles, text, graphics, 3D content, and video. In the live streaming video generating device,
Communication unit for communicating with the server;
A display unit configured to display a first streaming image generated based on platform information obtained from the server through the communication unit, wherein the platform information includes broadcast platform information associated with live broadcasting; And
Select, by user input, at least one of a sticker and an effect for decorating at least a portion of the displayed first streaming image, and inserting at least one of the selected sticker and effect in the first streaming image It includes a control unit for generating a second streaming video by applying in real time,
Application of at least one of the stickers and effects is made by paying a fee from a user account to a live broadcast server. In the live streaming service providing method in a server device that provides a live streaming service,
Providing platform information associated with live streaming to the first client terminal;
Receiving a second streaming video from the first client terminal, the second streaming video is at least one of the stickers and effects for decorating at least a portion of the first streaming video generated based on the platform information in real time Applied to the image; And
Providing a second streaming video obtained from the first client terminal to a second client terminal,
Application of at least one of the stickers and effects is by payment to the server device from at least one of (i) a user account associated with the first client terminal and (ii) a user account associated with the second client terminal, How to provide live streaming service. The method of claim 10,
The live streaming service includes a live broadcast service based on the first client terminal.
And the second client terminal is a terminal associated with a viewer entering a broadcasting station established by the first client terminal. In the device for providing a live streaming service,
A communication unit communicating with the first client terminal and the second client terminal; And
A control unit for transmitting platform information associated with live streaming to the first client terminal through the communication unit, and receiving a second streaming image from the first client terminal and providing the second streaming image to the second client terminal.
The second streaming image is an image to which at least one of a sticker and an effect for decorating at least a portion of the first streaming image generated based on the platform information is applied,
Application of at least one of the stickers and effects is achieved by payment of at least one of (i) a user account associated with the first client terminal and (ii) a user account associated with the second client terminal. Providing device. In the system for providing a live streaming service,
A first client terminal generating a second streaming video based on platform information associated with a live streaming service and providing the second streaming video to a server device;
A server device for providing the platform information to a first client terminal, receiving the second streaming video from the first client terminal, and providing the second streaming image to a second client terminal; And
Including a second client terminal for receiving the second streaming video,
The second streaming image is an image to which at least one of a sticker and an effect for decorating at least a portion of the first streaming image generated based on the platform information is applied,
Application of at least one of the stickers and effects is achieved by payment of at least one of (i) a user account associated with the first client terminal and (ii) a user account associated with the second client terminal. Providing system.

Images