KR20180092873A - Live streaming image generating method and apparatus, live streaming service providing method and apparatus, live streaming system - Google Patents

Abstract

One aspect of the present invention discloses a method for generating a live streaming image at a client terminal. The method comprises the following steps: obtaining platform information for generating a streaming image; generating a first streaming image based on the platform information; displaying the first streaming image; selecting at least one of a sticker and an effect for decorating at least a part of the displayed first streaming image by user input; and generating a second streaming image by applying at least one of the selected sticker and effect to the first streaming image in real time. Therefore, an additional profit can be generated by utilizing a brand association sticker through live broadcasting.

Description

TECHNICAL FIELD [0001] The present invention relates to a live streaming video generating method and apparatus, a live service providing method and apparatus, and a live streaming system.

The present invention relates to an image processing and a service providing method based on a photograph taken through a camera or a sticker added to a screen.

In recent years, not only TV broadcast and cable broadcast through a broadcasting network but also broadcasting via the Internet have been activated, so viewers can experience various contents broadcasting through an IP (Internet Protocol) network. In particular, in the case of broadcast jockey (BJ: Broadcast Jackey), which deals with interesting contents, it is difficult to consider all of these parts in general personal broadcasts, There is a problem that it is not well utilized. In addition, it is difficult to utilize various broadcast special effects available in broadcasting stations in general personal broadcasting.

The object of the present invention is to provide a live streaming image generation method and apparatus for broadcasting jokes of a live broadcast service to enable a much richer media broadcast using various types of real time animation stickers and effects, A providing method and apparatus, and a live streaming system.

According to an aspect of the present invention, there is provided a method for generating a live streaming image in a client terminal, the method comprising: acquiring platform information for generating a streaming image; Selecting at least one of a sticker and an effect for decorating at least a part of the first streaming image to be displayed by a user input, generating a first streaming image, And generating a second streaming image by applying at least one of the selected stickers and effects to the first streaming image in real time.

The generating of the first streaming image may include photographing a subject through a camera and generating a first streaming image based on the photographed subject.

At least one of the sticker and the effect may include at least one of a sticker and an effect for decorating the subject.

Wherein the step of generating the second streaming image comprises the steps of: tracking the subject in real time to obtain tracking information; and applying at least one of the selected stickers and effects to the subject based on the obtained tracking information, And generating a second streaming image.

The tracking information includes information associated with at least one feature point of the subject and at least one of the selected stickers and effects may be applied to the subject via the at least one feature point included in the tracking information.

Wherein the information associated with the at least one feature point includes motion information of the at least one feature point and at least one of the selected stickers and effects is associated with the movement of the subject based on motion information of the at least one feature point, .

At least one of a sticker and an effect corresponding to the motion information may be selected based on motion information of the at least one feature point.

The method for generating a live streaming image may further include transmitting the generated second streaming image to a live broadcasting server that provides a service associated with the live broadcasting, May be transmitted to the client terminal of the viewer who watches the live broadcast.

The second streaming image transmitted to the live broadcasting server can be simultaneously transmitted to client terminals of a plurality of viewers in real time.

Wherein the platform information includes broadcast platform information associated with a live broadcast obtained from a live broadcast server providing a service associated with live broadcast, wherein the obtained broadcast platform information includes information about at least one of the sticker and the effect . ≪ / RTI >

The sticker may include a sticker that arbitrarily sets position information in the first streaming image according to a user input to cover a part of the first streaming image.

The sticker may include a moving sticker.

The effect may include at least one of audio, subtitles, text, graphics, 3D content, and video.

The effect may include the effect of disabling at least a portion of the sticker previously applied to the first streaming image by user input.

Wherein at least one of the sticker and the effect includes at least one of a sticker and an effect associated with an advertisement, wherein when a sticker and an effect associated with the advertisement are selected by user input, And the like.

According to an aspect of the present invention, there is provided an apparatus for generating a live streaming image, the apparatus comprising: a communication unit for communicating with a server; and a platform for generating a streaming image, which is acquired from the server through the communication unit, A display unit for displaying a generated first streaming image and a user for selecting at least one of a sticker and an effect for decorating at least a part of the first streaming image to be displayed, And a controller for applying at least one of the selected stickers and effects to the first streaming image in real time to generate a second streaming image.

According to an aspect of the present invention, there is provided a method of providing a live streaming service in a server device for providing a live streaming service, the method comprising: providing platform information associated with live streaming to a first client terminal; Receiving a second streaming image from the terminal, in which at least one of a sticker and an effect for decorating at least a part of a first streaming image generated based on the platform is applied in real time, and a second streaming image obtained from the first client terminal And providing a second streaming image to the second client terminal.

The platform information may include information about at least one of a sticker and an effect.

The live streaming service may include a live broadcast service based on the first client terminal, and the second client terminal may be a terminal associated with a viewer who has entered a broadcasting station established by the first client terminal.

The first streaming image may include a streaming image generated based on a subject photographed by the camera.

Wherein the platform information includes information on a tracking algorithm for tracking the subject in real time to acquire tracking information, and the second streaming image includes information on the selected sticker, based on the tracking information obtained using the tracking algorithm, ≪ / RTI > and / or effects.

The sticker may include a sticker that arbitrarily sets position information in the first streaming image according to a user input to cover a part of the first streaming image.

The effect may include an effect of causing at least a part of the sticker applied to the display image to disappear by a user input at the second client terminal.

At least one of user input at the first client terminal for applying the sticker and user input at the second client terminal for activating an effect of disappearing at least a portion of the applied sticker is associated with the first client terminal A user account, and a user account associated with the second client terminal.

At least one of the sticker and the effect may include at least one of a sticker and an effect associated with the advertisement.

At least one of a sticker and an effect associated with an advertisement applied to the streaming image by a user associated with the second client terminal when the streaming image is generated by applying at least one of a sticker and an effect associated with the advertisement at the first client terminal Related information corresponding to at least one of a sticker and an effect associated with an advertisement applied to the streaming video, and when the detected advertisement-related information is based on the advertisement-related information, The user associated with at least one of the first client terminal and the first client terminal may share at least some of the revenue associated with the advertisement.

Sharing the revenue associated with the advertisement may be performed by estimating the number of times of application of at least one of the sticker and the effect associated with the advertisement applied to the second streaming image.

The method of providing a live streaming service may further include providing a sticky / effect selection menu including at least one of a sticker candidate and an effect candidate applicable at a first client terminal based on the platform information.

At least one of the sticker candidates and the effect candidates included in the sticker / effect selection menu includes at least one of a sticker candidate and an effect candidate corresponding to at least one of a history, a content, and a service type of the user associated with the first client terminal It can be composed of one.

According to an aspect of the present invention, there is provided an apparatus for providing a live streaming service, comprising: a communication unit for communicating with a first client terminal and a second client terminal; Receives a second streaming video from the first client terminal, applying at least one of a sticker and an effect for decorating at least a part of the first streaming video generated based on the platform, And provide a second streaming image obtained from the first client terminal to the second client terminal.

According to an aspect of the present invention, there is provided a live streaming system for displaying a first streaming image generated based on platform information for generating a streaming image, At least one of a sticker and an effect for decorating at least a part of an image is selected and at least one of the selected stickers and effects is applied to the first streaming image in real time to generate a second streaming video A server device for providing platform information related to live streaming to a first client terminal and receiving the second streaming video from the first client terminal and providing the second streaming video to a second client terminal; 2 < / RTI > client receiving the streaming video It can include the horse.

According to the live streaming image generating method and apparatus, the live streaming service providing method and apparatus, and the live streaming system of the present invention, additional profit can be generated by utilizing the brand association sticker through live broadcasting.

In addition, advertisement revenue from various brands and advertisers who want to advertise through live broadcasting is shared by a live broadcast provider, BJ, and a sticker provider, thereby effectively distributing sales.

1 is a conceptual diagram schematically showing a live streaming system according to an embodiment of the present invention;
FIG. 2 is a diagram illustrating a broadcasting screen including a streaming image to which a sticker and / or an effect is applied according to a live streaming image generating method according to an embodiment of the present invention;
FIG. 3 is a flowchart schematically illustrating a live streaming image generating method and a live streaming service providing method according to an embodiment of the present invention.
4 is a conceptual diagram specifically illustrating broadcast platform information provided from a server to a client terminal;
5 is a detailed flowchart specifically illustrating a method of generating a streaming image based on tracking information,
FIG. 6 is a flow diagram illustrating a method of applying a sticker associated with an advertisement and sharing advertising revenue based on the applied sticker,
7A and 7B are conceptual diagrams showing examples of various kinds of stickers and effects,
8 is a block diagram illustrating a live streaming image generating apparatus according to an embodiment of the present invention.
9 is a block diagram illustrating an apparatus for providing a live streaming service according to an embodiment of the present invention.
10 is a block diagram illustrating a live streaming system according to another embodiment of the present invention.
11 is a flowchart schematically illustrating a method of generating a streaming image based on a sticker manually authored by a user in a live streaming image generating apparatus according to an embodiment of the present invention.
12A is a diagram showing a screen of an actual application for implementing a sticker authored by a user,
FIG. 12B is a conceptual diagram showing a first embodiment of associating a subject with a sticky design element displayed on the screen of FIG. 12A;
12C is a conceptual diagram illustrating a second embodiment of associating a sticky design element displayed on the screen of FIG. 12A with a subject.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail.

It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the terms "comprises" or "having" and the like are used to specify that there is a feature, a number, a step, an operation, an element, a component or a combination thereof described in the specification, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the relevant art and are to be interpreted in an ideal or overly formal sense unless explicitly defined in the present application Do not.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In order to facilitate the understanding of the present invention, the same reference numerals are used for the same constituent elements in the drawings and redundant explanations for the same constituent elements are omitted.

Throughout this specification, it is to be understood that the stickers / effects all refer to digitized forms of information for decorating the display image, and in the case where only the term sticker is used, effects may also be considered. It will be self-evident.

The live streaming image generating apparatus may be referred to as a client terminal and a live broadcast streaming image generating apparatus, and the live streaming service providing apparatus may be referred to as a server apparatus and a live broadcast service providing apparatus. It also provides a selection menu including sticker candidates and effect candidates for application of stickers / effects, application of stickers / effects according to selection in the selection menu, and creation of streaming images based on sticker / effects application Or may be performed in a server device.

In the present embodiment, although the sticker / effect is based on the use of the broadcast service, it is not necessarily used only for the broadcast service. It is applicable to a general communication method. In a broadcasting service or other service, the sticker / effect may be utilized as a virtual advertisement for inserting a virtual image into a computer graphic.

1 is a conceptual diagram schematically showing a live streaming system according to an embodiment of the present invention. 1, a live streaming system according to an exemplary embodiment of the present invention may include a first client terminal 110, a server device 120, and a second client terminal 130. As shown in FIG.

Referring to FIG. 1, a first client terminal 110 is a terminal of a broadcast jockey that initiates streaming in the IP network-based broadcasting system described above. That is, it receives the platform information from the server device 120, executes live personal broadcasting, generates a streaming video associated with the live personal broadcasting, and transmits the generated streaming video to the server device 120. The platform information basically includes basic information for generating a streaming image. The platform information may include broadcast platform information. The first client terminal 110 may be called a broadcast jockey (BJ) terminal. The first client terminal 110 creates an environment for personal broadcasting based on the broadcasting platform. The generated environment is basically a video-based broadcast environment, and may include a form in which a broadcast jockey is played as a host. That is, there may be a moving subject such as a person or an animal or an automobile. However, it is not necessarily limited to this form. That is, the first client terminal 110 generates a first streaming image (which may be referred to as a basic streaming image) by photographing a moving or stationary subject based on a photographing means such as a camera interlocked with the terminal 110. Alternatively, the first streaming image can be generated based on the previously stored image and / or the real-time or non-real-time image received from the other terminal without special shooting means.

The first client terminal 110 can make the broadcast content of the broadcast jockey richer by using various broadcast related control means existing in the broadcast platform. Means for enriching broadcast content may include stickers and / or effects. The sticker includes elements associated with design and design for decorating at least a portion of a subject photographed by the camera and displayed on the display means, a display image containing the subject, and / or a first streaming image generated based on the platform information It means comprehensive. For example, it may include objects designed such as fur hat, crown shape, eyeglass shape, headband shape, various shapes, animation shapes, etc., which are superimposed on the subject. The sticker may exist without moving in relation to the subject at a specific position in the image, or may exist in a form moving in accordance with the movement of the subject in association with the subject. In addition, the sticker may include a moving sticker and an immobile stationary sticker as well as a 3D sticker. A sticker can be used as a sticker / effect when it is selected, or it can be used to express a specific emotion and disappear. An effect may include at least one of a sound-based audio effect, a subtitle, text, a graphic, sound, a functional graphic capable of calling a specific animation, and video. Stickers and effects can be applied at the same time, and multiple stickers and effects can be applied.

The first client terminal 110 selects a sticker and / or an effect to be applied to the first streaming image among the plurality of sticker candidates and / or effect candidates. Here, the sticker candidates and effect candidates may refer to sticker-selectable stickers / effects stored in the sticker store (or inventory) before being selected as the actual applied sticker / effect. The sticker candidates and effect candidates are displayed on the sticker / effect selection menu screen, and at least one of the displayed sticker candidates and the sticker / effect sets including the effect candidates may be selected and applied to at least a part of the display image (first streaming image) have. A second streaming image is generated based on the applied stickers and / or effects. The generated second streaming video is provided to the server device 120. The first client terminal 110 may transmit the basic streaming image (first streaming image) to which the sticker / effect is not applied and the second streaming image to which the sticker / effect is applied together to the server apparatus 120.

The server device 120 is a terminal that hosts a live broadcast service. The server device 120 receives the streaming video from the first client terminal 110 executing the personal broadcasting and broadcasts the streaming video to the second and third client terminals 130. The server device 120 can manage basic broadcast platform information for broadcasting as well as transmission of such broadcast data. Broadcasting may be performed through an application or other software program, which includes installation and management information of the application or software program. The server device 120 occupies a specific URL (Uniform Resource Locator), allocates a part of the occupied URLs in response to a request from the first client terminal 110 to perform personal broadcasting, . In addition, it is possible to allow other users (viewers) to enter the established broadcasting station, thereby supporting the broadcasting between the broadcasting jockey and the viewer.

According to an embodiment of the invention, the broadcast platform information includes information related to the sticker and / or effect. It may also include policy information related to the calculation of the advertising revenue associated with the sticker and / or effect and the distribution of the calculated advertising revenue.

The second client terminal 130 is connected to the server apparatus 120 and is connected to a plurality of personal stations (including a broadcasting station established by the first client terminal 110) managed by the server apparatus 120 Terminal. The second client terminal 130 may view the streaming video provided by the first client terminal 110 by entering the URL-based personal broadcasting station provided by the server apparatus 120. At this time, data can be exchanged between a plurality of viewers and a broadcast jockey using a chat window provided by the server device 120. [ In addition, the user can view various advertisement contents derived from the stickers and / or effects applied to the streaming video during broadcasting, and play a role of generating advertising related revenues such as purchasing related goods.

At least one of the first client terminal 110 and the second client terminal 120 may be a mobile station (MS), a user equipment (UE), a user terminal (UT) A wireless terminal, an access terminal (AT), a terminal, a fixed or mobile subscriber unit, a subscriber station (SS), a cellular telephone, a wireless device, a wireless communication device, a wireless transmit / receive unit (WTRU) (PDA), a smart phone, a laptop, a netbook, a personal computer, a wireless sensor, a consumer electronics (CE) or other terminology . Various embodiments of the terminal may be used in various applications such as cellular telephones, smart phones with wireless communication capabilities, personal digital assistants (PDAs) with wireless communication capabilities, wireless modems, portable computers with wireless communication capabilities, Devices, gaming devices with wireless communication capabilities, music storage and playback appliances with wireless communication capabilities, Internet appliances capable of wireless Internet access and browsing, as well as portable units or terminals incorporating combinations of such functions However, the present invention is not limited thereto.

According to another embodiment of the present invention, the first client terminal 110 does not use only the subject currently photographed through the camera as a source video, but uses the existing stream or the recorded video as a source, / Effects can be applied. For example, a sticker can be applied even when relaying a streaming image being broadcasted by another terminal.

In addition, a sticker / effect may be applied to a terminal of a viewer like the second client terminal 130 to provide it to another client terminal.

2 is a view illustrating a broadcast screen including a streaming image to which a sticker and / or an effect is applied according to a live streaming image generating method according to an embodiment of the present invention.

Referring to FIG. 2, a broadcast jockey can perform a personal broadcast through a first client terminal. Broadcasting can be performed on an image basis and can be executed on both desktop devices such as mobile devices and PCs. In the broadcast screen, the broadcast screen includes a portion in which a streaming video is displayed, a chat window portion, and a portion in which a chat message is created. At this time, the broadcast jockey may apply a sticker and / or an effect to the streaming image.

2 is a screen before applying a sticker and / or effect, and the right side of FIG. 2 is a screen to which a sticker and / or an effect is applied. To apply stickers and / or effects, go to the Sticky / Effects application menu and the Sticky / Effects menu (210) will be displayed. At this time, the selection menu 210 for applying the sticker and the icon for entering the selection menu 210 may be displayed only on the first client terminal interface separately from the viewers' screen. Here, various kinds of stickers / effects acquired from the server apparatus are displayed. A number can be displayed around the sticker / effect. This is related to the amount you can earn from advertising revenue when you use that sticker / effect. That is, when using a sticker associated with a particular brand that has a number of "10 ", a user of the first client terminal (i.e., broadcast jockey) can obtain 10 points of advertising revenue from the brand. At this time, 10 points can be converted to a value of KRW 1, KRW 10 or KRW 100 according to the fixed point-KRW conversion standard. According to the embodiment of the present invention, there is a policy that the advertisement revenue related points can be obtained only by using the sticker / effect, and a sticker / effect is applied to the live broadcast streaming image through use of the sticker / effect, / There may be a policy that when a secondary action is performed using the sticker / effect applied in the second client terminal (the protector), the point is acquired by the user of the first client terminal. For this, monitoring of the secondary action at the second client terminal may be performed.

2, the broadcast jockey can apply a general hair stick-shaped sticker 220-1 (sticker different from advertisement sticker) to decorate itself. At this time, for example, an effect including an eye and a snowman may be applied to the sticker 220-1 at the same time as a winter image is displayed on the lower part of the displayed image (first streaming image). Here, a lipstick sticker 220-2 of a specific brand can be selected and applied to be displayed at a specific position on the screen. A particular brand of lipstick sticker 220-2 may be triggered to display relevant advertising content and / or associated advertising links in a prototype 230 at the top right of the screen as an advertising sticker. The broadcast viewer can view advertisement contents and / or links associated with the advertisement sticker or sticker displayed in the broadcast live streaming video of the broadcast jockey, and click on the surroundings to jump to a page that guides the goods associated with the advertisement. At this time, the server device can detect a secondary operation of the viewer's sticker, and distribute the point corresponding to each action to the broadcast jockey. The secondary action may be classified into a shopping activity through the web such as a purchase of goods, a product tour, a shopping cart addition, and the like, and the classified activity may have points matched to each activity. The server recognizes the viewer's actions on the commodities and products associated with the advertisements, and allows the broadcast jockey to share compensation for the commodities. The compensation may be shared not only with the broadcast jockey but also with the viewers who entered the broadcast station.

According to the embodiment of the present invention, the specific sticker / effect included in the sticky / effect menu provided through the server device may include a plurality of sticky elements. That is, when a single sticker is clicked and applied to a broadcasting streaming image, it is possible to display 10 or more lip shapes throughout the screen. That is, a sticker in which a sticker element is disposed on the entire screen, such as one sticker including a plurality of sticker elements, in which a first sticker element is disposed at a first position and a second sticker element is disposed at a second position have. The information related to such a sticker / effect may be held by the server device in the broadcast platform information, and may also include a first client terminal that acquires the broadcast platform information and proceeds with the broadcast.

According to an embodiment of the present invention, the broadcast streaming video is generated at the first client terminal. In some cases, with respect to the application of the sticker / effect, the server device may generate a streaming image to which the sticker / effect is applied. That is, the first client terminal generates a basic streaming image created by photographing a subject with a camera, generated based on an image received from a terminal or stored in advance, transmits the basic streaming image to a server device, and selects a sticker / effect The server device parses the signal related to the selection of the sticker / effect to apply the selected sticker / effect to the basic streaming image, and transmits the applied streaming image again to the first client terminal And / or broadcast to the second client terminal.

According to another embodiment of the present invention, the sticker / effect information included in the sticky / effect selection menu included in the broadcast platform information and provided to the first client terminal is the identification information of the user (i.e., broadcast jockey) (E.g., a user account of a broadcast jockey). That is, when the first user pursues personal broadcasting related to music, a sticker / effect set including a plurality of stickers / effects associated with music may be provided as a selection menu. Alternatively, when seeking personal broadcasts related to food, a menu may be provided that includes a number of stickers / effects sets associated with affiliates associated with various foods / desserts and payouts. That is, sticker / effect information may be generated differently based on the broadcasting characteristics of the user, and the different generated sticker / effect information may be included in the broadcasting platform information and provided to the first client terminal. In addition, a different sticker / effect set may be provided for each IP and service type of the first client. To provide these personalized stickers / effects, data related to the user's broadcast, or histories related to sticker / effect usage, are stored on the server device, and the server device uses customized stickers / effects To the first client terminal.

According to another embodiment of the present invention, a screen for selecting a sticker / effect corresponding to analyzed text and / or moment by analyzing a message used in a chat window and / or a broadcast jockey during broadcasting is automatically It is also possible to add a function of displaying the image. This can be accomplished by storing specific texts and notes in the server device and automatically popping up a menu for selecting the corresponding sticker / effect when the stored text and notes are recognized.

3 is a flowchart schematically illustrating a live streaming image generating method and a live streaming service providing method according to an embodiment of the present invention.

Referring to FIG. 3, in generating a live broadcast streaming image, the first client terminal 310 obtains broadcast platform information from the server device 320 (S310). At this time, the second client terminal 330 participating in the broadcast as a viewer may also acquire broadcast platform information from the server device 320 (S310). The broadcast platform information includes various information for broadcasting. This will be described in more detail with reference to FIG.

4 is a conceptual diagram for explaining the platform information provided from the server to the client terminal in detail.

Referring to FIG. 4, platform information is provided from the server 410 to the client terminal 420. The client terminal 420 may include a first client and / or a second client. The platform information includes information for installing an application or a program associated with the streaming in the terminal. The platform information may include broadcast platform information. The broadcast platform information includes information for installing an application or a program related to broadcasting in the terminal. Accordingly, the terminal is installed by an application / program that enables broadcasting on the terminal based on the information.

The platform information includes a tracking algorithm. The tracking algorithm includes a face tracking algorithm. That is, the tracking algorithm is an algorithm for acquiring information for following the movement of a moving subject. The tracking algorithm specifies at least one feature point of a specific subject (for example, a face, a face, or a nose among faces) to which a specific sticker is applied, (E.g., movement of coordinates) associated with the movement of at least one feature point. Based on the tracking algorithm, the movement of the moving subject is detected, the movement distance of the sticker corresponding to the detected movement distance is calculated, the sticker / effect is moved by the calculated movement distance, and the sticker / You can move it. However, in some cases, depending on the nature of each sticker, tracking may or may not be required. Therefore, it is preferable that the movement of the sticker according to the tracking information is applied corresponding to the nature of the sticker.

The platform information includes various sticker / effect information. This includes all sticker / effect information stored by the server device as well as custom sticker / effect set information for each user and service. These sticker / effect information will continue to be updated as new information associated with the sticker / effect is created. Also, the statistical analysis such as the user's use history is continuously updated. Platform information including updated sticky / effect information may be periodically provided to each terminal. The sticker / effect information also includes information related to the activation of the sticky / effect selection menu corresponding to the specific text, or the detection of the specification according to the speech recognition. According to another embodiment of the present invention, in relation to the broadcast platform information, the sticker / effect set may be configured based on the location information of the first client terminal, based on the ad sticker / . That is, the broadcast jockey at a specific location may transmit broadcast platform information including an advertisement sticker / effect set corresponding to the specific location set in the server to the broadcast jockey so that the location based sticker / effect can be applied to the streaming video have.

Alternatively, according to another embodiment of the present invention, a specific advertisement sticker may be applied to a predetermined position based on the position of the viewer, i.e., the second client terminal. The area based sticker may be applied directly to the broadcast streaming image received at the second client terminal. Alternatively, it may be applied to a streaming image relayed from a second client terminal to a third client terminal.

The broadcast platform information includes various functional information for broadcasting. This may include information such as chat, screen layout, management of viewers of each broadcasting station, and the like.

The broadcast platform information includes link sense capability information, which indicates information associated with the ability of a viewer to act upon a sticker / effect through a second client terminal to sense an advertisement product, advertisement content, . In other words, since the viewer 's secondary behavior can be used as an important factor in the distribution of advertisement revenue, an algorithm for sensing the advertisement can be importantly used.

Broadcast platform information includes policy information related to cost sharing. This indicates information related to the policy related to the use of the sticker / effect in the broadcast. For example, whether a point can be obtained only by use of a specific sticker, a point can be obtained if there is an action of a viewer associated with a particular sticker, Detailed policy information about the policy can be included.

Various information related to the above streaming and / or broadcasting and information related to use of the sticker / effect in streaming and / or broadcasting are included in the platform information and provided to the client terminal 420.

Referring again to FIG. 3, the first and second client terminals 310 and 330 that have acquired the platform information install the streaming and / or broadcast-related application / program (S312). Then run the installed application / program.

First, in an embodiment of streaming associated with broadcasting, the first client terminal 310 establishes a broadcasting station to start personal broadcasting (S314). At this time, the broadcasting establishment information for informing the broadcasting establishment is transmitted to the server 320 together with the broadcasting establishment. The broadcast establishment information may include information such as a user account associated with the first client terminal 310, a total number of viewable viewers, and a content type based on which content is based. The broadcasting establishment information may be provided to a terminal of a viewer who wishes to enter a broadcasting station, i.e., a second client terminal 330 (S318). This may be done in response to the request of the second client terminal 330. [ When information on the currently opened broadcasting station is requested, information on the opened broadcasting stations is provided in response thereto. The second client terminal 330 may enter the selected broadcasting station among the provided broadcasting stations (S330).

The first client terminal 310 that has opened the broadcast can photograph a subject using a camera interfaced with the terminal or a camera included in the terminal (S320). At this time, the subject can be anything other than the user himself / herself. Then, the photographed image can be displayed through the display means of the terminal (S322). The display screen of the photographed subject is streamed and broadcast. If you want to apply stickers and / or effects during the broadcast, click the icon that pops up a menu window for selecting stickers and / or effects through the user interface. Then, various stickers / effects included in the broadcast platform information are displayed on the screen. At this time, a set of stickers / effects corresponding to the user's account, the type of terminal, and information related to the content type of the opened broadcasting station may be generated and provided in a customized manner. The first client terminal 310 selects at least one from the provided sticker / effect set (S324). The selected sticky / effect is applied to at least a part of the display image including the moving subject to generate a streaming image (S326). The streaming video to which the sticker / effect is applied is transmitted to the server 320 in step S328, and the server 320 provides the streaming video to all the viewer terminals 330 that have entered the broadcasting station in step S334.

5 is a detailed flowchart specifically illustrating a method of generating a streaming image based on tracking information.

5, when the first client terminal selectively applies the sticker / effect, it may be applied in a stationary state, but may be applied in a form moving according to a moving subject. If the sticky / effect to be applied to the first client terminal is selected (S510), the first client terminal executes the tracking algorithm included in the broadcast platform information to track the movement of the object (S520). In order to track, the tracking target must be determined first, which can be specified based on the target region to which each individual sticker is applied. For example, since the hat-shaped sticker is applied to the head portion of the human body, the upper head portion of the face portion of the person is fixed as the tracking subject through the face tracking. In this manner, when the object to be tracked related to the application region of each sticker is determined, the feature points of the determined object are extracted. The extraction feature point may be preset in the tracking algorithm. For example, in a mesh-based face tracking, a point at a predetermined specific position of the outline based on the mesh at the upper part is extracted as a feature point. Then, a sticker / effect is applied via the corresponding feature point. Since movement of the subject means movement of the feature point, the movement path of the feature point is calculated to acquire tracking information (S530). Then, if the selected sticker is moved and applied along the movement path of the minutiae of the target subject based on the acquired tracking information, the sticker / effect may be moved naturally along with the movement of the subject in operation S540. As described above, the streaming image can be generated based on the sticker applied while tracking the movement (S550).

FIG. 6 is a flow diagram illustrating a method of applying a sticker associated with an advertisement and sharing advertising revenue based on the applied sticker.

6, the first client selects a sticker / effect to be applied to the broadcast streaming image (S610), and applies the sticker / effect (S620).

At this time, it can be confirmed whether the applied sticker / effect is related to the advertisement (S630). The relevance to the advertisement means that the sticker can be a sticker created by affiliating with a specific business (for example, a brand). As such, the sticker may contain information about it in association with a particular business product. That is, when a sticker is clicked, a linking operation may be performed such as linking to a product, a business site, or the like of a business, or reproducing an associated advertisement video.

If it is not relevant to the ad, it will fulfill the purpose of applying the sticker / effect, since the application of the sticker / effect carries out the function of giving the viewer fun. However, if there is an association with the advertisement, the profit distribution part of the advertisement effect can be additionally considered. The association with the advertisement can be confirmed by parsing the individual feature information of each sticker, and a flag is given according to the case so that the ad-based sticker / effect can be displayed as "1", and the sticker / have.

If the applied sticker / effect is relevant to the advertisement, it parses the policy information related to the profit distribution of the advertisement. At this time, if the revenue is distributed with use, the broadcast jockey can obtain a portion of the fee only by using the advertisement-based sticker at the first client terminal. At this time, since the advertisement effect is maximized as the number of viewers who are admitted to the broadcasting station is increased, the commission can be set in consideration of the number of admitted viewers of the broadcasting station at the time of use of the sticker. For example, in the case of a sticker of "10 ", 100 points may be allocated to 10 viewers and 1000 points may be provided to 100 viewers.

However, if the fee is not set at the same time as the use as described above and the fee is set based on the second action of the viewer of the broadcasting station, the action of the viewer associated with the applied sticker / effect is sensed at step S640. This can be done in the server device. When an action such as purchasing a product, touring a product, shopping cart, or the like is performed through a click of a sticker applied to a viewer's terminal to reproduce the advertisement video, a site connected with a brand of the sticker, etc., Compensation can be made to the broadcast jockey. Or a fee may be set in consideration of the number of click operations. That is, at least a part of the advertisement revenue calculated through various methods as described above is allocated to an account connected to the account of the broadcast jockey (S650). In other words, the broadcasting jockey considers the allocation of these fees as incentive to use the advertisement-based stickers as much as possible at the appropriate time, thereby allowing the business associated with the advertisement to enjoy various advertising effects.

7A and 7B are conceptual diagrams showing examples of various kinds of stickers and effects.

Referring to FIG. 7A, a sticker 710 covering a specific region in a streaming image may be utilized. For example, in the case of broadcast of a jockey that gives pleasure through makeup, it is possible to provide a sticker 710 that covers the face portion of the face. At this time, the first client terminal can directly determine the position to be blocked while moving the sticker 710, and the coordinate information of the sticker 710 determined through the first client terminal can be transmitted to the server. At this time, the basic streaming video information without the sticker 710 may be transmitted together with the server.

Referring to FIG. 7B, the sticker 710 covering the specific position is basically for erasing the obscured sticker area. However, at this time, the sticker area 710 can be utilized in such a manner that a certain area of the sticker 710 disappears according to the click due to the cost adjustment of the viewer, since the effect of screening the sticker area is less for all viewers. That is, when a specific viewer performs the payment of three times, he can click on a desired site to delete a part of the sticker 710 area. That is, the eye region 720 and the nose region 720 can be clicked three times to secure three erased regions 720. This can be done by replacing only the sticky area of the part with the basic streaming image since the server device also holds the basic streaming image. Alternatively, a sticker 710 may be created in the server stream on the basic streaming image, placed on the subject, and erased from the sticker 710. At this time, the size of the erased area and the click cost for the erased area can be determined by the broadcast jockey and / or the server. Further, it is possible to apply a sticker of a form in which a face covered with a sticker is changed to another person's face and the face is sent out. When a face change sticker is applied to a face portion of a subject of a first client terminal, a menu for selecting a face of another person to be changed is displayed. Through the menu, the user of the first client terminal Images can be selected, and the face portion of another person is extracted from the selected photograph or image and extracted. Then, the face portion of the position where the face change sticker exists can be deleted, and the face of the extracted other person can be changed. Alternatively, a sticker of another person's face type may be provided in advance in the menu and directly changed to the face of the sticker.

Based on these functions, each viewer can acquire different streaming images through different number of clicks. This allows the viewer to spend money, and by allowing the spent expenses to flow to the broadcast jockey (which allows the service provider to get some fees on the server device), the broadcast jockey gives the incentive to make the broadcast more interesting Effect can be given.

8 is a block diagram illustrating a live streaming image generating apparatus according to an embodiment of the present invention. 8, a live streaming image generating apparatus according to an exemplary embodiment of the present invention includes a communication unit 810, a camera 820, a display unit 830, a controller 840, an input unit 850, 860).

Referring to FIG. 8, the communication unit 810 has a function of performing communication with a server apparatus and another apparatus. The communication unit 810 includes equipment for wireless or wired communication. For example, an antenna and / or a communication processor.

The camera 820 is a device for photographing a subject, and the image photographed through the camera 820 is processed by the control unit 840. [ The camera 820 may be present in the live broadcast streaming image generating apparatus or may be connected to the outside through an interface such as an HDMI (High-Definition Multimedia Interface).

The display unit 830 outputs an image processed through the controller 830. The display unit 830 may be implemented as a monitor, a touch screen, a TV receiver, or the like.

The control unit 840 may include a broadcast application execution unit 842, a tracking information generation unit 844, an application unit 846, and a streaming image generation unit 848.

The broadcast application execution unit 842 installs and executes an application / program associated with streaming based on the platform information received through the communication unit 810. [ In the case of broadcast streaming, the broadcast application execution unit 842 installs a broadcast application / program on the basis of the broadcast platform information, and executes the broadcast application / program by input through the input unit 850. At this time, a program entry operation such as login can be performed.

The tracking information generation unit 844 generates tracking information according to the tracking algorithm included in the platform information and provides the tracking information to the application unit 846. [

The application unit 846 performs a function of applying a sticker / effect. The application unit 846 displays a suitable sticky / effect set based on at least one of the account of the user of the first client terminal that has entered the broadcast application / program, the IP of the first client terminal, 830). Then, the input related to the user's selection is inputted through the input unit 850, and the selected sticky / effect is applied to one area of the target subject or the display screen (first streaming image). At this time, the sticker / effect can be applied along the movement of the subject based on the tracking information.

The streaming image generation unit 848 processes the image captured through the camera 820 to generate a basic streaming image. Here, encoding and decoding can be performed. Also, processing of the second streaming image to which the sticker / effect is applied can be performed when the sticker / effect is applied.

The input unit 850 is an interface for processing user input such as a keyboard, a mouse, and a touch pad.

The memory 860 stores instructions (or directives) related to each function performed by the control unit 840. The memory 860 includes instructions based on broadcast platform information, and information received from a server device or other device via the communication unit 810 can be stored.

9 is a block diagram illustrating an apparatus for providing a live streaming service according to an embodiment of the present invention. 9, an apparatus for providing a live streaming service according to an exemplary embodiment of the present invention may include a communication unit 910, a controller 920, an input unit 930, and a memory 940. [

The communication unit 910 is a device responsible for communication with the first client terminal and the second client terminal.

The control unit 920 is an element for controlling an application to be managed and includes an account management unit 921, a broadcast platform management unit 923, a streaming video processing unit 925, an application determination unit 927 and an advertisement revenue processing unit 929 .

The account management unit 921 manages information related to an account that logs into an application (e.g., a broadcast application). You can manage the personal history of each account as well as the history of the broadcasts performed by each account. For example, information such as a broadcasting station opening time, an opening format, a handling content, a service type, etc. established through a specific user account, information related to admission to a broadcasting station established by another user, and / or history of stickers / Can be managed.

The broadcast platform management unit 923 manages platform information (including broadcast platform information) of FIG. This includes upgrading to periodic software, as well as creating customized sticker / effect sets based on your broadcast usage history.

The streaming image processing unit 925 processes the first streaming image and / or the second streaming image received from the first client terminal. The streaming image processing unit 925 can receive the basic streaming information (first streaming image) not subjected to the sticky / effect application from the first client terminal and the streaming image (second streaming image) to which the sticker / effect is applied together It can be processed independently. For example, in a state in which basic streaming image information is received, a signal associated with a request for applying a specific sticker is received, and a received signal is parsed to apply a requested sticker to the basic streaming image to directly generate a streaming image to which a sticker is applied have.

In addition, a request related to the sticky in the second client terminal is separately processed to perform image processing so that different streaming images can be viewed in relation to the area of the sticky in different terminals.

The application determination unit 927 determines whether or not the sticker is applied, as well as whether or not the sticker for advertisement is applied. In the case of a sticker for advertisement, it is judged to which business (or brand) is associated.

The advertisement revenue processing unit 929 distributes advertisement revenue associated with the sticker. In particular, in the case of a sticker for advertisement, a viewer who views the advertisement sticker (viewers admitted to the corresponding station) detects an arbitrary action linked to the corresponding brand, / / Sticker providing server (see Fig. 10)) can be shared together. In this case, the distribution of advertisement revenue includes a CPC (Cost Per Click) method, a PPC (Pay Per Click) method, a CPT (Cost Per Time) method, a CPM (Cost Per Mile) Per-Millenium (CPI), Cost Per Impression (CPI), and Cost Per Conversion (CPCon).

Input 930 is an interface for processing user input to change various user settings associated with the broadcast application.

The memory 940 stores instructions related to each function performed by the control unit 920. [ For example, the user history information associated with the account, the use history information of the sticker, the information of the sticker itself (whether it is a sticker for advertising, information on which provider and which link is associated with the sticker for advertisement, etc.) . ≪ / RTI >

10 is a block diagram illustrating a live streaming system according to another embodiment of the present invention. 10, a live streaming system according to another embodiment of the present invention includes a first client terminal 1010, a server 1020, a second client terminal 1030, a sticker providing server 1040, 1050 < / RTI >

10, operations of the first client terminal 1010, the server 1020, and the second client terminal 1030 are similar to those described above.

The sticker providing server 1040 may provide a face tracking development software development kit (SDK) to the live broadcasting server 1020 and may apply it to the web or app clients 1010 and 1030 of the live broadcasting service. Then, at the first client terminal 1010, a sticker inventory is opened in a broadcast service used by the broadcast jockey so that at least one of them can be selectively used. For various live broadcast services, various stickers stored in a sticker archive of the sticker providing server 1040 may be provided at the same time.

The sticker providing server 1040 may provide and operate a sticker on the media desired by advertisers, such as an Ad network. In addition to the advertisement sticker (brand association sticker), contents such as background sound source, professional effect, subtitle function, and intermediate advertisement used in broadcasting can be provided to the server 1030 in real time. The server 1030 can provide the provided information to the client terminals 1010 and 1030.

In order to realize this operation, the sticker providing server 1040 provides the face tracking SDK and the user using tracker to the live broadcasting server 1030, and the live broadcasting server 1030 transmits the information related to the user use data and the function promotion to the sticker To the providing server 1040. Then, both can share advertisement sales together.

As described above, the sticker can be applied to both the web and the mobile environment, and the broadcast jockey can maximize the fun of broadcasting by displaying a screen on which a sticker is applied to a viewer who is watching a broadcast on his or her broadcasting station. In particular, when a brand association sticker is utilized, viewers who view the brand association sticker may sense an arbitrary action linked to the brand, and transmit the revenue according to the advertisement sales to a broadcast jockey, a live broadcast service provider (i.e., server 1020) The sticker providing server 1040 can share the same. In this case, the distribution of advertisement revenue includes a CPC (Cost Per Click) method, a PPC (Pay Per Click) method, a CPT (Cost Per Time) method, a CPM -per-Millenium method, a CPI (Cost per Impression) method, and a CPcon (Cost Per Conversion) method.

According to another embodiment of the present invention, when allocating the advertisement revenue, the server apparatus can distribute the profit to the accounts of the broadcast performers appearing on the broadcast through the first client terminal and / or the accounts of the broadcast viewers. That is, when the advertisement sticker is used for the broadcast performer, the information on the broadcast performer to which the sticker is applied can be acquired, and some of the advertisement revenue can be distributed to the account associated with the acquired information. In addition, at the time when the viewer views the advertisement sticker of the streaming image, the user may leave a log and share some of the advertisement revenue.

11 is a flowchart schematically illustrating a method of generating a streaming image based on a sticker manually authored by a user in a live streaming image generating apparatus according to an embodiment of the present invention.

Referring to FIG. 11, a client terminal executes an application related to broadcasting and displays a subject photographed by a camera (S1110). Then, the sticky / effect application menu is started and the sticky authoring tool is executed in the menu (S1120). The sticker authoring tool can be provided as one of the sticker / effect application menus of the broadcast application. When the user selects the sticker authoring tool, the terminal displays a menu for sticker authoring on the display screen. According to another embodiment of the present invention, a sticker may be authored using photographs or moving pictures previously stored in the terminal, not during shooting.

Then, a mode for sticker authoring is selected (S1130). Here, the mode for authoring may include two modes, i.e., a first mode and a second mode.

The first mode is a mode for recognizing a subject such as a face, and is a mode for associating a sticky design element generated by a user with at least one of displayed objects. At this time, the outline of the subject can be considered as an element determining the relationship with the sticker design element. The contour of the subject can be identified using a pre-stored contour template. For example, when the subject is a face, an outline template including the overall layout information of a plurality of components such as eyes, nose, and mouth in the face is previously stored, and the outline of the face and the components in the face displayed on the displayed screen Matching the previously stored contour template, recognizing the face first, and identifying and storing the recognized contour information of the face. At this time, information such as brightness, motion, color, and eye position estimation can be used to distinguish the face and the background. At this time, it is effective to analyze the main part of the face, discard the minute errors, and capture only the large features and compare with the stored face template. In the embodiment of the present invention, the contour template is not necessarily related to the face but may be related to other parts of the human body such as arms, legs, hands, and the like.

In the case of the first mode, the sticky design element generated by the user in response to the movement of the subject and / or the size change of the subject may also move and change in size because of the association with the subject.

The second mode is a mode for designing to be related to the displayed screen, that is, to be displayed at a specific position in the display screen, without considering the relationship with the subject. In the second mode, the sticky design element can be displayed at a fixed size at a predetermined position in the display screen without being affected by the motion of the subject. In some cases, even in the second mode, a moving effect such as a change in size may be given depending on the setting of the user.

Upon completion of the mode selection, the user receives a user input for designing a sticker through a user interface (e.g., a touch screen, a keyboard, a mouse, etc.) (S1140). The user input for the sticker design may have a text input, and there may be an input via a drawing mode. The drawing mode may be referred to as a two-dozing mode. A sticky design element can be created by inputting a text input through typing using a keyboard, and a sticky design element can be created by inputting a drawing mode through a mouse or a touch screen based on a line or a shape selected by a user.

According to an embodiment of the present invention, figures and images can be called in addition to the text and drawings. The graphic may include a still image, an animation graphic, or a 3D object, and the image may be imported from a camera or album and used as a sticky design element. The shapes may include solid lines, triangles, squares, arrows, bent lines, half moon shapes, clouds, hearts, mathematical expressions, flow diagrams, and the like. Such figures may be stored in a template in advance.

According to another embodiment of the present invention, other effects such as a sound, a functional graphic capable of calling a specific animation, and the like can be utilized as a sticky design element.

The client terminal identifies the in-screen coordinate information of each sticker design element to memorize the inputted sticker design element as a user author sticker (S1150). The coordinate information is recognized by identifying the coordinates of the pixel associated with the sticker design element on the screen being displayed. The identified coordinate information can be utilized as relationship information indicating the association with the display screen or the object according to the mode selected by the user.

After identifying the coordinates of the inputted sticker design element, it is determined whether the mode is the first mode or the second mode (S1160). If it is the first mode (object recognition mode), it is related to the contour of the subject, and the second mode is related to the display screen. The switching to the first mode and the second mode is possible at any time before the sticky storage, so that upon sensing a mode change, all of the sticky design elements interpreted as the second mode can be interpreted as the first mode. That is, the sticker design elements analyzed through step S1165 can be interpreted through steps S1170 through S1190, and vice versa.

If the first mode is selected in step S1160, first, the outline of the subject is identified using the outline template (S1170). The terminal stores an image (e.g., a rectangle) including all the sticky design elements as an image and recognizes the object in the stored image to identify the outline of the identified object. At this time, there may be a plurality of objects in the image. For example, the face may be a subject, or another part of a person such as a hand, an arm, or the like may be a subject. The client terminal recognizes the object based on the contour template of the pre-stored object, and identifies the contour of the object. At this time, coordinate information of the pixel related to the outline can be confirmed.

In operation S1180, relationship information for associating the outline of the identified object with the sticky design element is generated based on the coordinate information of the sticky design element identified in step S1150. According to an embodiment of the present invention, the center point of the subject is searched based on the outline of the subject included in the stored image. Based on the distance from the center point to the contour of the subject, the relationship between the sticky design element and the subject can be grasped by considering the contour of the subject or the distance from the center point to the sticky design element. Then, the relationship information can be generated based on the identified relationship.

According to another embodiment, in step S1180, a point corresponding to the synchronization point is found on the pre-stored contour template. The synchronization point may be specified as a specific point on the contour of the subject, as a point for synchronizing the contour of the actual object currently displayed, the sticker design element, and the pre-stored contour template. For example, a specific point on the contour of the subject having an association with the end of the sticky design element becomes a synchronization point. Alternatively, the start point and / or the end point of the sticky design element and the nearest point on the outline can be a synchronization point. In some cases, the point at which the sticker design element and contour meet can be a synchronization point. The algorithm for determining the synchronization point may exist in various ways, and it may be included in the broadcast platform information received from the server apparatus. The synchronization point on the contour of the actual object corresponds to a specific point on the contour template that matches the object and can be appropriately applied to other objects to which the contour template is applied based on the point. The synchronization point thus generated can be considered as the relationship information.

When the relationship information is generated, the client terminal stores the user authoring sticker in the local area based on the relationship information and the sticker design element (S290). At this time, the user can instruct to perform the storage by pressing the storage icon. That is, if it is determined that the user has created a favorite sticker to the extent that the user wants to register as a sticker by performing the steps from step S210 to step S1180 as a fun factor, You can register and save stickers that are currently being implemented. That is, the storage of the sticker can also be selectively determined depending on the preference of the broadcast jockey. Then, the stored user authoring sticker information and the streaming image to which the user authoring sticker is applied are transmitted to the server device. And the server device receiving the message can broadcast a streaming video to which a user authoring sticker is applied to the viewer client.

At this time, the sticker related information to be stored and transmitted may include relationship information, contour of the subject, coordinate information of each sticker design element, and information of the sticker design element itself. Or associated contour template information may further be stored. At this time, the relationship information may include synchronization point information (e.g., a specific point on the subject or a corresponding point on the contour template), information indicating a relationship between the synchronization point and the sticky design element (e.g., distance information, etc.) . The self information of the sticker design element may include the type of the sticker design element (whether it is text or drawing), the color of the line constituting the sticker design element in the case of drawing, thickness, and the like.

If the second mode is selected in step S1160, the second mode is entered and a user authoring sticker is created and stored in consideration of the position in the display screen based on the coordinate information of the sticker design element (S1165). At this time, information related to the subject is not stored, and only the coordinate information within the display screen can be stored for association with the display screen.

The point of identifying the contour of the subject and the pixel coordinates of the sticky design element may be important, since it is not relevant when using pre-stored pictures, but the subject may continue to move and change size during shooting. According to an embodiment of the present invention, the time point at which the contour of the subject and the coordinates of the sticky design element are identified may include a moment when the user input for creating the sticky design element starts or a time point at which the user input is completed. The subject and the sticky design element are identified according to the time point, and the sticky design element changes in accordance with the change of the subject after the corresponding point in time.

12A is a diagram showing a screen of an actual application for implementing a sticker authored by a user.

12A, a tool for creating a user authoring sticker includes an icon 1202 for selecting a first mode and a second mode at the top of a broadcast screen, icons 1204 and 1206 for selecting a type of a sticky design element Display. When the icon 1202 is pressed, the first mode is operated and the sticker moves in conjunction with the contour of the subject. If the icon 1202 is not pressed, the contour of the subject is not recognized because there is no association with the subject. The embodiment of FIG. 12A assumes a situation where the first mode is selected.

Also, when the icon 1204 is selected, text input is possible, and when the icon 1206 is selected, the user can input in the drawing mode. The embodiment of FIG. 12A assumes a situation where the drawing mode is selected.

In the drawing mode, the user can select the thickness and shape of the line through the menu 1220 for selecting a basic line for drawing on the left side of the screen. The user can arbitrarily change the thickness and shape of the line using the menu 1220 during drawing. For example, with respect to the thickness of the line, a menu for selecting a plurality of thicknesses from the thickest line at the top to the thinnest line at the bottom can be provided. In addition, a menu for selecting a line shape (a menu for selecting a pen type or a brush type) may be a triangular line, a rectangular line, a circular line, a heart-shaped line, A line with a solid effect, a line with a shadow effect, and the like.

In addition, a menu 1230 for selecting a line color may be provided at the bottom of the screen. The user can perform drawing by selecting one of a plurality of provided colors.

In the embodiment of Fig. 12A, the user is photographing the upper half of the human body including the face, and the client terminal displays it. The broadcast jockey creates a plurality of sticker design elements 1210-1 to 1210-7 by performing drawing in the thickness, shape, and color of the line selected through the menu 1220 and the menu 1230 during shooting in real time. Different lines may be used through menu 1220 and menu 1230 whenever there is an input for each sticker design element 1210-1 through 1210-7. Overall, the sticker design elements 1210-1 and 1210-2 are formed at the top of the face to form a rabbit ear. The sticker design element 1210-3 is formed in the shape of a heart, near the nose, and the sticker design elements 1210-4 to 1210-7 form a beard shape in a shape diverging around the nose.

Specifically, the client terminal recognizes the sticker design elements 1210-1 to 1210-7 and stores the sticker design elements 1210-4 to 1210-7 as a whole user authoring sticker. Individually recognize and associate each sticker design element with the subject. That is, the subjects associated with the sticker design elements 1210-4 through 1210-7 may be different. The sticker design element can be recognized as a figure composed of one connected line, a typed character unit at a time.

12B is a conceptual diagram illustrating a first embodiment of associating a subject with a sticky design element displayed on the screen of FIG. 12A.

Referring to FIG. 12B, when a sticker design element is created by user input, the terminal stores a figure 1240 including all the sticker design elements in an image form. Then, the object included in the graphic form 1240 is identified. In the embodiment of FIG. 12B, the face can be identified by the subject in the image. At this time, the subject is recognized as a face by matching the outline of the subject with the pre-stored face template, and the face contour information can be obtained based on the recognized face. Once the face contour is acquired, the coordinates of the center point 1242 of the face can be secured based on the face contour information. Then, based on the relationship between the coordinates of the face center point 1242 and the outline of the face, relationship information between the subject (face) and the respective sticker design elements 1210-1 to 1210-7 is generated. For example, the relationship between the distance to the face center point 1242 and the face contour and the distance (including d1, d2, d3, d4) between the face center point 1242 and each of the sticker design elements 1210-1 to 1210-7 It can be generated as information. Or the distance between the face contour and each of the sticker design elements 1210-1 through 1210-7 may be considered as relationship information. It may be desirable to consider the starting and ending points of each sticker design element 1210-1 through 1210-7 in calculating the distance to each sticker design element 1210-1 through 1210-7.

Since the distance from the center point to the contour of the face is changed when the face contour is changed (for example, the shape of the face is changed or the size of the face is changed) based on the generated relationship information, And the distances between the first and second antennas 1210-1 to 1210-7 may be changed.

12C is a conceptual diagram illustrating a second embodiment of associating a sticky design element displayed on the screen of FIG. 12A with a subject.

Referring to FIG. 12C, the sticker design element 1210-1 starts inputting from a start point 1250-1, and ends inputting at an end point 1250-2. The client terminal may recognize the start point 1250-1 and the end point 1250-2 and identify the object to be associated. In the present embodiment, the closest to the sticky design element 1210-1 and the "face" having a high degree of matching is selected as the object to be associated. At this time, the subject is recognized as a face by matching the outline of the subject with the pre-stored face template, and the face contour 1212 information is acquired based on the recognized face. The client terminal looks for a synchronization point to clarify the relationship with the subject having the association with the sticker design element 1210-1, i.e., the face contour 1212. [ The synchronization point is preferably a point on the face contour 1212 and has a specific relationship with the starting point 1250-1 and the ending point 1250-2. For example, the nearest point from the start point 1250-1 and the end point 1250-2 may be preferable. Or a point on the contour 1212 that meets the sticky design element 1210-1 may be a synchronization point. The information related to the synchronization point includes coordinate information indicating which point on the contour of the subject, and the information related to the synchronization point is information stored in the terminal when the user's writing sticker is stored.

In this embodiment, the points 1260-1 and 1260-2, which are the points on the face contour 1212 closest to the starting point 1250-1 and the end point 1250-2, can be detected as synchronization points .

After detecting the synchronization points 1260-1 and 1260-2, the client terminal calculates the distances d1 'and d2' between the face contour 1212 and the start point 1250-1 and the end point 1250-2, This can be stored as relationship information. That is, even if applied to another face, the sticker can be reproduced in such a manner that a sticker design element is generated at distances d1 'and d2' based on the synchronization point. At this time, if the size of the face changes, the distances d1 'and d2' may also be reduced or enlarged in proportion to the changed size. That is, even if the shape or size of the face changes, a sticky design element can be appropriately applied to a face having a coordinate corresponding to the coordinate of the synchronous point of the current face.

In addition, the coordinates of all the pixels on which the sticky design elements are displayed are recognized and stored, and the size and shape of the sticky design elements are appropriately resized in response to changes in the size and shape (inclination, etc.) can do. For example, when the size of the subject is changed, it can be changed not only to the coordinates of the synchronization point but also to the size of the subject whose size is changed while maintaining the shape of the sticky design element.

Next, in the case of the sticker design element 1210-3, the relationship with the face contour 1212 can be considered, but the relationship with the contour 1214 is also considered. Therefore, in this case, a plurality of synchronization points can be detected with respect to the contours 1212 and 1214 of a plurality of subjects, and relationship information with the detected plurality of synchronization points can be generated and stored.

In accordance with another embodiment of the present invention, a sticky design element may be created that covers the entire portion of the face. In this case, considering the relationship with the face contour 1212, when the face is displayed small, the size of the sticker design element is small enough to cover the reduced face, and when the face is displayed large, The size of the sticker design element is increased. The size, shape, and position of the point of synchronization of such a sticker design element can be changed automatically in response to changes in the size and shape of the object being photographed in real time.

12A to 12C, the relationship information-based tracking method of a subject and a sticky design element according to an embodiment of the present invention is applicable to tracking of a previously stored sticky and a subject. In addition, the information related to the user authoring sticker generated and stored according to the above embodiment may be provided to the server device together with the streaming image.

According to another embodiment of the present invention, in a method of providing a live broadcast service, a server stores a hash tag for a sticker applied to a streaming image in a database, and automatically applies a hash tag stored in a database at a time of sharing the image , It is possible to easily retrieve the image to which the advertisement sticker is applied, thereby estimating the use frequency or the exposure frequency of the advertisement sticker. Of course, a watermark for the sticker may be inserted into the sticker according to an embodiment of the present invention, and the number of uses of the sticker may be estimated using the inserted watermark code.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the inventions as defined by the following claims It will be understood that various modifications and changes may be made thereto without departing from the spirit and scope of the invention.

Claims (31)

A method for generating a live streaming image in a client terminal,
Obtaining platform information for generating a streaming image;
Generating a first streaming image based on the platform information;
Displaying the first streaming image;
Selecting at least one of a sticker and an effect for decorating at least a part of the displayed first streaming image by user input; And
And generating a second streaming image by applying at least one of the selected stickers and effects to the first streaming image in real time. The method of claim 1, wherein generating the first streaming image comprises:
Photographing a subject through a camera; And
And generating a first streaming image based on the photographed subject. 3. The method of claim 2,
Wherein at least one of the sticker and the effect includes at least one of a sticker and an effect for decorating the subject. 3. The method of claim 2,
Wherein the generating the second streaming image comprises:
Tracking the subject in real time to obtain tracking information; And
And generating the second streaming image by applying at least one of the selected stickers and effects to the subject based on the obtained tracking information. 5. The method of claim 4,
Wherein the tracking information includes information associated with at least one feature point of the subject,
Wherein at least one of the selected stickers and effects is applied to the subject through the at least one feature point included in the tracking information. 6. The method of claim 5,
Wherein the information associated with the at least one feature point comprises motion information of the at least one feature point,
And applying at least one of the selected stickers and effects to the subject based on motion information of the at least one feature point in response to the motion of the subject. 5. The method of claim 4,
Wherein at least one of a sticker and an effect corresponding to the motion information is selected based on motion information of the at least one feature point. The method according to claim 1,
And transmitting the generated second streaming image to a live broadcasting server providing a service associated with the live broadcasting,
Wherein the second streaming image transmitted to the live broadcast server is transmitted to a client terminal of a viewer who watches the live broadcast. 9. The method of claim 8,
Wherein the second streaming image transmitted to the live broadcasting server is simultaneously transmitted to a client terminal of a plurality of viewers in real time. The method according to claim 1,
Wherein the platform information includes broadcast platform information associated with a live broadcast obtained from a live broadcast server providing a service associated with live broadcast,
Wherein the obtained broadcast platform information includes information on at least one of the sticker and the effect. The method according to claim 1,
Wherein the sticker includes a sticker that arbitrarily sets position information in the first streaming image according to a user input to cover a part of the first streaming image. The method according to claim 1,
Wherein the sticker comprises a moving sticker. The method according to claim 1,
Wherein the effect comprises at least one of audio, subtitles, text, graphics, 3D content, and video. The method according to claim 1,
Wherein the effect comprises the effect of disabling at least a portion of a sticker previously applied to the first streaming image by user input. The method of claim 1, wherein at least one of the sticker and the effect comprises at least one of a sticker and an effect associated with the advertisement,
Wherein at least a portion of the revenue associated with the advertisement is provided to an account associated with the client terminal when a sticker and effect associated with the advertisement is selected by user input. A live streaming image generating apparatus comprising:
A communication unit for communicating with the server;
A display unit for displaying a first streaming image generated based on platform information for generating a streaming image obtained from the server through the communication unit; And
Wherein at least one of a sticker and an effect for decorating at least a part of the first streaming image to be displayed is selected by user input and at least one of the selected stickers and effects is added to the first streaming image And generating a second streaming image by applying the second streaming image in real time. A method for providing a live streaming service in a server device providing a live streaming service,
Providing platform information associated with live streaming to a first client terminal;
Receiving, from the first client terminal, a second streaming image in which at least one of a sticker and an effect for decorating at least a part of a first streaming image generated based on the platform is applied in real time; And
And providing a second streaming image obtained from the first client terminal to a second client terminal. 18. The method of claim 17,
Wherein the platform information includes information about at least one of a sticker and an effect. 18. The method of claim 17,
Wherein the live streaming service includes a live broadcast service based on the first client terminal,
And the second client terminal is a terminal associated with a viewer who has entered a broadcasting station established by the first client terminal. 18. The method of claim 17,
Wherein the first streaming image includes a streaming image generated based on a subject photographed by a camera. 21. The method of claim 20,
Wherein the platform information includes information on a tracking algorithm for tracking the subject in real time to obtain tracking information,
Wherein the second streaming image is generated by applying at least one of the selected stickers and effects to the subject based on the tracking information obtained using the tracking algorithm. 18. The method of claim 17,
Wherein the sticker includes a sticker that arbitrarily sets positional information in the first streaming video according to a user input to cover a part of the first streaming video. 18. The method of claim 17,
Wherein the effect includes the effect of disabling at least a portion of the sticker applied to the first streaming image by a user input at the second client terminal. 24. The method according to claim 22 or 23,
At least one of user input at the first client terminal for applying the sticker and user input at the second client terminal for activating an effect of disappearing at least a portion of the applied sticker is associated with the first client terminal A user account, and a user account associated with the second client terminal. ≪ RTI ID = 0.0 > 31. < / RTI > 18. The method of claim 17,
Wherein at least one of the sticker and the effect comprises at least one of a sticker and an effect associated with an advertisement. 26. The method of claim 25,
At least one of a sticker and an effect associated with an advertisement applied to the streaming image by a user associated with the second client terminal when the streaming image is generated by applying at least one of a sticker and an effect associated with the advertisement at the first client terminal Upon detecting that it is performing an action associated with the advertisement based on the advertisement,
Related information corresponding to at least one of a sticker and an effect associated with the advertisement applied to the streaming video,
And sharing at least a portion of the revenue associated with the advertisement to a user associated with at least one of the first client terminal and the first client terminal based on the identified advertisement related information. 27. The method of claim 26,
Wherein sharing of revenue associated with the advertisement is performed by estimating the number of times of application of at least one of a sticker and an effect associated with an advertisement applied to the second streaming image. 18. The method of claim 17,
And providing a sticky / effect selection menu including at least one of a sticker candidate and an effect candidate applicable at the first client terminal based on the platform information. 29. The method of claim 28,
At least one of the sticker candidates and the effect candidates included in the sticker / effect selection menu includes at least one of a sticker candidate and an effect candidate corresponding to at least one of a history, a content, and a service type of the user associated with the first client terminal A method for providing a live streaming service composed of one. An apparatus for providing a live streaming service,
A communication unit for communicating with the first client terminal and the second client terminal; And
A sticker for transmitting platform information associated with live streaming provided to the first client terminal through the communication unit and for decorating at least a part of the first streaming video generated based on the platform from the first client terminal, And a controller for receiving a second streaming video and applying a second streaming video obtained from the first client terminal to the second client terminal, wherein the second streaming video is applied to at least one of the first client terminal and the second client terminal. In a live streaming system,
A first streaming image generated based on platform information for generating a streaming image, and a sticker and an effect for decorating at least a part of the first streaming image displayed by a user input A first client terminal for generating a second streaming image by applying at least one of the selected stickers and effects to the first streaming image in real time by selecting at least one streaming image;
A server device for providing platform information related to live streaming to a first client terminal, receiving the second streaming video from the first client terminal and providing the second streaming video to a second client terminal; And
And a second client terminal receiving the second streaming video.

Images