JP7159067B2 - Computer program and distribution server - Google Patents

Description

The present invention relates to live distribution of moving images in a moving image sharing system.

2. Description of the Related Art In recent years, some moving image sharing systems allow live distribution of moving images shot by users (hereinafter simply referred to as distributors) to a large number of spectators via the Internet. In such videos live-delivered via the Internet (hereinafter simply referred to as live programs), the distributors are assigned available time slots in the same way as television programs.

Television programs are not limited to ordinary recorded programs, and even live broadcast programs often have openings, endings, and other intermediate effects. However, the technology and human resources available to live program distributors are usually limited compared to TV program production companies, so it is not possible to add effects such as openings and endings to videos shot by distributors. It is realistically difficult to broadcast live. For this reason, live programs tend to abruptly begin and end abruptly in accordance with the allotted time frame, which is uncomfortable for audiences accustomed to TV program-like programming. Maybe.

Also, immediately after the start of live distribution, only a limited number of spectators, mainly followers of the distributor, often gather. Since the broadcaster wants to start the main project after gathering as many spectators as possible, there is a sense of procrastination, such as waiting for a while, conducting a mic test, and depending on the case, leaving the seat. Scenes can go on and on. When the audience encounters such a scene, they may feel bored and stop watching.

In Patent Document 1, when receiving a distribution request from a viewer (S100) ([0027]) in a program distribution system for individuals, program content stored in a program content database is extracted (S106), and programs for individuals are extracted. Contents are organized (S108) ([0028]) and distributed to viewer terminals ([0030]). Also, it is described that organized personal program content includes a title and an ending ([0027], [0029]).

In addition, in Patent Document 2 (especially [0022], [0057], [0059], abstract), a background portion is cut out from captured moving image data and synthesized with a background image or moving image data prepared in advance. A technique is disclosed for creating synthetic moving image data, further editing the synthetic moving image data to generate integrated moving image data including opening title and end roll data, and sharing the integrated moving image data via a network.

Japanese Patent Application Laid-Open No. 2006-339780 JP 2016-154021 A

Although Patent Document 1 describes that program content is distributed in a state in which real-time content is included at any time, this real-time content includes, for example, regular news, national news, local news limited to a region, and weather forecast information. ([0021]) including general current affairs information such as ( [0029]). Here, the news coverage moving image generally refers to a recorded moving image, so the real-time content described in Patent Document 1 is different from the live program. Also, the title 810 and the ending 850 included in the personal program content 800 are treated as separate content from the real-time content 820 (FIG. 3, [0029]).

The technique described in Patent Literature 2 generates and shares integrated moving image data based on fully captured moving images, and is not expected to be applied to live programs. Also, the opening title and the end credits are data independent of the composite video, and it is not assumed that the two will be composited.

Therefore, even if these techniques are applied to distribution of live programs, titles, openings, endings, end rolls, etc., are added independently of the main part of the live programs. Therefore, it is easy to improve the abruptness at the beginning and end of live programs, the dullness immediately after the start, and the lack of sharpness due to the limited range of expressions that can be used, which are unique to user-generated live programs. not.

SUMMARY OF THE INVENTION An object of the present invention is to automatically add effects to video data for live distribution to make it look good.

A computer program according to a first aspect of the present invention comprises: receiving at least video data for live distribution from a distributor terminal; determining whether or not a rendering condition is satisfied; A computer that, when satisfied, adds an effect to the video data for live distribution and distributes the video data with the effect added to the distributor terminal or the spectator terminal. In the program, the effect condition is a criterion for determining whether or not to add an opening effect, an ending effect, or an intermediate effect, and the determination is based on the opening effect, the ending effect, or the intermediate effect. and determining whether or not the rendering conditions are satisfied for each of the above. The performance conditions include at least a time frame allocated for live distribution of the video data, the number of spectators of the video data, an instruction to start the opening performance, comments posted on the video data, and video data. Alternatively, the effect condition may be determined to be satisfied after a predetermined condition is satisfied until the effect end condition is satisfied.

A distribution server according to a second aspect of the present invention includes a receiving unit that receives at least video data for live distribution from a distributor terminal, a determination unit that determines whether or not an effect condition is satisfied, and the effect a distribution unit that distributes data added with effects to the video data for live distribution to the distributor terminal or the spectator terminal when conditions are satisfied, and the effect conditions are an opening effect and an ending. It is a criterion for determining whether or not to add an effect or an intermediate effect, and the determination unit determines whether or not the effect condition is satisfied for each of the opening effect, the ending effect, and the intermediate effect. determine whether The performance conditions include at least a time frame allocated for live distribution of the video data, the number of spectators of the video data, an instruction to start the opening performance, comments posted on the video data, and video data. Alternatively, the effect condition may be determined to be satisfied after a predetermined condition is satisfied until the effect end condition is satisfied.

A computer program according to a third aspect of the present invention comprises: receiving at least video data for live distribution from a server ; obtaining a determination as to whether or not rendering conditions are satisfied; A computer program for causing a computer to add an effect to the video data for live distribution and to display the video data to which the effect is added on a display screen when a condition is satisfied. The effect condition is a criterion for determining whether or not to add an opening effect, an ending effect, or an intermediate effect. For each of them, it is determined whether or not the performance conditions are satisfied. The performance conditions include at least a time frame allocated for live distribution of the video data, the number of spectators of the video data, an instruction to start the opening performance, comments posted on the video data, and video data. Alternatively, the effect condition may be determined to be satisfied after a predetermined condition is satisfied until the effect end condition is satisfied.

According to the present invention, it is possible to automatically add an effect to video data for live distribution and adjust the appearance.

1 is a block diagram illustrating a live video delivery system according to a first embodiment; FIG. 2 is a block diagram illustrating a distribution server according to the first embodiment; FIG. The figure which illustrates video data. FIG. 4 is a view exemplifying synthesized video data obtained by synthesizing effect data with the video data of FIG. 3; FIG. 4 is a view exemplifying synthesized video data obtained by synthesizing effect data with the video data of FIG. 3; FIG. 4 is a view exemplifying synthesized video data obtained by synthesizing effect data with the video data of FIG. 3; FIG. 4 is a view exemplifying synthesized video data obtained by synthesizing effect data with the video data of FIG. 3; 3 is a flowchart illustrating the operation of the distribution server of FIG. 2; FIG. 11 is a block diagram illustrating a distribution server according to the second embodiment; FIG. FIG. 4 is a block diagram showing a modification of the terminal according to the first embodiment and the second embodiment; FIG. FIG. 11 is a block diagram illustrating a distribution server according to the third embodiment; FIG. The figure which illustrates video data. FIG. 13 is a view exemplifying processed video data obtained by processing the video data of FIG. 12; FIG. 13 is a view exemplifying processed video data obtained by processing the video data of FIG. 12; FIG. 11 is a block diagram showing a modification of the terminal according to the third embodiment; FIG. 11 is a block diagram illustrating a distribution server according to the fourth embodiment; FIG. FIG. 4 is a view exemplifying synthesized video data obtained by synthesizing dynamic effect data with the video data of FIG. 3; FIG. 4 is a view exemplifying synthesized video data obtained by synthesizing dynamic effect data with the video data of FIG. 3; FIG. 12 is a block diagram illustrating a terminal according to the fourth embodiment; FIG. FIG. 17 is a block diagram showing a modification of FIG. 16;

An embodiment will be described below with reference to the drawings. Elements that are the same as or similar to elements that have already been explained are denoted by the same or similar reference numerals, and overlapping explanations are basically omitted. For example, when there are a plurality of identical or similar elements, common reference numerals may be used to describe each element without distinction, and the common reference numerals may be used to distinguish and describe each element. In addition, branch numbers are sometimes used.

(First embodiment)
The distribution server according to the first embodiment can be incorporated into the live video distribution system illustrated in FIG. This live distribution system includes a distributor terminal 100, a distribution server 200, and spectator terminals 300-1, 300-2, .

The distribution server 200 is connected to the distributor terminal 100 and the spectator terminal 300 via a network, and can exchange data with each other. It should be noted that the number of each device shown in FIG. 1 is merely an example. For example, the number of spectator terminals 300 can be 0, or can be hundreds or thousands. Moreover, a Web server or a comment distribution server, which are not shown in FIG.

Distributor terminal 100 is an electronic device such as a computer connected to a video source such as a video camera, such as a television receiver (including Internet television), a PC (Personal Computer), a mobile terminal (such as a tablet, a smartphone, laptops, feature phones, portable game machines, digital music players, electronic book readers, etc.), VR (Virtual Reality) terminals, AR (Augmented Reality) terminals, etc., but are not limited to these. The distributor terminal 100 sequentially transmits the encoded video data output from the video source to the distribution server 200 .

The distribution server 200 receives the encoded video data sequentially transmitted from the distributor terminal 100 over the time frame assigned to the live distribution of the video data. The distribution server 200 basically distributes this video data to the distributor terminal 100 and the spectator terminal 300 as it is.

However, when the effect condition described later is satisfied, the distribution server 200 synthesizes the effect data described later with the video data, and transmits the synthesized video data (hereinafter referred to as synthesized video data) to the distributor terminal 100. and distributed to the spectator terminal 300 . The determination of satisfaction/dissatisfaction of the rendering conditions, the creation of rendering data, and the synthesis of rendering data may be realized by the distribution server 200 executing a script, for example.

Spectator terminal 300 can be an electronic device similar to broadcaster terminal 100, but unlike broadcaster terminal 100, does not need to be connected to a video source.

By having the distribution server 200 automatically synthesize such effect data, the distributor can easily use, for example, an opening effect, an ending effect, or an intermediate effect. Since such effects are added by the distribution server 200, there is no problem even if the distributor does not have advanced video editing skills, hardware or software for video editing, or does not have a supporter such as a director. Therefore, the distributor can easily live-distribute a moving image that is formatted like a TV program.

Hereinafter, the configuration and operation of distribution server 200 in FIG. 1 will be described in order using the drawings.

The distribution server 200 is a computer that controls distribution of video data received from the distributor terminal 100, determines performance conditions, creates performance data, synthesizes performance data, and performs various functions received from the distributor terminal 100 and the audience terminal 300. A processor that interprets requests and generates responses to them, and a memory that temporarily stores programs executed by the processor and data used by the processor to realize such processing contains.

Distribution server 200 can also use a communication device for connecting to a network and an auxiliary storage device for storing large amounts of data. The communication device and auxiliary storage device may be built in distribution server 200 or may be externally attached to distribution server 200 .

The communication device communicates with the distributor terminal 100 and the audience terminal 300 via the network. For example, the communication device receives video data from the distributor terminals 100 and transmits responses to distribution requests from the audience terminals 300 .

The auxiliary storage stores data such as effect conditions and effect data, for example. The auxiliary storage device is preferably a non-volatile storage medium such as HDD (Hard Disk Drive) or SSD (Solid State Drive). The auxiliary storage device may be a file server connected to distribution server 200 via a network.

Next, the description of the configuration example of the distribution server 200 will be continued with reference to FIG. Distribution server 200 in FIG. , and a video distribution unit 208 .

A receiving unit 201 receives various data via a network. The receiving unit 201 may be the communication device described above, or may be an interface with the communication device. Specifically, the receiving unit 201 receives video data for live distribution from the distributor terminal 100 . The receiving unit 201 sends this video data to the effect condition determining unit 204 . Also, the reception unit 201 receives the video data distribution request from the spectator terminal 300 . Receiving section 201 sends a distribution request to distribution control section 203 .

Furthermore, the receiving section 201 may receive information indicating that a predetermined user input has been made at the distributor terminal 100 or the spectator terminal 300 and send this to the effect condition determining section 204 . Further, the reception unit 201 may receive a comment posted on the video data from a comment distribution server (not shown in FIG. 1) and send it to the rendering condition determination unit 204 . If distribution server 200 has the function of a comment distribution server, reception unit 201 can directly receive comments from distributor terminal 100 or audience terminal 300 and send them to effect condition determination unit 204 .

A clock unit 202 indicates the date and time. The clock unit 202 includes, for example, a crystal oscillator that oscillates at a fixed frequency, a frequency dividing circuit that divides the output of the oscillator to obtain a signal of 1 Hz, and a counter that counts this signal and obtains a serial value indicating the date and time. . The date and time indicated by the clock unit 202 can be referred to by the distribution control unit 203 and the effect condition determination unit 204 .

The distribution control unit 203 controls live distribution of video data. The delivery control unit 203 may be the aforementioned processor and memory.
Specifically, the distribution control unit 203 compares the date and time indicated by the clock unit 202 with the time frame assigned to the live distribution of the video data, and manages the start and end of the live distribution of the video data. The distribution control unit 203 also receives a distribution request from the spectator terminal 300 from the receiving unit 201 and controls the video distribution unit 208 so that the video data is distributed to the spectator terminal 300 . Note that the distribution control unit 203 may manage the number of spectators based on the received distribution request. Information such as the time frame and the number of spectators managed by the distribution control unit 203 can be referred to by the effect condition determination unit 204 or the effect data creation unit 206 .

The effect condition determination unit 204 receives the video data from the reception unit 201, reads the effect conditions for the video data from the effect condition storage unit 205, and determines whether the effect conditions are satisfied. The effect condition determination unit 204 sends the video data to the effect data synthesizing unit 207 when determining that the effect condition is satisfied, and otherwise sends the video data to the video distribution unit 208 . The effect condition determination unit 204 may be the aforementioned processor and memory.

In order to determine whether or not the rendering conditions are satisfied, the performance condition determination unit 204 uses the date and time indicated by the clock unit 202, the time frame managed by the distribution control unit 203, the number of spectators, and the receiving unit as necessary. Comments, information indicating that a predetermined user input has been made at the distributor terminal 100 or the spectator terminal 300, reproduced video or reproduced audio of video data, etc., received by 201 can be referred to.

Here, the effect condition is, for example, a criterion for determining whether or not to add an opening effect, an ending effect, or an intermediate effect. Specifically, the effect condition for adding the opening effect (opening effect condition) is set at the beginning of the time frame assigned to the video data, or at the effect start point set after a predetermined time from the start. It may be determined to be satisfied in a period from the arrival of the date and time instructed by the unit 202 to the satisfaction of the rendering end condition.

The opening effect condition may be determined so as to be satisfied during the period from when the number of spectators of the video data satisfies a predetermined condition to when the effect end condition is satisfied. The condition here can be, for example, that the number of spectators has reached a predetermined number.

The opening effect condition may be determined so as to be satisfied during a period from when a predetermined user input is made at the distributor terminal 100 or the spectator terminal 300 to when the effect end condition is satisfied. The predetermined user input may be, for example, selection of a GUI (Graphical User Interface) part such as a button for instructing the start of a presentation displayed on the distributor terminal 100 .

The opening effect condition may be determined so as to be satisfied after the comment posted to the video data satisfies a predetermined condition until the effect end condition is satisfied. The conditions here can be, for example, that the number of comments has reached a predetermined number, that the number of comments that include a specific keyword has reached a predetermined number, that there is a comment that includes a rare word, or the like.

The opening effect condition may be determined so as to be satisfied during a period from when the reproduced video or reproduced sound of the video data satisfies a predetermined condition to when the effect end condition is satisfied. The conditions here are, for example, that it is recognized from the reproduced video that the distributor has taken a predetermined gesture or facial expression, that from the reproduced voice that the distributor has uttered a predetermined phrase, and the like. could be.

The condition for ending the opening effect is, for example, when a predetermined time has passed since the start of the effect, when the number of spectators or the number of comments reaches a required number, or when there is a predetermined user input to the distributor terminal 100 or the spectator terminal 300. Satisfaction is satisfied when, for example, it is recognized from the reproduced image that the distributor has taken a predetermined gesture or facial expression from the reproduced video data, or when it is recognized from the reproduced voice that the distributor has uttered a predetermined phrase, for example. may be defined to

By starting the opening performance from a point shifted from the beginning of the time frame, the video data is delivered as it is immediately after the start of the live distribution, and the opening performance starts when the audience has gathered to a certain extent, and the audience to the main part of the live program. It is possible to motivate continuous viewing by raising expectations of. On the other hand, by starting the opening effect from a point that coincides with or is close to the start of the time frame, it is possible to prevent the earliest spectators from leaving without waiting for the opening effect.

In addition, the effect condition for adding the ending effect (ending effect condition) is that the date and time indicated by the clock unit 202 reaches the effect start point set a predetermined time before the end of the time frame assigned to the video data. It may be determined to be satisfied during the period from when the production end condition is satisfied until the condition is satisfied. In addition, the ending effect condition can be determined in the same manner as the opening effect condition described above.

The rendering termination condition for the ending rendering may be the same as the rendering termination condition for the opening rendering, or is determined to be satisfied when the date and time indicated by the clock unit 202 reaches the end of the time frame assigned to the video data. may be

Intermediate effects may be realized, for example, by synthesizing effect images such as fanfare, and replay images or slow-motion playback images with the image data in the middle of the time frame assigned to the image data. The replay video or slow-motion playback video may be the previous video, but is not limited to this. Also, when a live program is shot with a multi-camera, images from different viewpoints may be synthesized or viewpoints may be switched.

The effect condition for adding the intermediate effect (intermediate effect condition) is set after the date and time indicated by the clock unit 202 reaches the effect start point set after a predetermined time from the start of the time frame assigned to the video data. , may be determined to be satisfied until the rendering end condition is satisfied. In addition, if the distributor or spectator clicks on the advertisement or product introduction given to the live program, or if the distributor or spectator purchases the product targeted by the advertisement or product introduction, the intermediate performance condition will be It may be determined to be satisfied during a predetermined period of time after the purchase until the performance end condition is satisfied. Here, advertisements and product introductions can be displayed, for example, in the display page of the live program on the distributor terminal 100 or the audience terminal 300, or on a page accessible from the display page. In addition, the intermediate effect condition can be determined in the same manner as the opening effect condition described above. Also, the effect ending condition for the intermediate effect may be the same as the effect ending condition for the opening effect.

For example, when the number of spectators of the video data or the number of comments posted on the video data rises sharply, an intermediate effect of synthesizing a fanfare video may be started. Also, for example, when more than a predetermined number of comments including specific keywords such as "replay", "slow", and "one more time" are posted to the video data within a predetermined time, the replay video or slow motion playback video is displayed. An intermediate effect to be synthesized may be started.

The effect condition storage unit 205 stores various effect conditions described above. The effect conditions stored in the effect condition storage unit 205 are read by the effect condition determination unit 204 and the effect data creation unit 206 as necessary. The rendering condition storage unit 205 may be the aforementioned auxiliary storage device, or may be an interface with the auxiliary storage device.

The effect data creating unit 206 creates effect data and sends it to the effect data synthesizing unit 207 . The effect data creation unit 206 may be the aforementioned processor and memory.
Effect data can be broadly classified into dynamic effect data and static effect data. Static effect data can be fixed moving image data or still image data that is not affected by the contents of video data. Fixed video data may be, for example, various animations or live-action videos with areas for embedding playback video of the video data to be live-distributed. The fixed still image data may be, for example, a frame that decorates the edge of the screen of the reproduced video of the video data, an object that covers the reproduced video of the video data, or the like. If dynamic effect data, which will be described later, is not used, the effect data creation unit 206 can be omitted by presetting static effect data.

The dynamic rendering data may be, for example, moving image data or still image data including information dynamically indicating the necessary conditions for satisfying the rendering end condition, or may be dynamic image data or still image data after live distribution of the video data is started. It may be video data or still image data containing information about an event that occurred, replay video or slow-motion playback video of video data to be distributed live, or video from another viewpoint. Alternatively, it may be moving image data or still image data including, for example, moving image titles, credit titles, distributor's comments or explanations. Such information may be extracted from information entered by the broadcaster, audience, or service operator prior to the start of the live broadcast, or information automatically provided by broadcast server 200 or other servers. For example, production data is information automatically generated from metadata such as video tags, which are keywords assigned to live programs, or sponsor names assigned to live programs. It may be moving image data or still image data including a URL (Uniform Resource Locator). Here, the video tag and sponsor name may be manually assigned by the distributor, spectator or service operator, or may be automatically assigned by the distribution server 200 or another server.

Additionally, dynamic rendition data may be automatically generated based on information displayed on web pages related to live programming. For example, on a program introduction page such as the front page of a moving image sharing service, thumbnails of many live programs that are currently open to the public may be displayed side by side. In such a program introduction page, supplementary information about the live program such as "rising popularity" may be displayed around the thumbnail. To the live program corresponding to the thumbnail to which such supplementary information is attached, the supplementary information itself or effect data including information extracted and processed from the supplemental information may be applied, or the effect data to be applied may be applied to the thumbnail. It may be switched depending on the supplementary information. For example, the effect data may include a program title automatically edited based on information extracted from the supplementary information, a sponsor name extracted from the supplementary information, and the like.

Specifically, the effect data creation unit 206, as a necessary condition for satisfying the effect end condition, indicates the gap between the current number of spectators or the number of comments in the video data and the required number, for example, in a countdown format. Effect data including texts such as "more XX people" and "more XX comments" may be created. Information on the current number of spectators or the number of comments for video data can be obtained from the distribution control unit 203, for example. Information on the number of spectators or the number of comments required to satisfy the rendering end condition can be obtained from the rendering condition storage unit 205, for example. By displaying such information, the audience can grasp the estimated time until the main part of the live program starts, and can decide whether or not they can wait until then and whether or not they can leave their seats.

In addition, the effect data creation unit 206 includes, as information related to events occurring after the start of live distribution of the video data, for example, the number of spectators of the video data, part of comments posted on the video data (eg, digest), Number of comments posted on video data, names of spectators who posted comments on video data, number of clicks on advertisements and product introductions given to live programs, information on products purchased from advertisements and product introductions ( For example, production data including information indicating product name, number of purchases, total purchase price, photograph, purchaser information, etc.) may be created. By using such information as, for example, the end credits, it is possible to produce an ending effect suitable for the mode of live distribution of individual video data.

Moreover, the effect data creation unit 206 may create effect data that changes in response to a user input made to the distributor terminal 100 or the spectator terminal 300 or a comment posted by the distributor terminal 100 or the spectator terminal 300. good. For example, the production data includes objects that block the screen of the playback video of the video data (for example, confetti, a symbol of "8" meaning applause, a bouquet, a heart mark, a cloud, a crowd of people, an icon set by the distributor, a moving image, etc.). mascot characters of shared services).

In this case, the effect data creating unit 206 creates effect data in which the entire screen is covered with objects at the start of the effect, and the distributor terminal 100 or the spectator terminal 300 receives a predetermined user input (for example, an object or a button). (selection of GUI) or a comment is posted, the object may be erased, moved, or reduced in size so that the screen of the reproduced video of the video data can be gradually seen. Conversely, the effect data creation unit 206 creates effect data including zero or a small number of objects at the start of the effect, and performs predetermined user input (for example, selection of a GUI such as an object or a button) to the distributor terminal 100 or the spectator terminal 300 . ), or when a comment is posted, the object may appear, move, or enlarge so that the screen of the reproduced video of the video data gradually disappears.

The effect of gradually showing the screen of the reproduced video of the video data is suitable for the opening, but is not limited to this. An effect in which the screen of the reproduced image of the video data gradually disappears is suitable for the ending, but is not limited to this.

Note that even if the object states are not completely synchronized among all the terminals, the viewing experience of the main part of the live program is not affected, so the frequency of synchronization may be reduced to reduce the load on the server.

Also, in this example, the rendering end condition may be determined by the number, size, position, or the like of the objects. In this case, the distributor and the audience determine the necessary conditions for satisfying the rendering end condition from the state of the objects included in the reproduced video of the composite video data, for example, how many more objects need to be reduced, how much more objects need to be increased, and so on. You can figure out what you need. Therefore, such an object also corresponds to an example of information dynamically indicating a necessary condition for satisfying the rendering end condition.

The effect data synthesis unit 207 receives video data from the effect condition determination unit 204 and receives effect data from the effect data creation unit 206 . The effect data synthesizing unit 207 synthesizes the effect data with the video data to generate synthesized video data. The rendering data synthesizing unit 207 may synthesize rendering data with video data using known techniques such as alpha blending and picture-in-picture. Effect data synthesizing section 207 sends the synthetic video data to video distribution section 208 . The effect data synthesizing unit 207 may be the aforementioned processor and memory. The type of effect data used for synthesis may be automatically determined by a predetermined algorithm, may be determined by user input information from the distributor and/or the audience, or may be determined by a combination of both. may For example, the number of clicks on advertisements and product introductions given to live programs, information on products purchased from advertisements and product introductions, such as the number of products purchased, the total purchase amount, etc. may be switched.

For example, the effect data synthesizing unit 207 may resize a part or all of the screen of the reproduced video of the video data, or trim a part of the screen and synthesize it with the effect data, or may combine the reproduced video of the video data. The image data may be combined with the effect data such that a part of the screen is exposed and the rest is hidden. FIG. 3 illustrates video data, and FIGS. 4 to 7 illustrate composite video data.

As illustrated in FIG. 4, the effect data synthesizing unit 207 resizes a part (for example, the distributor's face area) and the entire screen of the reproduced video of the video data in FIG. may be embedded in moving image data or still image data. In addition, in the example of FIG. 4, the effect data includes the text "Appearance XX" as the credit title. In this way, by resizing the entire screen of the reproduced image of the video data and combining it with the effect data, it is possible to prevent the screen from being masked by the effect data. Therefore, it is possible to prevent the occurrence that the part that the distributor wants to show to the audience is hidden by the effect data. On the other hand, by resizing a part of the screen of the reproduced video of the video data rather than the whole screen to a smaller size or a larger size and then synthesizing it with the performance data, the part that the distributor wants to show to the audience is not excessively reduced or emphasized. can do. The part of the screen may be a static area, such as an area near the center of the screen, or a dynamic area, such as an area where the face of the broadcaster is present or an area with large movements. When extracting a dynamic area, for example, known face recognition technology, moving object detection technology, or the like may be used.

In this way, by resizing the screen of the reproduced video of the video data and then synthesizing it with the effect data, the distributor and the audience are made aware that a special effect different from the main part of the live program is added. be able to.

As illustrated in FIGS. 5 and 6, the effect data synthesizing unit 207 synthesizes the effect data with the video data of FIG. 3, and generates synthesized video data in which an object blocking the screen of the reproduced video of the video data is synthesized. You may In the example of FIG. 5, at the start of the production, the entire screen is filled with mascot character silhouette objects with a TV motif. It gradually disappears and the production is completed. Also, in the example of FIG. 6, at the start of the presentation, a small number of "8" symbols are piled up at the bottom of the screen, or they are not piled up. By posting a comment, this object gradually accumulates and the production is completed. 5 and 6, objects may automatically disappear or may automatically pile up over time even if there is no user input.

As illustrated in FIG. 7, the effect data synthesizing unit 207 resizes the entire screen of the reproduced video of the video data in FIG. good too. In the example of FIG. 7, information such as the number of spectators in the video data of FIG. 3 and the names of spectators who posted comments is displayed as part of the end credits. Note that the example of FIG. 7 is not limited to the ending effect, and may be applied as an intermediate effect for concluding the first half of the live distribution, for example.

The video distribution unit 208 receives the video data from the effect condition determination unit 204 or the composite video data from the effect data synthesis unit 207 and distributes it to the distributor terminal 100 and the audience terminal 300 . The video distribution unit 208 is controlled by the distribution control unit 203, for example, the distribution destination of the video data. The video distribution unit 208 may be the communication device described above, or may be an interface with the communication device.

Note that the video distribution unit 208 may immediately discard the distributed video data (or composite video data), but buffers it for a predetermined period of time in order to enable the replay or slow-motion reproduction described above, for example. , and may be allowed to be referenced by the effect data creation unit 206 .

Next, an operation example of the distribution server 200 will be described with reference to FIG. The operation of FIG. 8 begins, for example, with the beginning of the time frame assigned to the video data. When the operation of FIG. 8 starts, the process proceeds to step S401.

In step S<b>401 , the receiving unit 201 receives video data for live distribution from the distributor terminal 100 . The rendering condition determination unit 204 determines whether predetermined rendering conditions are satisfied for the video data received in step S401 (step S402). If the effect condition is satisfied, the process proceeds to step S403, otherwise the process proceeds to step S405.

In step S403, the effect data synthesizing unit 207 synthesizes the video data received in step S401 with the effect data. Note that the effect data synthesizing unit 207 may perform processing such as resizing and trimming of the reproduced image of the video data as necessary, and then synthesize it with the effect data. Then, the video distribution unit 208 distributes the synthesized video data generated in step S403 to the spectator terminal 300 (step S404). On the other hand, in step S<b>405 , the video distribution unit 208 distributes the video data received in step S<b>401 to the spectator terminal 300 .

At the end of step S404 or step S405, if the date and time indicated by the clock unit 202 has reached the end of the time frame assigned to the video data to be live-distributed, the operation of FIG. 8 ends; The process returns to step S401 (step S406).

As described above, the distribution server according to the first embodiment conditionally and automatically synthesizes effect data with video data for live distribution. As a result, distributors do not need advanced video editing skills or resources, and can easily live-distribute videos that are well-formed like TV programs, with additions such as opening effects, ending effects, and intermediate effects. can be done.

In particular, by adding an opening performance, the broadcaster and the audience are made aware of the beginning of the live program, and the broadcaster prepares for the live program by taking a mic test and taking a break, and comments before the main part of the live program. It is possible to communicate with the audience via the program, or to have the audience wait for the start of the main part of the live program without feeling too slow.

In addition, by adding an ending effect, the distributor and the audience are made aware of the end of the live program, and the distributor enters the summary, makes a notice for the next time, communicates with the audience through comments, etc. Makes it easier to take time.

In addition, as an intermediate production, the range of expressions that can be used in live distribution is expanded by adding visual effects such as fanfare to inspire the distributor in exciting scenes, replaying decisive moments, and slow motion playback. can be expanded.

(Second embodiment)
The distribution server according to the second embodiment allows the distributor and/or the audience to specify how the live distribution video data and effect data are to be synthesized. Specifically, you can specify what layout (alignment) to merge the two, that is, how much to translate one or both, and how to rotate, flip, resize, and/or trim one or both. It is possible to specify settings.

FIG. 9 illustrates a distribution server 500 according to the second embodiment. In the following description, the points of difference between the distribution server 500 and the distribution server 200 will be mainly described. The distribution server 500 receives the encoded video data sequentially transmitted from the distributor terminal 100 over the time frame assigned to the live distribution of the video data. When the performance condition is satisfied, the distribution server 500 synthesizes the performance data with the video data according to the specification of the distributor and/or the audience, and distributes the synthesized video data to the distributor terminal 100 and the audience terminal 300. - 特許庁Here, the rendering conditions may be the same as or similar to the rendering conditions described in the first embodiment.

By having the distribution server 500 perform such automatic synthesizing of effect data in accordance with the specifications of the distributor and/or the audience, the range of expression of the opening effect, ending effect, or intermediate effect described in the first embodiment can be further expanded. be able to. For example, when video data is embedded into rendering data and combined, it is possible to make the data look more natural or deliberately make it look unnatural.

The functional configuration of distribution server 500 will be described below with reference to FIG. Note that the hardware configuration of the distribution server 500 may be the same as or similar to that of the distribution server 200, so the description thereof will be omitted.

The distribution server 500 includes a reception unit 501, a clock unit 502, a distribution control unit 503, an effect condition determination unit 504, an effect condition storage unit 505, an effect data creation unit 506, an effect data synthesis unit 507, an image and a distribution unit 508 . Of these, the clock unit 502, the distribution control unit 503, the effect condition storage unit 505, the effect data creation unit 506, and the video distribution unit 508 may be the same as or similar to the same-named elements in the distribution server 200, respectively. Other elements will be described below, focusing on differences from the same-named elements in the distribution server 200 .

The receiving unit 501 can receive specified data from the distributor terminal 100 and/or the audience terminal 300 in addition to the data received by the receiving unit 201 . This designation data designates how the video data for live distribution and the effect data are synthesized. Specifically, the specified data includes the combining position of the video data and the effect data, the enlargement/reduction ratio of the video data, the enlargement/reduction ratio of the effect data, the amount of rotation of the image data, the amount of rotation of the effect data, and the specified data in the image data. , as well as specific areas in the rendering data, and the like. The receiving unit 501 sends this designated data to the effect condition determining unit 504 .

The effect condition determination unit 504 sends specified data to the effect data synthesizing unit 507 in addition to the video data when the effect condition is satisfied. If the distributor and the audience do not specify how to synthesize the video data for live distribution and the rendering data, there may be no designated data.

The effect data synthesizing unit 507 receives the video data and the specified data from the effect condition determining unit 504 and receives the effect data from the effect data creating unit 506 . The effect data synthesizing unit 507 pre-processes the video data and/or the effect data in accordance with the specified data, and then synthesizes both to generate synthesized video data.

Specifically, when the designation data designates the combining position of the video data and the effect data, the effect data synthesizing unit 507 can translate the video data and/or the effect data to control the combining position of both.

Further, when the designated data designates an enlargement/reduction rate of video data and/or an enlargement/reduction rate of effect data, the effect data synthesizing section 507 can resize the video data and/or the effect data. Furthermore, when the designation data designates the amount of rotation of the video data and/or the amount of rotation of the effect data, the effect data synthesizing section 507 can rotate the video data and/or the effect data. Here, the rotation is not limited to two-dimensional rotation on the image plane, and may mean three-dimensional rotation including up/down and left/right inversion and combinations thereof.

Furthermore, if the specified data specifies a specific area in the video data and/or a specific area in the effect data, the effect data synthesizing unit 507 can trim the specified area in the video data and/or the effect data. .

By performing the preprocessing as described above, the effect data synthesizing unit 507 can synthesize the video data and the effect data according to the preferences of the distributor and/or the audience. As an example, when the performance data is a frame without a face, it is possible to synthesize the face of the video data and the embedded portion of the frame without the face so as to look natural or intentionally unnatural.

As described above, the distribution server according to the second embodiment receives, from the distributor terminal and/or the audience terminal, designation data that designates how to synthesize the video data and the effect data, and distributes the video image accordingly. The data and/or the performance data are preprocessed and then synthesized. Therefore, according to this distribution server, it is possible to further expand the expression range of the opening effect, ending effect, or intermediate effect described in the first embodiment.

(Modification)
By the way, in the above-described first embodiment and this embodiment, the distribution server synthesizes the video data and the effect data. This has the advantage of facilitating synchronization of the synthesized video data viewed by the distributor and the audience, and reducing the processing load on the terminal. It also has the advantage of reducing the amount of communication from the distribution server to the terminal. However, it is also possible for the terminal to combine the video data and the effect data. A brief description will be given below.

A terminal 600 according to this modification is illustrated in FIG. Terminal 600 may be either a distributor terminal or a spectator terminal, so explanations of the parts specific to each of the distributor terminal and spectator terminal will be omitted.

Terminal 600 can be any of the electronic devices described with respect to distributor terminal 100 . Terminal 600 includes, for example, a processor that performs input/output control, communication control, determination of rendering conditions, creation of rendering data, synthesis of video data and rendering data, and the like. Here, the processor is typically a CPU (Central Processing Unit) and/or a GPU (Graphics Processing Unit), but may be a microcomputer, FPGA (Field Programmable Gate Array), DSP (Digital Signal Processor), or the like. may Terminal 600 also includes a memory that temporarily stores programs executed by a processor and data used by the processor to implement such processing.

Note that the terminal 600 may handle all data in the on-memory state, or part of the data may be saved in an auxiliary storage device. The auxiliary storage device may be, for example, an internal or external HDD, SSD, flash memory, or the like, or may be a database server accessible from the terminal 600 .

Terminal 600 may also be a combination of multiple electronic devices. For example, different functional units in terminal 600 may be distributed and implemented in separate electronic devices.

Terminal 600 can also use a communication I/F (interface) for connecting to a network, an output device for outputting content, and the like. The communication I/F and/or display device may be built in terminal 600 or may be externally attached to terminal 600 .

The communication I/F can be a module for communicating with distribution server 500 (or distribution server 200) or another server (not shown) via a network. The output device may include a display device for displaying moving images, still images, text, etc., and/or a speaker for outputting audio, music, and the like. The display device is, for example, a liquid crystal display, an organic EL (electroluminescence) display, a CRT (Cathode Ray Tube) display, or the like. Note that the display device may have the function of an input device like a touch screen.

Terminal 600 includes receiving section 601 , effect condition determination section 604 , effect condition storage section 605 , effect data creation section 606 , effect data synthesizing section 607 , and display control section 608 .

The receiving unit 601 receives various data from, for example, the distribution server 500 via the network, and sends the received data to, for example, the effect condition determining unit 604 . The receiving unit 601 may be the aforementioned communication I/F.

Specifically, the receiving unit 601 receives video data for live distribution. This video data may be the video data itself transmitted (uploaded) from the distributor terminal 100 to the distribution server 500, or it may be processed in some way, for example, re-encoded. Also, the receiving unit 601 may receive the aforementioned designated data. This designated data may be collected from the distributor terminal 100 and/or the audience terminal 300 by the distribution server 500 and distributed to each terminal 600, for example. Furthermore, the receiving unit 601 can receive data necessary for determining the effect condition, for example, as described in the first embodiment.

Effect condition determination unit 604 may be the same as or similar to effect condition determination unit 504 (or effect condition determination unit 204). That is, the effect condition determination unit 604 receives video data (and specified data) from the reception unit 601, reads the effect conditions from the effect condition storage unit 605, and determines whether the effect conditions are satisfied. The effect condition determination unit 604 sends the video data and the designated data to the effect data synthesizing unit 607 if it determines that the effect condition is satisfied, and otherwise sends the video data to the display control unit 608 . The effect condition determination unit 604 may be the aforementioned processor and memory.

Effect data creation unit 606 may be the same as or similar to effect data creation unit 206 . That is, the effect data creation unit 606 creates effect data and sends it to the effect data synthesizing unit 607 . The effect data creation unit 206 may be the aforementioned processor and memory.

Effect data synthesizer 607 may be the same as or similar to effect data synthesizer 507 (or effect data synthesizer 207). That is, the effect data synthesizing unit 607 receives the video data and the specified data from the effect condition determining unit 604 and receives the effect data from the effect data creating unit 606 . The effect data synthesizing unit 607 pre-processes the video data and/or the effect data in accordance with the designated data, and then synthesizes both to generate synthesized video data. The effect data synthesizing unit 607 sends the synthetic video data to the display control unit 608 . The effect data synthesizing unit 607 may be the aforementioned processor and memory.

A display control unit 608 controls the above-described display device to display video data or composite video data on the screen of the display device. The display controller 608 may be, for example, the processor and memory described above.

As described above, the terminal according to this modification combines the video data and effect data described in the first and second embodiments. Therefore, according to this terminal, even if the distribution server cannot use such functions, it is possible to add the effects described in the first and second embodiments to the video data for live distribution. Become. In addition, in the example of FIG. 10, the terminal determines the effect condition, but it is also possible to further modify it so that the distribution server determines the effect condition.

(Third Embodiment)
In the above-described first and second embodiments, effects are realized by synthesizing effect data with video data. A distribution server according to the third embodiment realizes an effect by processing the video data itself for live distribution.

FIG. 11 illustrates a distribution server 700 according to the third embodiment. In the following description, differences between distribution server 700 and distribution server 200 or distribution server 500 will be mainly described. The distribution server 700 receives encoded video data sequentially transmitted from the distributor terminal 100 over the time frame assigned to the live distribution of the video data. The distribution server 700 processes the video data and distributes the processed video data to the distributor terminal 100 and the audience terminal 300 when the rendering conditions are satisfied. Here, the rendering conditions may be the same as or similar to the rendering conditions described in the first embodiment.

By processing the video data by the distribution server 700 in this manner, it is possible to use effects of expressions different from the effects described in the first and second embodiments. For example, there may be processing to apply a filter to a specific area (for example, a face area) or the entire area of video data. Filters can change the visual impression of video data by adding various effects such as mosaic, sepia, monochrome, retro, blur, night vision scope effect, distortion, and cartoon effect. can. Furthermore, there may be processing such as pasting a copy of a specific area of video data onto the video data.

The functional configuration of distribution server 700 will be described below with reference to FIG. Note that the hardware configuration of the distribution server 700 may be the same as or similar to that of the distribution server 200 or the distribution server 500, so the description thereof will be omitted.

Distribution server 700 includes a reception unit 701 , a clock unit 702 , a distribution control unit 703 , an effect condition determination unit 704 , an effect condition storage unit 705 , a video processing unit 707 and a video distribution unit 708 . Among them, the elements other than the image processing unit 707 may be the same as or similar to the same-named elements in the distribution server 200 or the distribution server 500, respectively. Therefore, the image processing unit 707 will be described below.

The video processing unit 707 receives video data (and designated data) from the rendering condition determination unit 704 . First, the image processing unit 707 performs the preprocessing described in the second embodiment on the image data when the designated data is received. Then, the image processing unit 707 performs predetermined processing on the image data from the rendering condition determination unit 704 or the image data preprocessed thereon to generate processed image data. The image processing unit 707 sends the processed image data to the image distribution unit 708 . The video processing unit 707 may be the processor and memory described above.

Specifically, the image processing unit 707 may apply any one of the filters described above to the image data. Alternatively, the image processing unit 707 may paste a copy of a specific area of the image data to the image data. For example, the image processing unit 707 may perform the processing illustrated in FIG. 13 or 14 on the image data as shown in FIG.

In the example of FIG. 13, an image obtained by mirror-reversing a copy of the area on the left side of the central axis indicated by the dashed line is pasted on the right side of the central axis. Note that the number of symmetrical axes, their respective positions, and their angles, which serve as a reference for inversion, may be automatically determined, or may be designated by the distributor and/or spectators, for example, through designated data.

In the example of FIG. 14, a copy of the area around the person in the video data itself, or an image obtained by preprocessing the copy, is pasted in the area indicated by the dashed line. It should be noted that which area in the video data is to be duplicated, whether preprocessing is to be performed on the duplication, what kind of preprocessing is to be performed on the duplication, the position and number of pasting, etc. are automatically determined. may be designated by the broadcaster and/or the audience, for example, through designation data.

As described above, the distribution server according to the third embodiment realizes effects by processing video data. Therefore, according to this distribution server, it is possible to use effects of expressions different from the effects described in the first and second embodiments.

(Modification)
By the way, in this embodiment, the distribution server processes the video data. This has the advantage of facilitating synchronization of the synthesized video data viewed by the distributor and the audience, and reducing the processing load on the terminal. It also has the advantage of reducing the amount of communication from the distribution server to the terminal. However, it is also possible for the terminal to process the video data. A brief description will be given below.

FIG. 15 illustrates a terminal 800 according to this modification. In addition, the hardware configuration of the terminal 800 may be the same as or similar to that of the terminal 600, so the description thereof will be omitted. Terminal 800 may be either a distributor terminal or a spectator terminal, and thus descriptions of parts specific to each of the distributor terminal and spectator terminal will be omitted.

Terminal 800 includes a receiving section 801 , an effect condition determination section 804 , an effect condition storage section 805 , an image processing section 807 and a display control section 808 . Among them, the elements other than the image processing unit 807 may be the same as or similar to the same-named elements in the terminal 600 . Therefore, the image processing unit 807 will be described below.

Image processing unit 807 may be the same as or similar to image processing unit 707 . That is, the video processing unit 807 receives video data (and specified data) from the rendering condition determination unit 804 . First, the image processing unit 807 performs the preprocessing described in the second embodiment on the image data when the designated data is received. Then, the image processing unit 807 performs the predetermined processing described in the third embodiment on the image data from the rendering condition determination unit 804 or the image data preprocessed thereon to obtain processed image data. to generate The video processing unit 807 sends the processed video data to the video distribution unit 708 . The video processing unit 807 may be the processor and memory described above.

As described above, the terminal according to this modified example processes the video data described in the third embodiment. Therefore, according to this terminal, even if the distribution server cannot use such functions, it is possible to add the effects described in the third embodiment to the video data for live distribution. In addition, in the example of FIG. 15, the terminal determines the effect condition, but it is also possible to further modify so that the distribution server determines the effect condition.

(Fourth embodiment)
In the above-described first embodiment, it is possible to produce an effect by synthesizing dynamic effect data with video data. In the fourth embodiment, a highly interactive presentation is realized by controlling this dynamic presentation data in accordance with operation requests from the distributor and/or the audience.

FIG. 16 illustrates a distribution server 900 according to the fourth embodiment. In the following description, differences between distribution server 900 and distribution server 200, distribution server 500, or distribution server 700 will be mainly described.

The distribution server 900 receives the encoded video data sequentially transmitted from the distributor terminal 100 over the time frame assigned to the live distribution of the video data. When the rendering conditions are satisfied, the distribution server 900 receives from the distributor terminal 100 and/or the audience terminal 300 an operation request for dynamic rendering data to be combined with the video data, and based on this request, performs the dynamic rendering. to generate operation data of such effect data. The distribution server 900 then transmits this operation data to the distributor terminal 100 and the audience terminal 300 in addition to the video data. The distributor terminal 100 and the spectator terminal 300 control the dynamic effect data based on the received operation data, combine it with the received video data, and display it on the screen of the display device. Here, the rendering conditions may be the same as or similar to the rendering conditions described in the first embodiment.

The distribution server 900 generates and distributes the operation data of such dynamic effect data based on the operation request, and the distributor terminal 100 and the spectator terminal 300 control the dynamic effect data based on the operation data. By synthesizing with video data, it is possible to produce highly interactive performances.

The functional configuration of distribution server 900 will be described below with reference to FIG. The hardware configuration of the distribution server 900 may be the same as or similar to that of the distribution server 200, the distribution server 500, or the distribution server 700, so the description thereof will be omitted.

Distribution server 900 includes reception unit 901 , clock unit 902 , distribution control unit 903 , effect condition determination unit 904 , effect condition storage unit 905 , video distribution unit 908 , and operation data generation unit 909 . Of these, the clock unit 902, the distribution control unit 903, and the effect condition storage unit 905 may be the same as or similar to the same-named elements in the distribution server 200, the distribution server 500, or the distribution server 700, respectively. In the following, other elements will be described, focusing on differences from elements with the same name in distribution server 200, distribution server 500, or distribution server 700. FIG.

In addition to the data received by the receiving unit 201 , the receiving unit 901 can receive operation requests for dynamic presentation data from the distributor terminal 100 and/or the audience terminal 300 . This operation request is, for example, a drawing application, a handwriting input application, a stopwatch, a timer, a counter, a dice, a playing card, an omikuji, a bingo game, a rock-paper-scissors game, a still image viewer, etc. may correspond to input data, such as object selection, etc., provided to the corresponding program to operate a player of the content quoted during the live broadcast, including; The receiving unit 901 sends this operation request to the effect condition determining unit 904 .

The effect condition determination unit 904 sends the video data to the video distribution unit 908 regardless of the determination result of the effect condition. Furthermore, the effect condition determination unit 904 sends an operation request to the operation data generation unit 909 when the effect condition is satisfied. It should be noted that if the distributor and the audience have not made an input for issuing an operation request, there may be no operation request.

An operation data generation unit 909 receives an operation request from the effect condition determination unit 904, and based on this, generates operation data of dynamic effect data. The operation data generator 909 sends the operation data to the video distribution unit 908 . The operation data generator 909 may be the aforementioned processor and memory. Note that if a plurality of terminals transmit operation requests substantially at the same time, collision of the operation requests may occur. That is, although an operation request from one terminal may conflict with an operation request from another terminal, the operation data generation unit 909 may generate operation data after arbitrating such conflicts.

In addition, it is a heavy load for the operation data generation unit 909 to generate operation data considering all the operation requests by transmitting operation requests from a large number of terminals at substantially the same time. Therefore, in relation to the conflict mediation, the operation data generation unit 909 may perform filtering to discard a part of the operation request, and generate operation data based on the remaining operation request. As an example, the operation data generation unit 909 maintains operation requests from spectators who have paid a fee, members of the distributor's channel, people in a specific area, people who have won a lottery, and others. operation request may be discarded.

In the present embodiment, the dynamic effect data can be video data that causes screen transition, that is, appearance change, at least according to the operation data. Specifically, as an example of screen transitions according to operation data, updating the hand image according to operation data for selecting a deck of playing cards, updating the number of dice according to operation data for selecting a dice, For example, updating a drawing image according to operation data based on handwriting input. In addition to operation data, there may be screen transitions according to the passage of time. For example, depending on the operation data for selecting the start button, the image on the display of the stopwatch or timer starts changing to count down or count up. The image on the display unit thereafter is updated as time passes.

Screen transitions may or may not be common to all terminals. As an example of the latter, the dynamic effect data may be execution screen data of a rock-paper-scissors game illustrated in FIGS. 17 and 18 . In this example, as shown in FIG. 17, an image representing a selectable hand is initially synthesized with the video data, and the operation data generation unit 909 receives operation requests from the distributor and/or the audience until a predetermined deadline. Stand by and generate operation data after the deadline. After the deadline is over, a screen transition occurs to an image showing the result of rock-paper-scissors. As a result, the distributor terminal 100, as illustrated in FIG. can be generated and synthesized with video data. In addition, the spectator terminal 300 generates performance data representing, for example, the hand selected by the distributor and the hand selected by the spectator who operated the spectator terminal 300 as a result of the game with the distributor, and synthesizes it with the video data. be able to. Alternatively, the distributor terminal 100 and/or the spectator terminal 300 may generate effect data representing the sum of the winning/losing results such as the winner's name, the loser's name, and the winner's name, and synthesize them with the video data.

The video distribution unit 908 receives video data from the rendering condition determination unit 904 and may also receive operation data from the operation data generation unit 909 . The video distribution unit 908 distributes video data (and operation data) to the distributor terminal 100 and the audience terminal 300 .

FIG. 19 illustrates a terminal 1000 according to the fourth embodiment. In the following description, differences from terminal 600 or terminal 800 will be mainly described. In addition, the hardware configuration of the terminal 1000 may be the same as or similar to that of the terminal 600 or the terminal 800, so the description thereof will be omitted. Also, the terminal 1000 may be either a distributor terminal or an audience terminal, so descriptions of the parts unique to the distributor terminal and the audience terminal are omitted.

The terminal 1000 includes a reception unit 1001, an effect condition determination unit 1004, an effect condition storage unit 1005, an effect data control unit 1006, an effect data synthesizing unit 1007, a display control unit 1008, an input unit 1010, an operation request A generator 1011 and a transmitter 1012 are included. Of these, effect condition storage unit 1005, effect data synthesizing unit 1007, and display control unit 1008 may be the same as or similar to the same-named elements in terminal 600 or terminal 800, respectively. In the following, other elements will be described, focusing on differences from the same-named elements in terminal 600 or terminal 800. FIG.

The input unit 1010 receives user input and generates user input data. The input unit 1010 sends user input data to the effect condition determination unit 1004 . The input unit 1010 may be an input device built into the terminal 1000 or externally attached. The input device may be, for example, a keyboard, mouse, numeric keypad, or the like, or may have the function of a display device such as a touch screen. User inputs are typically taps, clicks, specific key presses, etc., and user input data can be data representing the occurrence of these events. In addition, user input data includes, for example, voice captured by a microphone, biometric data (e.g. body temperature, facial expression, etc.) detected by a biosensor, position data identified by GPS (Global Positioning System) or base station information, acceleration A user's action (for example, swinging the terminal 1000) estimated based on acceleration data detected by a sensor can also be included.

The effect condition determination unit 1004 sends the user input data to the operation request generation unit 1011 when the effect condition is satisfied.

The operation request generator 1011 receives user input data from the effect condition determination unit 1004 and generates an operation request accordingly. The operation request generator 1011 sends this operation request to the transmitter 1012 . The operation request generator 1011 may be the processor and memory described above.

The transmission unit 1012 receives the operation request from the operation request generation unit 1011 and transmits it to the distribution server 900 . The transmission unit 1012 may be the aforementioned communication I/F.

The receiving unit 1001 receives video data for live distribution. The video data may be the video data itself transmitted (uploaded) from the distributor terminal 100 to the distribution server 900, or may be processed in some way. Also, the receiving unit 1001 can receive the operation data when the effect condition is satisfied. Furthermore, the receiving unit 1001 may receive the aforementioned specified data, data necessary for determining the rendering conditions, and the like. Receiving section 1001 sends operation data to effect data control section 1006 , sends video data and specified data to effect data synthesizing section 1007 , and sends data necessary for determining effect conditions to effect condition determining section 1004 .

The effect data control unit 1006 receives operation data from the receiving unit 1001 and controls dynamic effect data based on the operation data. Specifically, the effect data control unit 1006 executes programs (applications) corresponding to the various tools described above, and provides operation data to the programs. The operation data may include time data and operation input data corresponding to the time data. In this case, the effect data control section 1006 can provide the program with operation input data corresponding to the time data at the time indicated by the time data. The program can generate, as an execution result, a video that transitions between screens according to given operation input data, that is, as dynamic effect data. Effect data control section 1006 sends dynamic effect data to effect data synthesizing section 1007 . The effect data control unit 1006 may be the aforementioned processor and memory. Here, the program may be installed in a player for reproducing video data for live distribution, or may be an external program from the player's point of view.

As described above, the distribution server according to the fourth embodiment collects operation requests for dynamic performance data from distributor terminals and/or audience terminals, generates operation data for the dynamic performance data, To deliver. Then, the terminal according to the present embodiment controls dynamic effect data based on this operation data, and synthesizes it with video data for live distribution. Therefore, according to this distribution server and terminal, it is possible to produce a highly interactive performance, for example, in which the distributor and the audience play together in real time.

(Modification)
By the way, in this embodiment, the terminal controls the dynamic effect data and synthesizes the video data and the dynamic effect data. However, it is also possible for the distribution server to control the dynamic effect data and sequentially transmit the effect data to be synthesized with the video data to the terminal. It is also possible to perform even synthesis with typical performance data. This has the advantage of facilitating synchronization of the synthesized video data viewed by the distributor and the audience, and reducing the processing load on the terminal. An example in which the distribution server controls dynamic effect data and synthesizes video data and dynamic effect data will be briefly described.

FIG. 20 illustrates a distribution server 1100 according to this modification. The hardware configuration of distribution server 1100 may be the same as or similar to that of distribution server 200, distribution server 500, distribution server 700, or distribution server 900, and thus description thereof will be omitted.

The distribution server 1100 includes a reception unit 1101, a clock unit 1102, a distribution control unit 1103, an effect condition determination unit 1104, an effect condition storage unit 1105, an effect data control unit 1106, an effect data synthesizing unit 1107, an image It includes a distribution unit 1108 and an operation data generation unit 1109 . Of these, the reception unit 1101, the clock unit 1102, the distribution control unit 1103, the effect condition determination unit 1104, the effect condition storage unit 1105, the effect data synthesizing unit 1107, the video distribution unit 1108, and the operation data generation unit 1109 are included in the distribution server 200. , distribution server 500 , distribution server 700 or distribution server 900 . Therefore, the effect data control unit 1106 will be described below.

The performance data controller 1106 may be the same as or similar to the performance data controller 1006 . That is, the effect data control unit 1106 receives operation data from the operation data generation unit 1109, and controls dynamic effect data based on the operation data. Effect data control section 1106 sends dynamic effect data to effect data synthesizing section 1107 . The effect data control unit 1106 may be the aforementioned processor and memory.

As described above, the distribution server according to this modification performs the effects described in the fourth embodiment, and distributes the synthesized video data to the distributor terminal and the audience terminal. Therefore, according to this distribution server, it is possible to easily synchronize synthesized video data viewed by the distributor and the audience, and to reduce the processing load of the terminal. It also has the advantage of reducing the amount of communication from the distribution server to the terminal.

In this embodiment, both the terminal and the distribution server determine the effect condition, but only one of them may perform the determination. For example, only the terminal may determine the effect condition and notify the distribution server that the effect condition is satisfied. Conversely, only the distribution server may determine the effect condition and notify the terminal that the effect condition is satisfied.

The above-described embodiments merely show specific examples to aid understanding of the concept of the present invention, and are not intended to limit the scope of the present invention. Embodiments can add, delete, or convert various components without departing from the gist of the present invention.

Various functional units described in the above embodiments may be realized by using circuits. A circuit may be a dedicated circuit that implements a specific function, or it may be a general-purpose circuit such as a processor.

At least part of the processing in each of the above embodiments can also be implemented using a general-purpose computer as basic hardware. A program that implements the above process may be provided by being stored in a computer-readable recording medium. The program is stored in the recording medium as an installable format file or an executable format file. Recording media include magnetic disks, optical disks (CD-ROM, CD-R, DVD, etc.), magneto-optical disks (MO, etc.), semiconductor memories, and the like. Any recording medium may be used as long as it can store the program and is readable by a computer. Alternatively, the program that implements the above processing may be stored on a computer (server) connected to a network such as the Internet, and downloaded to the computer (client) via the network.
The invention described in the scope of claims at the time of filing of the present application will be additionally described below.
[C1] a receiving unit that receives video data for live distribution from a distributor terminal;
a determination unit that determines whether or not the rendering conditions are satisfied;
a synthesizing unit for synthesizing performance data with the video data to generate synthetic video data when the performance condition is satisfied;
a distribution unit that distributes the composite video data to the distributor terminal and the spectator terminal;
A distribution server comprising:
[C2] The effect conditions are as follows: (1) The first effect ends after the date and time referred to by the distribution server reaches the effect start point set within the time frame allocated for the live distribution of the video data. (2) after the number of spectators of the video data satisfies a predetermined condition until the second performance end condition is satisfied; (4) from the time the comment posted to the video data satisfies a predetermined condition until the fourth effect end condition is satisfied; and (5) The distribution server according to C1, which satisfies at least one of a period from when the reproduced video or reproduced audio of the video data satisfies a predetermined condition to when the fifth rendering end condition is satisfied.
[C3] Any one of the first effect end condition, the second effect end condition, the third effect end condition, the fourth effect end condition and the fifth effect end condition is satisfied. The distribution server according to C2, further comprising a creation unit that creates the effect data including information dynamically indicating a necessary condition.
[C4] The distribution server according to C3, wherein the creation unit creates the effect data including information indicating a gap between the number of spectators or the number of comments in the video data and the required number.
[C5] The distribution server according to C1 or C2, further comprising a creation unit that creates the effect data including information about an event that occurred after the start of live distribution of the video data.
[C6] The creating unit includes the number of spectators of the video data, part of comments posted on the video data, the number of comments posted on the video data, and comments on the video data. The distribution server according to C5, which creates the effect data including information indicating at least one of the posted spectator names.
[C7] Reacting to at least one of (1) a user input made to at least one of the distributor terminal and the spectator terminal and (2) a comment posted by at least one of the distributor terminal and the spectator terminal The distribution server according to C1 or C2, further comprising a creation unit that creates performance data that changes by
[C8] Further comprising a creation unit that creates the effect data including information generated based on at least one of metadata attached to the video data and information displayed on a web page related to the video data. , C1 or C2.
[C9] The synthesizing unit according to any one of C1 to C7, wherein a part or all of a screen of the reproduced video of the video data is resized and synthesized with the effect data to generate the synthesized video data. distribution server.
[C10] The synthesizing unit synthesizes the video data with the effect data such that a part of the screen of the reproduced video of the video data is exposed and the remaining part is hidden, thereby generating the synthesized video data, C1 to C7. The distribution server according to any one of Claims 1 to 3.
[C11] computer,
means for receiving video data for live distribution from a distributor terminal;
Means for determining whether or not performance conditions are satisfied;
means for synthesizing performance data with the video data to generate synthetic video data when the performance condition is satisfied;
Means for distributing the composite video data to the distributor terminal and the spectator terminal
A distribution program for functioning as a
[C12] The receiving unit receives from at least one of the distributor terminal and the spectator terminal a combination position of the video data and the effect data, an enlargement/reduction ratio of the video data, an enlargement/reduction ratio of the effect data, further receiving designation data designating at least one of the amount of rotation of the image data, the amount of rotation of the effect data, a specific area in the image data, and a specific area in the effect data;
The synthesizing unit performs at least one of translation, resizing, rotation, inversion, and trimming on at least one of the video data and the effect data according to the specified data, and then synthesizes the video data and the effect data. , generating said composite video data;
The distribution server according to any one of C1 to C10.
[C13] a receiving unit that receives video data for live distribution from a distributor terminal;
a determination unit that determines whether or not the rendering conditions are satisfied;
a processing unit that performs predetermined processing on the video data to generate processed video data when the performance condition is satisfied;
a distribution unit that distributes the processed video data to the distributor terminal and the spectator terminal;
A distribution server comprising:
[C14] When the effect condition is satisfied, the processing unit duplicates the specific area in the video data as it is, or after performing at least one of rotation, inversion and resizing, pastes it into the video data. The distribution server according to C13, which generates the processed video data.
[C15] a determination unit that determines whether or not the rendering conditions are satisfied;
Video data for live distribution is received from a distributor terminal, and when the performance condition is satisfied, at least one of the distributor terminal and the spectator terminal is further sent to at least one of the distributor terminal and the spectator terminal. a receiving unit for receiving an operation request for dynamic effect data synthesized with the video data in
a generation unit that generates operation data of the dynamic effect data based on the operation request;
a distribution unit that distributes the video data and the operation data to at least one of the distributor terminal and the audience terminal;
A distribution server comprising:
[C16] The distribution server according to C15, wherein the dynamic effect data is video data that causes screen transitions in accordance with the operation data.
[C17] a determination unit that determines whether or not the rendering conditions are satisfied;
a generation unit that generates an operation request for dynamic performance data according to a user input when the performance condition is satisfied;
a transmission unit that transmits the operation request to a distribution server;
a receiving unit that receives, from the distribution server, video data for live distribution and operation data of the dynamic effect data generated based on the operation request when the effect condition is satisfied;
a control unit that controls the dynamic effect data based on the operation request;
a synthesizing unit that synthesizes the dynamic presentation data with the video data;
terminal.

100 Distributor terminal 200,500,700,900,1100 Distribution server 201,501,601,701,801,901,1001,1101 Reception unit 202,502,702,902,1102 Clock unit 203, 503, 703, 903, 1103 Distribution control unit 204, 504, 604, 704, 804, 904, 1004, 1104 Effect condition determination unit 205, 505, 605, 705, 805, 905, 1005, 1105... Effect condition storage unit 206, 506, 606... Effect data creation unit 207, 507, 607, 1007, 1107... Effect data synthesizing unit 208, 508, 708, 908, 1108 Video distribution unit 300 Audience terminal 600, 800, 1000 Terminal 608, 808, 1008 Display control unit 707, 807 Video processing unit 909, 1109 Operation data Generating units 1006, 1106 Effect data control unit 1010 Input unit 1011 Operation request generating unit 1012 Transmitting unit

Claims (12)

Receiving at least video data for live distribution from a distributor terminal;
Determining whether the production conditions are satisfied;
adding an effect to the video data for live distribution when the effect condition is satisfied;
A computer program for causing a computer to distribute the video data to which the effect has been added to the distributor terminal or the spectator terminal,
The effect condition is a criterion for determining whether or not to add an opening effect, an ending effect, or an intermediate effect, and the effect condition is at least a time frame allocated for live distribution of the video data. , the number of spectators of the video data, an instruction to start the opening effect, a comment posted on the video data, and a condition based on the playback sound of the video data,
The determining includes determining whether or not the performance conditions are satisfied for each of the opening effect, the ending effect, or the intermediate effect.
computer program. Receiving at least video data for live distribution from a distributor terminal;
Determining whether the production conditions are satisfied;
adding an effect to the video data for live distribution when the effect condition is satisfied;
distributing the video data to which the effect is added to the distributor terminal or the spectator terminal;
A computer program that causes a computer to execute
The effect condition is a criterion for determining whether or not to add an opening effect, an ending effect, or an intermediate effect. is determined to satisfy between
The determining includes determining whether or not the performance conditions are satisfied for each of the opening effect, the ending effect, or the intermediate effect.
computer program. 3. The computer program according to claim 1, wherein adding an effect to said video data for live distribution includes synthesizing said effect with said video data to generate synthesized video data. The receiving includes receiving designation data that designates how to synthesize the video data for live distribution and the presentation;
3. The computer according to claim 1 , wherein adding an effect to said video data for live distribution includes synthesizing said video data or said effect according to said designated data to generate synthesized video data. program. 3. The computer program according to claim 1, wherein adding an effect to said video data for live distribution includes performing predetermined processing on said video data to generate processed video data. . Adding an effect to the video data for live distribution includes generating operation data for a dynamic effect based on an operation request from at least one of the distributor terminal or the spectator terminal,
3. The computer program product according to claim 1, wherein said distributing includes distributing said video data and said operation data to at least one of said distributor terminal or spectator terminal. further causing a computer to perform creating the rendition;
7. A computer program according to any one of claims 1-6 . 8. The computer program product of claim 7 , wherein said creating includes creating said presentation including information about events that occurred after live distribution of said video data began. a receiving unit that receives at least video data for live distribution from a distributor terminal;
a determination unit that determines whether or not the rendering conditions are satisfied;
a distribution unit that distributes data added with effects to the video data for live distribution to the distributor terminal or the spectator terminal when the effect condition is satisfied;
The effect condition is a criterion for determining whether or not to add an opening effect, an ending effect, or an intermediate effect, and the effect condition is at least a time frame allocated for live distribution of the video data. , the number of spectators of the video data, an instruction to start the opening effect, a comment posted on the video data, and a condition based on the playback sound of the video data,
The determination unit determines whether or not each of the opening effect, the ending effect, or the intermediate effect satisfies the effect condition.
distribution server. a receiving unit that receives at least video data for live distribution from a distributor terminal;
a determination unit that determines whether or not the rendering conditions are satisfied;
a distribution unit for distributing, when the performance condition is satisfied, data added with performance to the video data for live distribution to the distributor terminal or the spectator terminal;
and
The effect condition is a criterion for determining whether or not to add an opening effect, an ending effect, or an intermediate effect. is determined to satisfy between
The determination unit determines whether or not each of the opening effect, the ending effect, or the intermediate effect satisfies the effect condition.
distribution server. Receiving at least video data for live distribution from a server;
obtaining a determination as to whether or not the rendering conditions are satisfied;
adding an effect to the video data for live distribution when the effect condition is satisfied;
displaying the video data to which the effect has been added on a screen of a display device;
A computer program that causes a computer to execute
The effect condition is a criterion for determining whether or not to add an opening effect, an ending effect, or an intermediate effect, and the effect condition is at least a time frame allocated for live distribution of the video data. , the number of spectators of the video data, an instruction to start the opening effect, a comment posted on the video data, and a condition based on the playback sound of the video data,
The determining includes determining whether or not the performance conditions are satisfied for each of the opening effect, the ending effect, or the intermediate effect.
computer program. Receiving at least video data for live distribution from a server;
obtaining a determination as to whether or not the rendering conditions are satisfied;
adding an effect to the video data for live distribution when the effect condition is satisfied;
displaying the video data to which the effect has been added on a screen of a display device;
A computer program that causes a computer to execute
The effect condition is a criterion for determining whether or not to add an opening effect, an ending effect, or an intermediate effect. is determined to satisfy between
The determining includes determining whether or not the performance conditions are satisfied for each of the opening effect, the ending effect, or the intermediate effect.
computer program.

Images