KR102567931B1 - Contents generation flatfrom device undating interactive scenario based on viewer reaction - Google Patents

Abstract

A content generation platform device according to the technical idea of the present disclosure comprises: a viewer reaction collection unit which receives voice reactions and text reactions on virtual content from viewer devices receiving the virtual content and based on the voice reactions and text reactions, produces real-time emotion kinds of viewers watching the virtual content and a real-time emotion figure on the real-time emotion kinds; a storyline generation unit which generates an interactive scenario of the virtual content and updates the interactive scenario based on the real-time emotion kinds and the real-time emotion figure; and a virtual human generation unit which generates a virtual human of the virtual content and adjusts a facial expression of the virtual human based on the real-time emotion kinds and the real-time emotion figure. Accordingly, the content generation platform device updates the scenario in consideration of viewer reactions and the facial expressions of the virtual human to provide the interactive content.

Description

Content generation platform device for updating interactive scenarios based on viewer reactions

The technical idea of the present disclosure relates to a content creation platform device that provides virtual content, and more particularly, to a content creation platform device that updates a scenario based on a viewer's response.

Recently, as movement between countries and gatherings of people are restricted due to corona, interest in non-face-to-face technology based on virtual reality is growing. In accordance with this trend, attempts to introduce virtual production technology are increasing in the field of video production, such as broadcasting, movies, and performances.

Virtual production is a video shooting technology that enables instant visualization and video synthesis based on real-time graphics rendering and camera tracking technology, and is a technology that can improve the efficiency and quality of the video production process. am. For example, virtual production can be used as a concept that visualizes graphics reflecting camera movements in various forms at the filming site and synthesizes them with real-time images of cameras.

These virtual productions are mainly made indoors to reduce restrictions such as shooting space, time, and weather, and since complex elements can be filmed simultaneously, it can reduce the post-production time of video production. It has the advantage of improving production productivity.

The technical idea of the present disclosure may provide a content creation platform device that provides interactive content by updating a scenario and updating a facial expression of a virtual human in consideration of a viewer's reaction.

In order to achieve the above object, a content creation platform device according to an aspect of the present disclosure receives a voice response and a text response for virtual content from viewer devices that receive virtual content, and receives a voice response and a text response. a viewer response collection unit that calculates real-time emotion types and real-time emotion values for real-time emotion types of viewers who watch virtual content based on the basis, generates interactive scenarios of virtual content, and creates interactive scenarios based on real-time emotion types and real-time emotion values and a storyline generating unit that updates the virtual human, and a virtual human generating unit that creates a virtual human of virtual content and adjusts the facial expression of the virtual human based on the real-time emotion type and real-time emotion value.

Furthermore, the viewer response collection unit converts the voice response into text, extracts a plurality of predetermined core texts from the converted text, and determines real-time emotion types based on the frequency of each of the plurality of core texts mapped to the emotion types. Decide.

Furthermore, the viewer response collection unit calculates a real-time emotion value based on the real-time emotion type and the frequency of the mapped first core text, from the text response and the converted text.

Furthermore, the viewer response collection unit calculates a real-time emotional value by giving a weight to the first core text extracted from the converted text more than the first core text extracted from the text response.

Furthermore, the storyline generating unit, when a current story among a plurality of stories included in the interactive scenario is completed, compares a target emotional value of the current story with a real-time emotional value, and the viewer device selects the next story as the next story based on the comparison result. update the list of story candidates.

Furthermore, the storyline generating unit, when a current sub-story among a plurality of sub-stories included in each of a plurality of stories is completed, compares a target emotional value of the current story with a real-time emotional value, and based on the comparison result, the next sub-story. update the story

Furthermore, the virtual human generator primarily renders the expression of the virtual human based on the text of the interactive scenario and secondarily corrects the expression of the virtual human based on the real-time emotion value.

According to an embodiment of the present disclosure, a content creation platform device may provide an apparatus for providing interactive content by updating a scenario and updating a facial expression of a virtual human in consideration of a viewer's reaction.

1 is a diagram illustrating a content service platform system 1 according to an exemplary embodiment of the present disclosure.
2 is a block diagram illustrating a content creation platform 10 according to an exemplary embodiment of the present disclosure.
3 is a flowchart illustrating the operation of a reaction analyzer according to an exemplary embodiment of the present disclosure.
4 is a flowchart illustrating an operation of a storyline generator according to an exemplary embodiment of the present disclosure.
5 is a diagram illustrating sub-story updating according to an exemplary embodiment of the present disclosure.
6 is a flowchart illustrating an operation of the storyline generating unit 200 according to an exemplary embodiment of the present disclosure.
7 is a diagram for explaining a method of selecting a subsequent story candidate group according to an exemplary embodiment of the present disclosure.
8 is a flowchart illustrating an operation of a virtual human generator according to an exemplary embodiment of the present disclosure.

Hereinafter, a preferred embodiment of the present invention will be described with reference to the accompanying drawings. It should be noted that the same reference numerals and symbols refer to the same elements in the drawings as much as possible, even if they are displayed on different drawings. In the following description of the present invention, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted.

In addition, when a certain component is said to "include", this means that it may further include other components without excluding other components unless otherwise stated.

Terms used in this specification are for describing the embodiments and are not intended to limit the present invention. In this specification, singular forms also include plural forms in some cases unless otherwise specified in the text. In this specification, terms such as "comprise", "have", "provide" or "have" do not exclude the presence or addition of one or more other elements other than the mentioned elements.

In this specification, terms such as “or” and “at least one” may represent one of the words listed together, or a combination of two or more. For example, "or B" and "at least one of B" may include only one of A or B, or may include both A and B.

In this specification, descriptions following "for example" may not exactly match the information presented, such as cited characteristics, variables, or values, and tolerances, measurement errors, limits of measurement accuracy and other commonly known factors It should not be limited to the embodiments of the present invention according to various embodiments of the present invention with effects such as modifications including.

In this specification, terms such as 'first' and 'second' may be used to describe various elements, but the elements should not be limited by the above terms. In addition, the above terms should not be interpreted as limiting the order of each component, and may be used for the purpose of distinguishing one component from another. For example, a 'first element' may be named a 'second element', and similarly, a 'second element' may also be named a 'first element'.

Unless otherwise defined, all terms used in this specification may be used in a meaning commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly specifically defined.

Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings.

1 is a diagram illustrating a content service platform system 1 according to an exemplary embodiment of the present disclosure.

Referring to FIG. 1 , a content service platform system 1 may be a system that provides multimedia content such as performances, movies, and dramas (hereinafter referred to as movies) to viewers 40 . The content service platform system 1 may include a content creation platform 10 and a content delivery platform 20 . The content creation platform 10, the content transmission platform 20, and the viewer 30 may mutually transmit/receive data through a network.

The content creation platform 10 may be a content creation device provided in a server of a virtual content producer. The content generation platform 10 may create a virtual human that tracks the action of the actor through a plurality of 2D cameras and/or 3D cameras that capture the action of the actor in a 3D space. The content creation platform 10 may create interactive scenarios and create virtual humans that track voices and appearances of actors performing according to the interactive scenarios.

The content creation platform 10 may provide the viewer 30 with a sense of immersion in the virtual content by modifying the interactive scenario according to the viewer's 30 response. In some embodiments, the content generation platform 10 may provide the viewer 30 with a sense of immersion in the virtual content by dynamically modifying the expression of the virtual human according to the reaction of the viewer 30 .

For example, the viewer 30 may provide the content creation platform 10 with a real-time response to the virtual content through text or voice. The content creation platform 10 may quantify the type and degree of emotion felt by the viewer 30 after viewing the virtual content, based on the viewer's 30 real-time response. The content creation platform 10 may assign a target emotion value to each of the stories included in the scenario, and update a subsequent sub-story by comparing the viewer's real-time emotion value and the target emotion value of the story whenever a sub-story ends. can do.

For example, the target emotion type of the first story is 'sadness', the target emotion value is '10', the viewer's real-time emotion type is 'sadness' after the first sub-story of the first story ends, and the real-time emotion type of the first story is 'sadness'. When the emotional value is '5', the subsequent sub-story may be updated to a sub-story having a relatively high emotional value.

When the reaction expected by the content creation platform 10 (ie, the target emotional value) is not provided from the viewer 30, the content creation platform 10 may change the scenario, so the content creation platform 10 30), a new experience can be provided to the viewer 30 each time the virtual content is viewed.

In some embodiments, when the current story included in the scenario ends, the content creation platform 10 may provide the viewer 30 with the right to decide on the next story. Specifically, the content creation platform 10 provides the viewer 30 with the right to decide on the next story by providing actions that the character can take next as options to develop into the next story according to the context of the current story when the current story ends. can provide.

According to an exemplary embodiment of the present disclosure, the content creation platform 10 may determine candidate stories that may become the next story according to the real-time reaction of the viewer 30 . Specifically, when the current story ends, the content generation platform 10 may collect real-time reactions of the viewer and compare the target emotional value of the current story with the real-time emotional value. If the real-time emotion value is lower than the target emotion value, since a more dramatic story needs to be provided to the corresponding viewer, the content creation platform 10 may classify candidate stories having a relatively high target emotion value.

The content transmission platform 20 may be a performance video transmission device provided in a server of a virtual performance distributor. The content delivery platform 20 may provide virtual content to the viewer 30 through an Over The Top (OTT) service, theater, movie theater, or TV.

The viewer 30 may view virtual content using a plurality of clients (eg, client 1 and client 2), and each of the plurality of clients may be various types of video reproducing devices. For example, each of the plurality of clients may include at least one of an augmented reality (AR) device, a virtual reality (VR) device, an extended reality (XR) device, a media playback device, a personal computer (PC), a tablet, and a mobile device. can be

2 is a block diagram illustrating a content creation platform 10 according to an exemplary embodiment of the present disclosure. FIG. 2 may be described with reference to FIG. 1 .

Referring to FIG. 2 , the content creation platform 10 may include a viewer response collection unit 100 , a storyline generation unit 200 and a virtual human generation unit 300 .

The viewer response collection unit 100 may include an STT module 110 , a core text extraction unit 120 and a response analysis unit 130 . The viewer response collection unit 100 may receive a text response or a voice response from the viewer 30 . The viewer 30 may mean a plurality of viewers. The viewer reaction collection unit 100 may analyze real-time reactions of a plurality of viewers and calculate emotion types and real-time emotion values of the plurality of viewers.

The speech-to-text (STT) module 110 may convert the viewer's 30 voice response into text.

The core text extractor 120 may extract core text representing emotions of the viewer 30 from text converted from a voice response or text received from the viewer 30 . For example, a character string representing laughter, a character string representing sadness, and a specific word may be extracted.

The reaction analyzer 130 may analyze the reaction of the viewer 30 based on the extracted text. Specifically, in the voice response and the text response, the emotional type and real-time emotional value of the viewer 30 may be calculated by calculating the frequency of core text. For example, since a voice response can be understood as an actual response of the viewer 30 rather than a text response, the response analyzer 130 can determine the emotional type of the viewer 30 based on the core text extracted from the voice response. there is. In addition, since the frequency of text responses is higher than that of voice responses, the frequency of the core text can be calculated with more weight on voice responses than on text responses. The response analyzer 130 may calculate a real-time emotional value based on the frequency of core text. For example, the reaction analyzer 130 may calculate a real-time emotional value by dividing the frequency of core text by a reference value.

The storyline generator 200 may include an interactive scenario generator 210 and a scenario update module 220 . The storyline generator 200 may create a scenario that varies according to a viewer's reaction.

The interactive scenario generator 210 may initially generate an interactive scenario including a plurality of stories. A target emotion and a target emotion value may be mapped to each of the plurality of stories. For example, in the first story, a target emotion of 'impression' may be mapped, and a target emotion value corresponding to 'impression' may be mapped as '10'. That is, by mapping the target emotion type and the target emotion value, when the first story ends, the real-time emotion type according to the viewer's reaction becomes 'impression' and the real-time emotion value becomes '10' in the first story. The contents of the first story may be continuously updated while the .

The scenario update module 220 may update the initial interactive scenario according to viewer reactions.

A story may include a plurality of sub-stories, and the scenario update module 220 may update subsequent sub-stories in consideration of a viewer's reaction whenever the plurality of sub-stories are finished. Specifically, the scenario update module 220 compares the target emotion type and target emotion value of the story with the real-time emotion type and real-time emotion value whenever a plurality of sub-stories are completed, and determines a subsequent sub-story based on the comparison result. can For example, when the target emotion type of the story is 'impression', the target emotion value is '10', the real-time emotion type is 'impression', and the target emotion value is '5', the scenario update module 220 performs the existing A sub-story with a higher emotional value than the subsequent sub-story set in can be updated as a new subsequent sub-story. A specific method may be described below based on FIGS. 4 and 5 .

When a current story among a plurality of stories included in a scenario ends, the scenario update module 220 may determine candidate stories to be subsequent stories in consideration of a viewer's reaction. Specifically, the scenario update module 220 may compare the target emotion type and target emotion value of the current story with the real-time emotion type and real-time emotion value, respectively, and select candidate stories according to the comparison result. For example, when the target emotional value of the current story is '10' and the real-time emotional value is '5', the scenario update module 220 may determine that the viewer 30 does not watch emotionally more than expected; Candidate stories more emotional than the current story may be determined as subsequent stories. That is, the target emotion value of the candidate stories may be higher than '10', which is the target emotion value of the current story.

The scenario update module 220 may provide a decision authority so that the viewer 40 may select one of the candidate stories. Specifically, the scenario update module 220 may provide the viewer 30 with the right to decide on the next story by providing actions that the character may take next as options to develop into the next story according to the context of the current story. there is. Actions that the character can take next and candidate stories may correspond to each other. A method of selecting a candidate story may be described later with reference to FIGS. 6 and 7 .

The virtual human generator 300 may include a motion capture module 310 , a synthesis blending module 320 and a TTS module 330 .

The motion capture module 310 may capture the action of an actor in a 3D space based on a plurality of 2D cameras and/or 3D cameras, and generate the appearance of a virtual human that simulates the action of the actor based on the captured images. can

The synthesis blending module 320 may synthesize or animate a virtual human face into a desired actor's face. The synthetic blending module 320 may modify the virtual human's expression based on viewer reactions and interactive scenarios. For example, the composition blending module 320 may render a face of a virtual human with facial expressions matching words and sentences of an interactive scenario. The synthetic blending module 320 may modify the expression of the virtual human based on viewer reaction. Specifically, the synthesis blending module 320 may collect viewer reactions in real time, compare target emotional values of the story with real-time emotional values, and adjust the level of liveliness of virtual human expressions based on the comparison result. For example, since the viewer's real-time emotion value for the target emotion type 'impression' may exceed the target emotion value of the story only by the story itself, the real-time emotion value can be controlled by adjusting the liveliness of the virtual human's expression. .

The text-to-speech (TTS) module 330 may convert the text of the scenario into the voice of a virtual human.

3 is a flowchart illustrating the operation of a reaction analyzer according to an exemplary embodiment of the present disclosure.

Referring to FIG. 3 , the response analyzer 130 may collect viewer responses from a plurality of viewer devices (S31). Viewer responses may be voice responses or text responses.

The reaction analyzer 130 may determine the emotional type of viewers based on the voice reaction (S32). That is, the reaction analyzer 130 may determine the type of emotion in real time.

Specifically, the reaction analysis unit 130 converts the voice response into text, extracts a plurality of core texts included in the text, and counts the types of emotions represented by the plurality of core texts, thereby determining viewers' statistical emotions from the voice response. type can be determined.

The response analysis unit 130 may calculate the emotional level as a numerical value based on the text response and the voice response (S33). That is, the reaction analysis unit 130 may calculate a real-time emotional value.

Specifically, the response analysis unit 130 may calculate a real-time emotion value by calculating a frequency of core text within a text response and a voice response converted into text. Since the voice response is more active than the text response, the response analyzer 130 may calculate a real-time emotional value by giving weight to the core text included in the voice response rather than the core text included in the text response. The real-time emotion value may be calculated in proportion to the frequency of the core text.

4 is a flowchart illustrating an operation of a storyline generator according to an exemplary embodiment of the present disclosure. 5 is a diagram illustrating sub-story updating according to an exemplary embodiment of the present disclosure.

Referring to FIG. 4 , the storyline generating unit 200 may generate an interactive scenario (S41). The initial interactive scenario may be a combination of predetermined stories.

For example, referring to FIG. 5 , an interactive scenario may include a plurality of predetermined stories such as story 1 and story 2. A story may also be referred to as a shot, cut, scene, sequence, or the like.

Referring to FIG. 4 , when the current sub-story ends (S42=Y), the storyline generator 200 compares a real-time emotion value with a target emotion value based on whether the current sub-story has ended, and returns the comparison result. Based on this, the next sub-story can be updated (S43). Referring to FIG. 5 , when the current sub-story is sub-story 1 and sub-story 1 ends, the storyline generator 200 may compare the real-time emotion value '9' with the target emotion value '10'. . Since the real-time emotional value is lower, the storyline generating unit 200 may update the next sub-story to sub-story A instead of the preset sub-story B. Sub-story A may be preset to have a higher relative emotional value than sub-story B.

That is, whenever a sub-story ends, if the viewer does not respond as expected, the scenario of the next sub-story is changed to a dramatic content, thereby providing a sense of immersion to the viewer.

According to an exemplary embodiment of the present disclosure, since content of a scenario continuously changes according to a viewer's response, the viewer can be provided with realistic interactive content.

6 is a flowchart illustrating an operation of the storyline generating unit 200 according to an exemplary embodiment of the present disclosure. 7 is a diagram for explaining a method of selecting a subsequent story candidate group according to an exemplary embodiment of the present disclosure.

Referring to FIG. 6 , when the existing story ends (S31=Y), the storyline creation unit 200 may receive real-time emotional values from the viewer response collection unit 100 (S32).

Specifically, referring to FIG. 7 , when story 1 is completed, that is, when sub-story 4 is completed, the storyline generator 200 converts a real-time reaction value '5' to story 1 into a viewer response collection unit 100. ) can be received from

The storyline generating unit 200 may determine a target value range of a subsequent story based on the real-time emotion value of the previous story (S33). Referring to FIG. 7 , the storyline generator 200 may compare a real-time emotional value '5' of story 1 and a target emotional value '10' of story 1, and determine a candidate group for story 2 based on the comparison result. .

Specifically, since the real-time emotional value of Story 1 is lower than the target emotional value, it can be understood that it received a poor response from the viewers, contrary to expectations. Accordingly, the storyline generator 200 may adjust the story 2 candidate group so that the story 2 following the story 1 has a higher target emotional value than the story 1.

That is, the storyline generator 200 may include stories whose target emotion value exceeds 10, that is, stories 2-2, 2-3, and 2-4, in the candidate group for story 2, and the goal emotion value may be included. A story with a numerical value of 10 or less, that is, story 2-1, can be removed from the candidate group for story 2. Story 2-1 to story 2-4 are stories that can be connected to story 1 and can be set when the initial interactive scenario is created.

The storyline generating unit 200 may update the scenario to provide the viewer with the right to select stories belonging to the candidate group (S34).

Specifically, after story 1 is over, among actions that a character can take, an option indicating an action that is smoothly connected to stories belonging to the candidate group may be provided to the viewer (S34).

8 is a flowchart illustrating an operation of a virtual human generator according to an exemplary embodiment of the present disclosure.

Referring to FIG. 8 , the virtual human generating unit 300 may render facial expressions of the virtual human according to the scenario context (S81).

Specifically, words and sentences included in the scenario may be extracted, and the virtual human's face may be rendered with facial expressions mapped to the words or sentences.

The virtual human generating unit 300 may receive real-time emotional values of the viewer 30 from the viewer reaction collecting unit 100 (S82).

Specifically, the virtual human generator 300 may receive real-time emotional values not only when a sub-story or a story ends, but also in units of words or sentences within a scenario.

The virtual human generating unit 300 may correct the expression of the virtual human based on the real-time emotional value of the viewer 30 (S83).

Specifically, the virtual human generation unit 300 may compare a target emotion value of the story with a real-time emotion value, and adjust the level of liveliness of the virtual human expression based on the comparison result. For example, since the viewer's real-time emotion value for the target emotion type 'impression' may exceed the target emotion value of the story only by the story itself, the real-time emotion value can be controlled by adjusting the liveliness of the virtual human's expression. . Through this, since the expression of the virtual human changes according to the reaction of the viewer 30 even in the same storyline, various interactive contents can be provided to the viewer 30 .

The present invention can also be implemented as computer readable codes on a computer readable recording medium.

A computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. include In addition, the computer-readable recording medium may be distributed to computer systems connected through a network so that computer-readable codes may be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the present invention can be easily inferred by programmers in the technical field to which the present invention belongs.

As above, exemplary embodiments have been disclosed in the drawings and specifications. Although the embodiments have been described using specific terms in this specification, they are only used for the purpose of explaining the technical idea of the present disclosure, and are not used to limit the scope of the present disclosure described in the claims. . Therefore, those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Therefore, the true technical scope of protection of the present disclosure should be determined by the technical spirit of the appended claims.

10: content creation platform 20: content transmission platform
30: viewer 100: viewer response collection unit
200: story line generation unit 300: virtual human generation unit
110: STT module 120: core text extraction unit
130: reaction analysis unit 210: interactive scenario generation unit
220: scenario update module 310: motion capture module
320: synthetic blending module 330: TTS module

Claims (7)

As a content creation platform device for generating virtual content,
Real-time emotional types of viewers who receive voice and text responses to the virtual content from viewer devices that receive the virtual content, and view the virtual content for the virtual content based on the voice and text responses; and a viewer response collection unit that calculates a real-time emotion value for the real-time emotion type;
a storyline generating unit generating an interactive scenario of the virtual content and updating the interactive scenario based on the real-time emotion type and the real-time emotion value; and
a virtual human generator configured to generate a virtual human of the virtual content and adjust a facial expression of the virtual human based on the real-time emotion type and the real-time emotion value;
The storyline generator,
A real-time emotion value for a first story provided as virtual content is compared with a target emotion value of the first story, and based on the comparison result, a target value range for a second story following the first story is determined; A scenario that varies according to the reaction of the viewer is created by determining a candidate group for the second story based on the target numerical range and providing the viewer with options for a story belonging to the candidate group to actions that the character can take next. A content creation platform device, characterized in that for generating. According to claim 1,
The viewer reaction collection unit,
Converting the voice response into text, extracting a plurality of predetermined core texts from the converted text, and determining the real-time emotion type based on a frequency of each of the plurality of core texts mapped to emotion types. Characterized by a content creation platform device. According to claim 2,
The viewer reaction collection unit,
The content creation platform device, characterized in that for calculating the real-time emotion value based on the real-time emotion type and the frequency of the first core text mapped from the text response and the converted text. According to claim 3,
The viewer reaction collection unit,
The real-time emotional value is calculated by giving the first core text extracted from the converted text more weight than the first core text extracted from the text response. According to claim 1,
The storyline generator,
When a current story among a plurality of stories included in the interactive scenario is completed, a target emotional value of the current story is compared with the real-time emotional value, and a story candidate group selected as the next story by the viewer device based on the comparison result. Content creation platform device, characterized in that for updating. According to claim 5,
The storyline generator,
When a current sub-story among a plurality of sub-stories included in each of the plurality of stories is completed, a target emotional value of the current story and the real-time emotional value are compared, and a next sub-story is updated based on the comparison result. A content creation platform device, characterized in that. According to claim 1,
The virtual human generation unit,
The content creation platform device, characterized in that for primarily rendering the expression of the virtual human based on the text of the interactive scenario, and secondarily correcting the expression of the virtual human based on the real-time emotion value.