JP2021182369A - Method and device for providing interactive content re-programmed on the basis of playing of user - Google Patents

Abstract

To provide a user with a new content which reflects a progress step itself of an interactive content that the user has determined and completed.SOLUTION: An interactive content provision device of the present invention includes: an input interface for sensing input of a user; a display unit for outputting a picture; a memory for storing a command word to make a predetermined operation done; and a processor connected to the memory in an operable manner, the processor executing the command word. The processor executes the operations of: providing a user with an interactive content which is played while an object is moving according to drive information as a plurality of animation units is regenerated, the animation units including object information specifying an object output to a picture and drive information defining an operation of the object; receiving input of the user information reflected on the interactive content; and inserting the user information into a part of the object information and playing back one of the animation units to play the interactive content.SELECTED DRAWING: Figure 2

Description

The present invention relates to an interactive content providing method and device that are reprogrammed based on the user's playing, and more specifically, the interactive content provides information on the process in which the user plays the interactive content and information that can reflect the user's own personality. Reflecting on the technology to generate new unique content unique to users.

Content such as existing movies, dramas, and books have a synopsis that follows the plot of the story set by the creator, leading to the end. However, unlike the defined plots, there are people's desires for the flow and different endings they want, so people intervene in the story to create a Gaiden or imagine themselves in the Internet community. It also describes stories about different endings.

Reflecting the desires of these people, interactive content, in which the flow and outcome of the content can be selected by the user's decision, has recently been in the limelight. In interactive content, the synopsis may be different depending on the user's choice as the story progresses, and not only the synopsis but also the conclusion may change depending on the user's choice.

However, most of the existing interactive content focuses on differentiating only the story at the user's choice, and these days, it is a function that allows the user to autonomously intervene in more diverse elements beyond the story of the content. Is required. For example, there is a demand for personalized content (Personalized Content), realistic content, and the like, in which user information (face, voice, name, etc.) is incorporated into the content so that the user can further experience the content.

Since the above-mentioned interactive content can change not only the synopsis but also the conclusion depending on the user's choice, it leads to the result that even if the same kind of content is enjoyed, each individual can create a completely different story by the user's individual choice.

Along with this, the problem to be solved by the embodiment of the present invention is to provide a technique for providing the user with new content that reflects the progress process itself of the interactive content determined and completed by the user himself / herself. be.

In addition to this, the problem to be solved by the embodiment of the present invention can reflect the user's own individuality in various elements such as characters and voices appearing in the interactive content beyond the autonomy that affects the story. Try to provide technology.

For this reason, the examples of this document include object information that identifies the object output to the image and driving information that defines the operation of the object, unlike the format of interactive content in which the image output in frame units is defined. We try to provide a technology that configures interactive content so that the video is played in units of animation, makes it easy to change the flow of the interactive content, and effectively reflects the information of the user who plays the content in the content.

However, the technical problems to be achieved by the embodiments of the present invention are not limited to the problems mentioned above, and various technical problems are derived from the contents described below within a range that is obvious to ordinary engineers. obtain.

In the method for providing interactive content according to an embodiment of the present invention, the object is reproduced as a plurality of animation units including the object information specifying the object output to the video and the driving information defining the operation of the object are reproduced. Provides the user with interactive content that is played while moving by the driving information; the stage of receiving the input of the user information reflected in the interactive content; and the user as part of the object information. It may include a step of inserting information and playing at least one of the plurality of animation units to play the interactive content.

An interactive content providing device according to an embodiment of the present invention is an input interface that senses user input; a display unit that outputs video; one or more memories that store command words that perform a predetermined operation; And one or more processors operably linked to the one or more memories and configured to execute the command word, the one or more processors being an object output to the video. As a plurality of animation units including the object information specifying the object and the driving information defining the operation of the object are played, the operation of providing the user with interactive content in which the object is played while being moved by the driving information; The operation of receiving the input of the user information reflected in the interactive content; and the operation of inserting the user information into a part of the object information and playing back at least one of the plurality of animation units. Can include.

The interactive content according to the embodiment of the present invention is composed of animation units including object information that identifies the object output to the video and driving information that defines the movement of the object, and the user interacts while the interactive content is in progress. Not only information about the process of playing the content, but also the individuality of the user such as the user's face, voice, and play characteristics can be reflected in various elements of the content.

Along with this, the user can not only select the flow of the story, but also generate unique content with the user's personality so that the user and his / her family can play the content as characters of the content. Can be done.

In addition to this, users can generate, store and share new content that is played by reflecting the faces and voices of themselves, friends and family on the character's face, character's voice, etc. of interactive content. can.
In addition to this, various effects that are directly or indirectly grasped through this document may be provided.

FIG. 1 is a functional block diagram of an interactive content providing device according to an embodiment of the present invention. FIG. 2 is an exemplary diagram for explaining object information and driving information included in the animation unit according to the embodiment of this document. FIG. 3 is an exemplary diagram for explaining audio information and subtitle information included in the animation unit according to the embodiment of this document. FIG. 4 is an exemplary diagram for explaining play information used to generate the second content. FIG. 5A is an exemplary diagram for explaining the difference between the subtitles output in the process of playing the interactive content and the subtitles output in the second content. FIG. 5B is an exemplary diagram for explaining the difference between the subtitles output in the process of playing the interactive content and the subtitles output in the second content. FIG. 6 is an exemplary diagram for explaining an operation of inputting a user voice into interactive content. FIG. 7 is a first exemplary diagram for explaining an operation of substituting a user's face image into interactive content. FIG. 8 is a second exemplary diagram for explaining an operation of generating an operation of substituting a user's face image into the interactive content. FIG. 9 is an exemplary diagram for demonstrating an operation in which an interactive content substitutes a user's face image and embodies a facial expression or movement in the substituted face image. FIG. 10 is an exemplary diagram for explaining an operation of changing the position of object information depending on the usage environment of the interactive content providing device. FIG. 11 is an exemplary diagram for explaining an operation of specifying a class of object information according to an embodiment through an artificial intelligence model and mapping drive information corresponding to the class. FIG. 12 is a flowchart of an interactive content providing method according to an embodiment of the present invention.

FIG. 1 is a functional block diagram of an interactive content providing device 100 according to an embodiment of the present invention. Referring to FIG. 1, the interactive content providing apparatus 100 according to an embodiment can include a memory 110, a processor 120, an input interface 130, a display unit 140, a communication interface 150, and a camera unit 160.

The memory 110 includes an animation DB 111, an interactive DB 113, and a command word DB 115, and can store information and command words for carrying out the embodiments of this document.

The animation DB 111 can store information for the animation unit constituting the interactive content. The animation unit is a unit that divides the animation that defines how the object contained in the video must move, and the animation unit is the object information that identifies the object output to the continuous video and the object information. It can include driving information that defines the movement of such an object.

That is, the object information is information for each object that is seen or provided to the user when the video is played, and is information that identifies the object such as a character, an object, a background, a voice, and subtitles. .. For example, the object information may further include a three-dimensional user model that reflects the face of the character or the face of the user, and information about the second object that can be represented with the user model based on the relative distance to the user model. can.

The drive information can include information about when and how each object should behave, or information about how it should react to user input.

The interactive content according to the embodiment of this document is composed of such animation units, the interactive content is advanced by the operation of the animation unit, and the process of the user playing the interactive content is included in the information included in the animation unit. By being reflected, information that can reflect the individuality of the user can be reflected in the object information or the driving information.

The interactive DB 113 can receive and save the input of the user information reflected in the interactive content. In addition, the interactive DB 113 can store user information that has been improved or modified by a predetermined algorithm. For example, the user information can include a user's face image, a user's voice, and various information input by the user by playing interactive content.

The instruction word DB 115 can store an instruction word for performing the operation of the processor 120 described later, or a computer code for performing an operation corresponding to the operation of the processor 120 described later.

The processor 120 can control the overall operation of the configuration, memory 110, input interface 130, display unit 140, and communication interface 150 included in the interactive content providing device 100. The processor 120 can include an interactive module 121, an audio subtitle module 123 and an image module 125. The processor 120 can execute the instruction words stored in the memory 110 to drive the interactive module 121, the audio subtitle module 123, and the image module 125. The operations performed by the interactive module 121, the audio subtitle module 123 and the image module 125 can be understood as the operations performed by the processor 120.

When the user plays the interactive content, the interactive module 121 can control the progress of the interactive content in general, store the information related to the user, and generate the second content. For example, the interactive module 121 can determine the progress of the interactive content based on the user input, can reflect the information input by the user in the interactive content, and save a record of the user playing the interactive content. Second content can be generated.

The audio subtitle module 123 controls subtitles and general audio-related operations. For example, when the user senses the user input for the subtitle output to the display unit 140, the user's voice input can be received, and this is mapped to the subtitle information as the voice information corresponding to the subtitle and saved. Can be done.

The image module 125 controls the overall behavior associated with the image. For example, when receiving input of a face image from a user, the face image can be changed or modified according to a predetermined purpose, and it is determined whether the input image matches the purpose for input. , The input image can be reflected in the interactive content.

The input interface 130 can receive inputs such as physical stimuli, audio, and video applied by the user. For example, the input interface 130 can include hardware modules such as decompression sensors, electrostatic sensors, microphones, cameras and the like.

The display unit 140 can output an image so that the user can play the interactive content. For this purpose, the display unit 140 can include a hardware configuration that includes a display panel to output video, but the type of hardware that constitutes the display unit 140 is not limited to such an example. ..

The communication interface 150 communicates with an external device so that information can be transmitted and received. For this purpose, the communication interface 150 can include a wireless communication module or a wired communication module.

The camera unit 160 can generate a peripheral image taken by photographing the surrounding environment of the interactive content providing device. For this purpose, the camera unit 160 can include a camera module which is a hardware configuration.

Hereinafter, with reference to FIGS. 2 and 3, the characteristics of the interactive content provided by the interactive content providing device 100 according to the embodiment of this document will be described first, and the processor 120 will be used to achieve the task presented by this document. After explaining the overall flow of the operation performed by the user, a specific example of reflecting the user information in the interactive content will be described together with FIGS. 4 to 7.

First, the animation unit constituting the interactive content according to the embodiment of this document will be described in more detail.

FIG. 2 is an exemplary diagram for explaining object information and driving information included in the animation unit according to the embodiment of this document, and FIG. 3 illustrates audio information and subtitle information included in the animation unit according to the embodiment of this document. It is an example figure for this.

The interactive content according to one embodiment is composed of a plurality of animation units. The animation unit can include object information, driving information, subtitle information, and audio information. The object information is information that identifies the object that is output to the video. Drive information is information that defines the movement of the object. The subtitle information is information for the text that is output together with the video as a description of the story of the video or a dialogue of the character while the video is output. The voice information is information for the voice that reads the text corresponding to the subtitles by voice. Along with this, when the user starts playing the interactive content, the interactive module 121 plays the animation unit included in the interactive content so that the object information contained in the animation unit is moved by the driving information and is interactive. Content can be advanced, and subtitles based on subtitle information and audio based on audio information can be output as the content progresses.

The interactive module 121 can control the order or time point in which each drive information should be driven so that it is driven in accordance with the defined order and time point.

Further, in the interactive content according to the embodiment of this document, when the input to a specific object is sensed while the interactive content is being played separately from a plurality of animation units, the motion itself that occurs in the object is already defined. Further object reaction animation information can be included. The interactive module 121 can control the specific object to move by the object reaction animation information based on the user input to the object information output to the display unit 140. For example, in the interactive content, when an object called "clock" exists in the object information, the user faces the "clock" while the "clock" is being output to the display unit 140, even while any animation unit is being played. When an input (ex. Touching the clock part of the object) is detected, the object reaction animation information that defines the operation such as operating the clock to move or outputting the sound of what time it is now. Can be saved.

FIG. 3 is an example in which the audio information and the subtitle information included in one animation unit are illustrated, and the audio information and the subtitle information can be mapped and stored with one drive information. When the specific drive information is executed, the audio subtitle module 123 can output the subtitle information and the audio information together with the execution of the corresponding drive information.

Next, an example of a method in which each animation unit is advanced will be described.

The interactive module 121 can determine the order in which the plurality of animation units included in the interactive content are played by various embodiments. For example, the animation reproduction method according to the embodiment can include an animation method in which a plurality of animation units are reproduced in a preset order in the method in which the animation unit is reproduced.

Further, the animation reproduction method according to the embodiment outputs the user's first input for a plurality of selection items that are selected to reproduce any one of the plurality of animation units, or during reproduction of a predetermined animation unit. It is possible to include a branch animation method for reproducing the next animation unit determined based on each first input or the second input by the user's second input for the object information to be performed. When the animation unit corresponding to the branch animation method is played back, the interactive module 121 outputs a plurality of selection items (ex. The main character's selection items along the flow of the story) to the display for a specific selection item from the user. The first input can be sensed, or the user's second input for specific object information can be sensed to reproduce an animation unit defined to be reproduced in response to the user input.

The interactive content providing device 100 can provide the object information and the driving information itself constituting the animation unit so that the user (ex. Player, administrator, developer, etc.) can directly edit them. Therefore, when the user requests editing, the interactive module 121 provides the user with the object information and the driving information constituting each animation unit, and the user for the reproduction order of the object information, the driving information, or the animation unit. You can receive the input of the editing information of. Along with this, the interactive module 121 modifies the object information or the driving information according to the user's preference by playing the interactive content including the information for the animation unit reflecting the input editing information, and the other object. Appears, or content that reflects the user's desired movement can be played. In addition, the playback order of each animation unit can be controlled so as to be played back based on the user's editing.

Next, the overall process in which the interactive content providing device 100 according to the embodiment operates will be described.

First, the processor 120 provides the user with the interactive content according to the embodiment of the above-mentioned document, and receives input of user information reflected in the interactive content from the user before or during the play of the interactive content. Can be saved. Along with this, the processor 120 inserts the input user information into a part of the object information included in the interactive content, and reproduces at least one of the plurality of animation units to play the interactive content. Can be done. After that, when the user's play is finished, the processor 120 can generate the second content in which the animation unit is reproduced by the information input by the user while the user is playing the interactive content.

The second content according to one embodiment is a video in which the animation unit progresses in the order corresponding to the input in which the user progresses the story while playing the interactive content, and reproduces the story played by the user. In addition to this, it is a video that reflects all the information input by the user (information that the specific object was pressed during play, play time, face image, sound, etc.). For this purpose, one embodiment of the present invention generates play information as shown in FIG.

FIG. 4 is an exemplary diagram for explaining play information used to generate the second content. Referring to FIG. 4, the interactive module 121 identifies the event type, input time, and input value information for the input information input by the user playing the interactive content, and maps the identified information to the play information. Can be generated. The event type specifies the type of event generated by the user, for example, "next page", "object touch", "previous page", and "select one of the three selection items". ", Etc., which means the type of event generated by the user. The input time means the time or time when the user generates the corresponding event. The input value can mean information generated or received by user input, such as coordinates of a point entered by the user to generate an event, input audio information, or input image.

Along with this, after the play of the interactive content is completed, the interactive module 121 can generate the second content which is a video in which the interactive content is played according to the information of the event type, the input time and the input value included in the play information. can. For example, when the input value included in the play information is the coordinates of the input point, the interactive module 121 can control so that the corresponding point is input at the input time included in the play information and reproduce the content. Further, when the input value included in the play information is audio information, the interactive module 121 causes the audio information included in the input value to be reproduced in the event type (ex. Audio recording for subtitles) included in the play information. be able to. In addition to this, when the input value included in the play information is a face image, the interactive module 121 assigns the face image included in the input value to the event type (ex. Image assignment to the character's face) included in the play information. Can be played back.

Next, the difference between the interactive content and the second content will be described.

The second content according to one embodiment is a video in which the animation unit progresses in the order corresponding to the input in which the user progresses the story while playing the interactive content, and records the story played by the user. In addition to this, it is a video that reflects all the information input by the user (ex. Information that a specific object is pressed during play, play time, face image, voice, etc.).

5A and 5B are illustrations for explaining the difference between the subtitle A output in the process of playing the interactive content and the subtitle B output in the second content.

Referring to FIGS. 5A and 5B, the audio subtitle module 123 drives a plurality of subtitle information at once in order to produce a picture book-like feeling while playing interactive content as in A. By outputting multiple lines of subtitles on the screen and outputting audio information corresponding to the output multiple subtitle information, it is possible to give the user the feeling of a picture book reading the subtitles. On the other hand, like B, the second content can output only the audio information to be reproduced and the corresponding subtitle information in order to produce a cartoon-like or movie-like feeling.

However, in contrast to the method of outputting only the audio information to be reproduced and the corresponding subtitle information so that the sync of the audio information and the subtitle information is matched, the video is configured to be reproduced in frame units. If so, the subtitles may be played according to the frame time when a specific sound is played as in the existing technology, but the embodiment of this document is not played in frame units, but is selected by the user. There is a problem that it is difficult to specify the time when the subtitles are output because the playback is performed in units of animation.

For this purpose, the audio information can include an audio file having a reproduction time of a preset time or less, and the audio subtitle module 123 corresponds to the audio information to be reproduced when the audio information is reproduced in the second content. Only the subtitle information to be output can be output to the second content only during the reproduction time. That is, by outputting the corresponding subtitle information based on the playback time of each voice information for the corresponding playback time, the audio information played even in the animation unit configuration and the sync of the corresponding subtitle information are combined to form a cartoon. Or you can embody a movie-like feeling.

Next, an embodiment in which various user information is input and reflected in the interactive content and the second content will be described.

FIG. 6 is an exemplary diagram for explaining an operation of inputting a user voice into interactive content.

When the user input for the subtitle output in the video is detected during the play of the interactive content, the audio subtitle module 123 can receive the input of the user audio corresponding to the subtitle information. For example, an input interface window as shown in FIG. 6 can be output to the display unit 140 to guide the user's voice input. Along with this, when the voice spoken by the user is input through the input interface 130, the voice subtitle module 123 inserts the user voice into the voice information corresponding to the subtitle information, and when the play of the interactive content is completed, the user voice is completed. It is possible to generate a second content in which the animation unit in which the audio information in which is inserted is output corresponding to the subtitle information is reproduced.

At this time, in the input interface 130 window, the first button for searching the subtitle to be output before the subtitle information to be recorded by the user, and the subtitle to be output after the subtitle information to be recorded by the user are displayed. The second button to search may be output, and the audio subtitle module 123 can output the previous subtitle or the next subtitle based on the user input to the first button or the second button. Further, the voice subtitle module 123 receives the input of the user voice corresponding to the subtitle information output based on the user input to the first button or the second button, and the user voice is inserted when the play of the interactive content is completed. It is possible to generate a second content in which the animation unit in which the generated audio information is output corresponding to the subtitle information is reproduced.

On the other hand, in the above-described embodiment that receives the input of the user voice, when the user voice is input for more than a preset time, or when the voice is input to various characters, the time or the number of characters. It is possible to charge the user in proportion to.

FIG. 7 is a first exemplary diagram for explaining an operation of substituting a user's face image for interactive content, and FIG. 8 is for explaining an operation of generating an operation of substituting a user's face image for interactive content. It is the second example figure of.

Referring to FIG. 7, the image module 125 can receive input of user information before or during the play of interactive content, and when a predetermined image of the user information is input, the corresponding image is a face. If the image includes an image, the image module 125 can standardize the face image so as to look at a specific point, and substitute the standardized face image into the object information corresponding to the character's face in the object information. ..

According to one embodiment of performing the standardization, when a face image is input from the user, the image module 125 outputs an image in which the face image is deformed to form a predetermined angle, shape, or facial expression. Various artificial intelligence models trained in the above can be utilized to standardize the facial image to form a predetermined angle, morphology, or facial expression. After that, the image module 125 derives the feature points of the face image by various algorithms for deriving the feature points, and the derived feature points can be specified as vertices. After that, the image module 125 can connect the vertices to each other to generate a three-dimensional model composed of a mesh structure which is a set of polygons consisting of vertices and square faces.

Along with this, the image module 125 can control the position of the apex and standardize the 3D model so as to face a predetermined angle and shape, so that the face image can be standardized so as to look at a specific point, and the object information can be standardized. A three-dimensional model can be substituted for the object information corresponding to the character's face. At this time, the frontal image and the occipital image of the character already saved in the user's face image can be combined, and the user's face image can be more naturally substituted into the character's face image. For example, when there are three types of frontal image, occipital image, and face image, the image module 125 arranges the layers of each image at the back of the head image at the end, with the face image in the middle, and the frontal image at the front. The user's face can be naturally assigned to the character's face by arranging them in the order in which they come to.

Further, the image module 125 can control the vertices constituting the three-dimensional model to move by the driving information to embody the movement or facial expression of the face image. At this time, when trying to additionally select the substitution of the face image to the character in addition to the basic character in the object information, the image module 125 tells the user in proportion to the number of characters to substitute the input face image. Billing can be levied.

Along with this, referring to FIG. 8, it is possible to substitute the image for which 3D modeling has been completed into the original object information based on the face image, the standardized face image, and the 3D modeling image input by the user. The movement can be embodied in the image for which 3D modeling has been completed. The motion that embodies the motion can include the embodiment of FIG.

FIG. 9 is an exemplary diagram for demonstrating an operation in which an interactive content substitutes a user's face image and embodies a facial expression or movement in the substituted face image.

First, the image module 125 can specify the first vertex at which the movement starts and the second vertex at which the movement ends among the vertices constituting the three-dimensional model. Each of the first and second vertices can contain one or more vertices. When both the first vertex and the second vertex include a plurality of vertices, the first vertex may be connected to at least one vertex different from each other, and the second vertex includes the same number of vertices as the plurality of vertices. be able to.

For example, when the 3D model has the shape of a face and embodies the movement of closing the eyes, the image module 125 has the apex located on the skin at the upper end of the eyeball as the first apex and the apex located on the skin at the lower end of the eyeball. Can be specified as the second vertex. Hereinafter, the examples of this document will be described by taking the movement of the eyelids as an example, but the technique according to the examples of this document is not limited to such an example.

After the first and second vertices have been identified, the image module 125 can determine the vertices, corners or faces that connect the first and second vertices between the first and second vertices. For example, the image module 125 can determine the vertices, angles, or faces that connect the first vertex to the second vertex at the shortest distance.

Further, when the first vertex or the second vertex includes a plurality of vertices, the image module 125 follows a predetermined line reaching each of the plurality of second vertices specified as the end points of the movements of the plurality of first vertices. Therefore, n points (n is a natural number of 1 or more) can be specified. The operation of the image module 125 to identify n points along a predetermined line between the first vertex and the second vertex can be embodied through various examples below, and the examples described in this document are exemplary. However, it is not limited to such an explanation.

According to one embodiment, the image module 125 can specify the vertices constituting the angle connecting the first vertex and the second vertex at the shortest distance as one of n points. For example, in the case of embodying the movement of the eyelid being closed by the eyeball, if there are four angles connecting the first and second vertices, each of the three vertices connecting the corners is n = 0 point, n. It can be specified as = 1 point and n = 2 points.

Further, when the first vertex or the second vertex includes a plurality of vertices, the image module 125 is connected until each of the plurality of second vertices specified as the end point L for the movement of each of the plurality of first vertices is reached. On the surface, the points where the shortest virtual line connecting the first vertex to the second vertex is divided into n + 1 equal parts can be specified as the n points. For example, when referring to FIG. 8, when embodying the movement of the eyelid being closed by the eyeball, the shortest distance connecting the first vertex and the second vertex along the face from the first vertex to the second vertex. The three points that divide the virtual line of the above into four equal parts can be specified as n = 0 points, n = 1 points, and n = 2 points.

After the n points have been identified, the image module 125 controls the position data of the first vertex so that the first vertex moves toward the second vertex through the n points in order to realize the movement. be able to. Further, the image module 125 can control the vertices other than the first vertex located within a predetermined range from the first vertex to move according to the first vertex, which is embodied through algorithms of various feature points. Obtained, not limited to a particular method.

The image module 125 can realize a natural movement by controlling the first vertex to reach the second vertex after passing through n = 0 point, n = 1 point, and n = 2 points, respectively. At this time, the image module 125 determines a velocity function that reflects the speed at which the first vertex moves until the first vertex moves toward any one of the n points, as shown in the following mathematical formula 1. can do. Along with this, the speed at which the first vertex moves is determined based on the characteristics that can reflect the actual movement, so that the movement of the three-dimensional model can be used more naturally.

In addition to this, the image module 125 can place the second object in a user model that reflects the face of the character or the face of the user. For example, when the user model is a three-dimensional model consisting of a plurality of vertices, the relative distance between one or more vertices included in the three-dimensional model and the second object is set to be kept constant, and is virtual. It is possible to generate a user model that is represented together with the second object in the space of. In this case, even if the facial expression or form of the user model is deformed by the driving information, the second object can be expressed together while maintaining a certain distance from the user model. Along with this, when the interactive content is played and the user model moves by the driving information, the movement of the second object can be expressed so that the relative distance from the user model is kept constant.

Further, when the image module 125 receives the input of the face image from the user, the first tag indicating the attribute of the face image by analyzing the input face image based on various artificial intelligence algorithms such as the AI face recognition device. Information (ex. Age, gender, race, accessories, hairstyle, hair color, pupil color, etc. of face image) can be extracted. After that, the second tag information (ex. Character's age, gender, race, accessories, hairstyle, hair color, pupil color, etc.) and the first tag information mapped to the object information corresponding to the character's face. It is possible to substitute the face image input to the character having the largest amount of the first tag information and the corresponding second tag information by comparing with. At this time, the second tag information is mapped to the object information corresponding to the face of the character and is already stored. Alternatively, like the existing AI face recognition device of the image module 125, the image of the character can be analyzed based on various artificial intelligence algorithms to extract the second tag information indicating the attribute of the character.

Further, when the image module 125 receives the input of the face image from the user, the tone of the input face image can be adjusted or the face image can be corrected via a photographic filter. For example, the image module 125 can correct the face tone based on the existing photo correction algorithm when the user's face image is input, and the face image is made according to a predetermined rule through the existing photo filtering technology. You can apply a transforming filter to apply a cute face, a young face, an old face, etc. and substitute it into the character's face.

Further, when the image module 125 receives an image input from the user, it can determine whether the input image matches the character's face. For example, when the image module 125 receives an input of a user's image, it deviates from a set face information standard based on various artificial intelligence algorithms such as an AI face recognition device (ex. Face is blocked). You can request to enter another facial image (for example, if you have closed your eyes, or if your front face does not contain more than 90%).

The interactive content according to the above-described embodiment is composed of animation units including object information that identifies the object output to the video and driving information that defines the movement of the object, and the user can interact with the interactive content while the interactive content is in progress. Not only information about the process of playing the content, but also the individuality of the user such as the user's face, voice, and play characteristics can be reflected in various elements of the content. Along with this, the user can not only select the flow of the story, but also generate unique content with the user's personality so that the user and his / her family can play the content as characters of the content. Can be done. In addition to this, the user can generate, store, and share new content that is played by reflecting the faces and voices of himself, friends, and family on the character's face, character's voice, etc. of the interactive content. Can be done.

FIG. 10 is an exemplary diagram for explaining an operation of changing the position of object information according to the usage environment of the interactive content providing device.

Referring to FIG. 10, the interactive module 121 can control the position of the object information output to the display unit 140 according to the usage environment according to the environment in which the user plays the interactive content.

For example, the interactive module 121 changes from the content standard of the original of FIG. 10 (ex. Horizontal: vertical = 2.5: 1 size) to the content standard of the book form (ex. Horizontal: vertical = 5: 1 size). When input is received, the position where the object information should be output is different. Based on the coordinate information already set for each standard, the object information is output by the coordinate information corresponding to the standard by the user's standard selection. It is possible to control the position where the object is output in one animation unit. Further, the interactive module 121 can move the position where the object is output based on the result coordinate information output by substituting the basic coordinate information into the function that defines the degree to which the object should move according to the content standard. ..

In addition, the interactive module 121 can output an image of object information on a layer of video that outputs the surrounding environment by using augmented reality (AR) technology. For example, the interactive module 121 drives the camera unit 160 to receive the input of the peripheral image of the surrounding environment, and the object is based on the coordinate information in which the position where the object information should be output in the display unit 140 is already set. The information can be arranged at the top of the layer of the peripheral image and output to the display unit 140.

Alternatively, the interactive module 121 drives the camera unit 160 to receive input of a peripheral image of the surrounding environment, and the object is based on a function designed so that the object information is mapped to one of the positions of the peripheral image. Information can be placed above the layer of the peripheral image and output to the display unit. In this case, since the object information is mapped to one point in the peripheral image, the object can be placed without moving from the mapped point even if the user turns the camera to take a picture of another place.

Although the driving information of the object information has already been mapped to each other in the above-described embodiment, according to the embodiment of FIG. 11, the mapping can be performed through artificial intelligence technology.

FIG. 11 is an exemplary diagram for explaining an operation of specifying a class of object information according to an embodiment through an artificial intelligence model and mapping drive information corresponding to the class.

Referring to FIG. 11, the interactive module 121 discriminates and discriminates a class of object information (ex. Information that identifies an object such as a character, a light, a tree, a car, a bird, a clock, etc.) input to a predetermined artificial intelligence model. It is possible to utilize an artificial intelligence model trained to map the driving information corresponding to the given class. The interactive module 121 can input the first object information among the objects included in the interactive content into the artificial intelligence model, map the first driving information determined by the artificial intelligence model, and store it in the animation DB 111.

At this time, the artificial intelligence model is the class of the input object information and the type of drive information mapped to the class (ex. Definition for movement such as vibrating movement, brightness adjustment, turning off the light, type of position movement, etc.) May be discriminated together. An artificial intelligence model trained to map the driving information corresponding to the determined class can be used. The interactive module 121 can input the first object information among the objects included in the interactive content into the artificial intelligence model and map the first driving information determined by the artificial intelligence model. The interactive module 121 can input the first object information among the objects included in the interactive content into the artificial intelligence model, map the class and the driving information output to the artificial intelligence model, and store the information in the animation DB 111.

Along with this, when the interactive module 121 senses the user input to the first object information, it can embody the movement by the first drive information.

The artificial intelligence algorithms, or artificial intelligence models included in the examples in this document, can use machine learning algorithms. Machine learning (Machine Learning) is a field that defines various problems to be dealt with in the field of artificial intelligence and studies methodologies for solving them. Machine learning is also defined as an algorithm that enhances the performance of some work through unwavering experience.

An artificial neural network (ANN) is a model used in machine learning, and can mean a model in general that has problem-solving ability and is composed of artificial neurons (nodes) that form a network by connecting synapses. .. An artificial neural network can be defined by a connection pattern between neurons in other layers, a learning process for updating model parameters, an activation function for generating output values, and the like.

The artificial neural network can include an input layer, an output layer, and optionally one or more hidden layers. Each layer contains one or more neurons, and artificial neural networks can contain neurons and synapses that connect them. In an artificial neural network, each neuron can output the function value of the activation function for the input signal, weight, and bias input through synapses.

Model parameters mean parameters determined through learning, including synaptic connection weights and neuronal bias. The hyperparameter means a parameter that must be set before learning by a machine learning algorithm, and includes a learning rate, a number of iterations, a mini-batch size, an initialization function, and the like.

The learning objective of an artificial neural network can be seen as determining the model parameters that minimize the loss function. The loss function can be used as an index for determining the optimum model parameters in the learning process of the artificial neural network.

Machine learning can be classified into supervised learning, unsupervised learning, and reinforcement learning according to the learning method.

Supervised learning means a method of training an artificial neural network with a label for the training data, and the label is inferred by the artificial neural network when the training data is input to the artificial neural network. It can mean the correct answer (or result value) that must be given. Unsupervised learning can mean a method of training an artificial neural network without giving a label to the training data. Reinforcement learning can mean a learning method in which an agent defined in an environment is trained to select an action or action sequence that maximizes cumulative compensation in each state.

Machine learning embodied in a deep neural network (DNN: Deep Natural Network) that includes multiple hidden layers in an artificial neural network is also called deep learning (deep learning), and deep learning is a part of machine learning. be. In the following, machine learning can be used to include deep learning.

On the other hand, in addition to the deep learning algorithm, decision tree (DT), k-nearest neighbor (KNN), logistic regression (LR), multilayer perceptron (MLP), random forest (RF), supportM. , You can select and use the appropriate algorithm according to the type and number of parameters to be used.

FIG. 12 is a flowchart of an interactive content providing method according to an embodiment of the present invention. Each step of the interactive content providing method according to FIG. 12 can be performed by the interactive content providing device 100 described through FIG. 1, and each step will be described as follows.

The device 100 produces interactive content in which the object is played while being moved by the driving information as a plurality of animation units including the object information for specifying the object output to the video and the driving information defining the movement of the object are reproduced. It can be provided to the user (S1210).

The device 100 can receive the input of the user information reflected in the interactive content (S1220).

The device 100 can insert user information into a part of the object information and reproduce at least one of the plurality of animation units to play interactive content (S1230).

On the other hand, since the process for carrying out each of the above-mentioned steps has been described with FIGS. 1 to 11, duplicated description will be omitted.

The embodiments of the present invention described above can be embodied through various means. For example, embodiments of the present invention may be embodied in hardware, firmware, software or combinations thereof.

In the case of hardware implementation, the method according to the embodiment of the present invention is one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital SignalProcessing). It can be embodied by Devices), FPGAs (Field Programmable Gate Arrays), processors, controllers, microprocessors, microprocessors and the like.

In the case of realization by firmware or software, the method according to the embodiment of the present invention may be embodied in the form of a module, procedure or function performing the function or operation described above. A computer program in which software code or the like is recorded may be stored in a computer-readable recording medium or a memory unit and driven by a processor. The memory unit is located inside or outside the processor and can exchange data with the processor by various means already known.

Further, the combination of each block of the block diagram and each step of the flowchart attached to the present invention may be performed by a computer program instruction. These computer program instructions can be mounted on a general purpose computer, specialty computer or other encode processor with programmable data processing, so that the instructions performed through a computer or other encode processor with programmable data processing are performed. , Will generate means to perform the functions described at each block of the block diagram or at each stage of the flowchart. Because these computer program instructions can also be stored in computer-enabled or computer-readable memory that can be aimed at computers or other programmable data processing equipment to embody functionality in a particular way. The instructions stored in the computer-available or computer-readable memory can also produce manufactured items that include instructional means to perform the functions described at each block of the block diagram or at each stage of the flowchart. .. Computer program instructions can also be mounted on a computer or other programmable data processing equipment, so that a series of operational steps can be performed on the computer or other programmable data processing equipment and performed on the computer. Instructions that generate the process to be performed to perform a computer or other programmable data processing equipment can also provide steps to perform the functions described at each block of the block diagram and at each stage of the flowchart. Is.

Together, each block or stage can represent a portion of a module, segment or code that contains one or more performable instructions to perform the specified logical function. It should also be noted that in some alternative embodiments it is possible for the features mentioned in the block or stage to occur out of order. For example, two blocks or stages shown in succession can actually be performed at substantially the same time, or the blocks or stages can sometimes be performed in reverse order by the corresponding function. ..

As described above, those skilled in the art to which the present invention belongs should understand that the present invention can be implemented in other concrete forms without changing its technical idea or essential features. Therefore, it should be understood that the examples described above are exemplary in all respects and are not limiting. The scope of the present invention is shown by the scope of claims described later rather than the detailed description, and the meaning and scope of the claims and all modified or modified forms derived from the equivalent concept thereof are also included in the scope of the present invention. It should be interpreted as being included.

100: Interactive content providing device 110: Memory 111: Animation DB
113: Interactive DB
115: Command word DB
120: Processor 121: Interactive module 123: Audio subtitle module 125: Image module 130: Input interface 140: Display unit 150: Communication interface 160: Camera unit

Claims (25)

In the interactive content providing method performed by the interactive content providing device,
As a plurality of animation units including the object information that identifies the object output to the video and the driving information that defines the movement of the object are played, the user uses interactive content in which the object is played while being moved by the driving information. Stage to provide to;
The stage of receiving the input of the user information reflected in the interactive content; and the user information is inserted into a part of the object information, and at least one of the plurality of animation units is reproduced to play the interactive content. An interactive content delivery method that includes the stage of playing content. The plurality of animation units are
Animation method played in a preset order; or during playback of the user's first input or a predetermined animation unit for a plurality of selection items that are selected to play any one of the plurality of animation units. By the user's second input to the object information output to, it is played back in a branch animation scheme; which is defined to play the next animation unit based on each input.
The method is
When the animation unit corresponding to the branch animation method is reproduced, the step of sensing the first input for the output of the plurality of selection items or sensing the second input for the object information output during the reproduction; The interactive content providing method according to claim 1, further comprising the step of reproducing the first input or the animation unit corresponding to the second input. The method is
It further includes a step of storing the user input during the play of the interactive content; and a step of generating a second content in which the animation unit is played by the sequence corresponding to the input and the user information when the play is finished. , The method for providing interactive content according to claim 2. The interactive content is
The first aspect of claim 1, further comprising object reaction animation information defining an action that occurs in the object when an input to the object is sensed while the interactive content is being played separately from the plurality of animation units. How to provide interactive content. The method is
The stage of specifying the type of event for the input information input by the user, the information for the input time and the input value, and generating the play information mapping the specified information; and the type of event included in the play information. The method for providing interactive content according to claim 1, further comprising a step of generating a second content which is an image in which the interactive content is reproduced based on information of an input time and an input value. The method is
The step further includes providing the user with the object information and the driving information constituting the animation unit, and receiving input of the object information, the driving information, or the user's editing information for the reproduction order of the animation unit.
The stage of playing the interactive content is
The method for providing interactive content according to claim 1, further comprising a step of playing interactive content including information for an animation unit to which the input editing information is reflected. The animation unit is
Further including the audio information output by the video and the subtitle information output corresponding to the audio information in the video.
The voice information is
Includes audio files with a playback time less than or equal to the preset time
The method is
The interactive content providing method according to claim 1, further comprising a step of outputting the subtitle information corresponding to the audio information to the video when the audio information is reproduced in the video. The method is
When the input to the subtitle output in the video is detected, the stage of receiving the input of the user voice corresponding to the subtitle information;
The stage of inserting the user's voice into the voice information corresponding to the sensed subtitle information; and the animation unit in which the voice information into which the user's voice is inserted is output corresponding to the subtitle information when the play is completed. The interactive content providing method according to claim 7, further comprising a step of generating a second content to be reproduced. When the input to the subtitle output in the video is detected, the stage of receiving the input of the user voice corresponding to the subtitle information is
A step of outputting a first button for searching for a subtitle output before the subtitle information and a second button for searching for a subtitle output after the subtitle information;
The step of outputting the previous subtitle or the next subtitle based on the user input for the first button or the second button; and the subtitle information output based on the user input for the first button or the second button. The interactive content providing method according to claim 8, which comprises the step of receiving the input of the user voice corresponding to the above. When the input to the subtitle output in the video is detected, the stage of receiving the input of the user voice corresponding to the subtitle information is
The interactive content providing method according to claim 9, further comprising a step of imposing a charge on the user when the user's voice is input for a time longer than a preset time. The method is
When the user information includes the user's face image, the step of standardizing the face image and substituting the standardized face image into the object information corresponding to the character's face among the object information is further included. The method for providing interactive content according to claim 1. The substituting step is
When a face image is input from the user, a predetermined artificial intelligence model trained to output an image in which the face image is deformed to form a predetermined angle, morphology, or facial expression is utilized. The step of standardizing the facial image to form a predetermined angle, morphology, or facial expression; and the step of identifying the feature points of the standardized facial image as vertex;
The stage of forming a three-dimensional model composed of a mesh structure composed by connecting the vertices;
The interactive content providing method according to claim 11, further comprising a step of substituting the three-dimensional model into the object information corresponding to the face of the character among the object information. The stage of playing the interactive content is
The interactive content providing method according to claim 12, further comprising a step of moving the vertices constituting the three-dimensional model by the driving information to realize a movement or a facial expression. The stage of embodying the movement or facial expression is
The stage of identifying the first vertex where the movement starts and the second vertex where the movement ends among the vertices constituting the 3D model; and the vertices constituting the 3D model move based on the driving information. 13. The method of providing interactive content according to claim 13, comprising a step of controlling the first vertex to move toward the second vertex so as to implement movement and expression. The stage of embodying the movement or facial expression is
At the stage of specifying the first vertex at which the movement starts and the second vertex at which the movement ends among the vertices constituting the three-dimensional model, the first vertex includes a plurality of vertices, and the plurality of vertices are It is concatenated with at least one vertex that is different from each other, and the second vertex contains the same number of vertices as the plurality;
A step of determining a surface that connects the faces constituting the three-dimensional model at the shortest distance until each of the plurality of second vertices specified as the end point of the movement of each of the plurality of first vertices is reached. ;
A step of identifying n points (n is a natural number of 1 or more) along a predetermined line from the first vertex to the second vertex based on the surface; and when the movement is realized, the first 13. The interactive content providing method according to claim 13, comprising a step of controlling the vertices to move toward the second vertex through the n points. The stage of substituting the standardized face image into the object information corresponding to the character's face in the object information is
The interactive content providing method according to claim 11, further comprising a step of imposing a charge on the user when receiving a selection of substituting a face image for an additional character in addition to the basic character in the object information. The method is
When the user information includes the user's face image, the step of analyzing the face image based on a predetermined artificial intelligence algorithm to extract the first tag information indicating the attribute of the face image; and the character's face. Further, a step of comparing the second tag information mapped to the object information corresponding to the above with the first tag information and substituting the face image into the character having the most corresponding first and second tag information. Including,
The second tag information is
The interactive content provision according to claim 11, which is attribute information that is mapped to object information corresponding to the character's face and has already been saved, or is attribute information extracted from the character's image based on a predetermined artificial intelligence algorithm. Method. The method is
A claim further comprising a step of correcting the tone of the face image when the user information includes the face image of the user; and a step of applying a face filter that deforms the face image according to a predetermined rule. 11. The method for providing interactive content according to 11. The method is
If the user information includes the user's face image, and if the user information deviates from the standard of the face information already set based on the predetermined artificial intelligence algorithm, the step of requesting the input of another face image is further included. , The method for providing interactive content according to claim 11. The method is
It further includes a step of outputting the object information by the coordinate information corresponding to the standard by the user's selection of the standard based on the coordinate information already set for each standard in which the position where the object information should be output is different. , The method for providing interactive content according to claim 1. The method is
At the stage of driving the camera unit of the interactive content providing device and receiving the input of the peripheral image of the surrounding environment;
Based on the coordinate information in which the position where the object information should be output is already set in the display unit of the interactive content providing device, the object information is arranged on the upper part of the layer of the peripheral image and output to the display unit. Or, based on a function designed so that the object information is mapped to one of the positions of the peripheral image, the object information is arranged on the upper part of the layer of the peripheral image and placed on the display unit. The interactive content providing method according to claim 1, further comprising an output step. The method is
Using a predetermined artificial intelligence model learned to discriminate the class of the input object information and map the drive information corresponding to the class, the first object information is input to the artificial intelligence model, and the above-mentioned The first aspect of claim 1, comprising a step of mapping the first drive information determined by the artificial intelligence model; and a step of driving the first drive information when the user input to the object corresponding to the first object information is sensed. How to provide interactive content. The object information is
It further includes information about a user model that reflects the character's face or the user's face and a second object that can be represented with the user model based on the relative distance to the user model.
The stage of playing the interactive content is
When the user model is moved by the driving information, the interactive content is set so that the relative distance between one or more vertices included in the user model and the second object is kept constant. The interactive content providing method according to claim 1, which includes a stage of playing. In the interactive content providing device
An input interface that senses user input; a display unit that outputs video; one or more memories that store instruction words that perform a predetermined operation; and concatenated so as to be able to operate with the one or more memories. Includes one or more processors configured to execute the instruction word
The one or more processors mentioned above
As a plurality of animation units including the object information that identifies the object output to the video and the driving information that defines the movement of the object are played, the interactive content that the object is played while moving by the driving information is used. Actions to be provided to the person;
The operation of receiving the input of the user information reflected in the interactive content; and the user information is inserted into a part of the object information, and at least one of the plurality of animation units is reproduced to play the interactive content. An interactive content providing device that performs the action of playing content. A computer-readable recording medium in which a computer program including an instruction word that enables a processor to perform the method according to claim 1 is recorded.

Images