KR102643211B1 - Method and apparatus for creating scenes with recommended assets - Google Patents

Abstract

This specification provides a method for a recommendation device to generate a scene included in content, including the steps of receiving a sentence from a user to recommend an asset for the scene; Based on the sentence, generating keywords for recommending the asset; 1) History information on content used by the user through the platform, 2) Information on components of content created by the user, and 3) User characteristic data based on information on content currently being created by the user generating a; generating a list of assets that can be recommended to the user based on the keyword and the user characteristic data; generating a list of the keywords; As, the list of keywords includes attribute values for each keyword; matching the keywords and the assets according to linguistic similarity, based on the list of keywords and the list of assets; and predicting an attribute value of a corresponding asset based on the attribute value of the matched keyword. may include.

Description

Method for creating a scene using recommended assets and device for the same {METHOD AND APPARATUS FOR CREATING SCENES WITH RECOMMENDED ASSETS}

This specification relates to a method for creating a scene using recommended assets in order to analyze user interests and provide customized content.

With the development of the Internet and e-commerce, large amounts of data are generated, and users are accessing numerous contents on various platforms. In order to quickly provide users with the content they want, many companies are developing artificial intelligence-based recommendation systems, and with the development of artificial intelligence and big data technology, recommendation systems using this are being provided.

However, existing recommendation systems are suitable for recommending simple elements or content, but have limitations in recommending complex content consisting of one or more elements and templates.

To overcome these limitations, the recommendation system must utilize artificial intelligence learning algorithms and big data analysis technology to understand and analyze users' preferences and behavior patterns. Additionally, the ability to dynamically modify content to reflect additional user requirements is required. This allows users to directly adjust content or transform it themselves as needed, resulting in an enhanced user experience.

This artificial intelligence-based recommendation system can be expected to contribute to enhancing interaction between users and companies and providing better user experience and content value.

1. Japanese Patent Publication No. 2013-182427 (2013.09.12.) 2. Registered Patent Publication No. 10-2525315 (2023.04.25.)

The purpose of this specification is to extract necessary elements by analyzing sentences entered by the user and predict the attribute value of each element through artificial intelligence.

Additionally, the purpose of this specification is to provide specialized elements and attribute values of specialized elements to individual users by utilizing user experience.

Additionally, the purpose of this specification is to complete content by creating a scene that matches the user's command using specialized elements and attribute values.

The technical problems to be achieved by this specification are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clear to those skilled in the art from the detailed description of the specification below. It will be understandable.

One aspect of the present specification provides a method for a recommendation device to generate a scene included in content, comprising: receiving a sentence from a user for recommending an asset for the scene; Based on the sentence, generating keywords for recommending the asset; 1) History information on content used by the user through the platform, 2) Information on components of content created by the user, and 3) User characteristic data based on information on content currently being created by the user generating a; generating a list of assets that can be recommended to the user based on the keyword and the user characteristic data; generating a list of the keywords; As, the list of keywords includes attribute values for each keyword; matching the keywords and the assets according to linguistic similarity, based on the list of keywords and the list of assets; and predicting an attribute value of a corresponding asset based on the attribute value of the matched keyword. may include.

Additionally, the attribute value for each keyword may include the position, size, and order values of the keyword.

Additionally, the attribute values of the predicted asset may include coordinate values, depth, horizontal length, and vertical length of the asset.

Additionally, based on the list of assets, adding the asset to a page for creating the scene; and setting the added asset to an attribute value of the predicted asset; may include.

In addition, the step of generating the user characteristic data is to acquire an image of the asset based on the history information of the content, information on the components of the content created by the user, and information about the content currently being created by the user. steps; generating an image caption based on the image of the asset; generating text related to the image of the asset based on the image caption; extracting the mood of the text based on the text; extracting a color based on the image of the asset; extracting the atmosphere of the color based on the extracted color; extracting asset characteristics related to the image of the asset based on the mood of the text, the color, and the mood of the color; and generating the user characteristic data based on the asset characteristic. may include.

In addition, based on the history information of the content, information on the components of the content created by the user, and information on the content currently being created by the user, extracting language mood and obtaining a thumbnail image step; extracting a color of the thumbnail image and an atmosphere of the color based on the thumbnail image; extracting asset characteristics included in content related to the thumbnail; extracting content characteristics based on the language mood, the color of the image, the color mood, and asset characteristics included in content related to the thumbnail; and generating the user characteristic data based on the content characteristics.

In addition, the step of generating the keyword includes converting the sentence into a story-type text through a language model; Converting the story-type text into a plot including 1) background, 2) main character, and 3) main elements through the language model; converting the story into sentence data associated with language-based data included in the information related to the asset through the language model; and generating the keyword based on the sentence data; may include.

Another aspect of the present specification is a recommendation device for generating a scene included in content, comprising: a communication unit; Memory containing language model; and a processor for functionally controlling the communication unit and the memory; Includes, wherein the processor receives a sentence for recommending an asset for the scene from the user, and generates keywords for recommending the asset based on the sentence, 1) the user receives a sentence through the platform, Based on history information of used content, 2) information on components of content created by the user, and 3) information on content currently being created by the user, user characteristic data is generated, and the keyword and the user Based on the characteristic data, generate a list of the assets that can be recommended to the user, generate a list of keywords, the list of keywords includes attribute values for each keyword, and the list of keywords and the asset Based on the list, the keyword and the asset can be matched according to linguistic similarity, and the attribute value of the corresponding asset can be predicted based on the attribute value of the matched keyword.

According to an embodiment of the present specification, through artificial intelligence, the sentence entered by the user can be analyzed to extract necessary elements and predict the attribute value of each element.

Additionally, according to an embodiment of the present specification, it is possible to provide specialized elements and attribute values of specialized elements to individual users by utilizing user experience.

Additionally, according to an embodiment of the present specification, content can be completed by creating a scene that matches a user command using specialized elements and attribute values.

The effects that can be obtained in this specification are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. .

1 is a block diagram for explaining an electronic device related to this specification.
Figure 2 is an embodiment to which this specification can be applied.
Figure 3 is an example of page creation to which this specification can be applied.
Figure 4 is an example of a controller 400 that can be applied to this specification.
Figure 5 is an example of a page list to which this specification can be applied.
Figure 6 is an example of elements to which this specification can be applied.
Figure 7 is an example of event detection of a convergence type manufacturing device to which the present specification can be applied.
Figure 8 is an example of a result execution method to which this specification can be applied.
Figure 9 is an example of element management to which this specification can be applied.
10 and 11 are examples of element uploads to which this specification can be applied.
Figure 12 is a block diagram of an AI device according to an embodiment of the present specification.
Figure 13 is an example of a recommendation method pipeline to which this specification can be applied.
Figure 14 is an example of visual coding to which this specification can be applied.
Figure 15 is an example of a recommendation device to which this specification can be applied.
Figure 16 is an example of sentence input to which this specification can be applied.
Figure 17 is an example of a recommended method to which this specification can be applied.
Figure 18 is an example of user characteristic data generation to which this specification can be applied.
19 and 20 illustrate the collection of characteristic data that may be included in a database to which this disclosure may be applied.
Figures 21 and 22 are examples of information related to content to which this specification can be applied.
Figure 23 is an example of asset characteristic extraction to which this specification can be applied.
Figure 24 is an example of content/template characteristic extraction to which this specification can be applied.
Figure 25 is another embodiment of a recommendation device to which this specification can be applied.
Figure 26 is an example of a platform to which this specification can be applied.
Figure 27 is an example of a convergence type manufacturing device to which this specification can be applied.
Figure 28 is an example of a scene creation method to which this specification can be applied.
Figure 29 is an example of attribute value prediction of an asset to which this specification can be applied.
Figure 30 is an example of scene creation to which this specification can be applied.
Figure 31 illustrates another embodiment to which this specification can be applied.
The accompanying drawings, which are included as part of the detailed description to aid understanding of the present specification, provide embodiments of the present specification and explain technical features of the present specification together with the detailed description.

Hereinafter, embodiments disclosed in the present specification will be described in detail with reference to the attached drawings. However, identical or similar components will be assigned the same reference numbers regardless of reference numerals, and duplicate descriptions thereof will be omitted. The suffixes “module” and “part” for components used in the following description are given or used interchangeably only for the ease of preparing the specification, and do not have distinct meanings or roles in themselves. Additionally, in describing the embodiments disclosed in this specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in this specification, the detailed descriptions will be omitted. In addition, the attached drawings are only for easy understanding of the embodiments disclosed in this specification, and the technical idea disclosed in this specification is not limited by the attached drawings, and all changes included in the spirit and technical scope of this specification are not limited. , should be understood to include equivalents or substitutes.

Terms containing ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When a component is said to be "connected" or "connected" to another component, it is understood that it may be directly connected to or connected to the other component, but that other components may exist in between. It should be. On the other hand, when it is mentioned that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between.

Singular expressions include plural expressions unless the context clearly dictates otherwise.

In this application, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

1 is a block diagram for explaining an electronic device related to this specification.

The electronic device 100 includes a wireless communication unit 110, an input unit 120, a sensing unit 140, an output unit 150, an interface unit 160, a memory 170, a control unit 180, and a power supply unit 190. ), etc. may be included. The components shown in FIG. 1 are not essential for implementing an electronic device, so the electronic device described in this specification may have more or fewer components than the components listed above.

More specifically, among the above components, the wireless communication unit 110 is used between the electronic device 100 and the wireless communication system, between the electronic device 100 and another electronic device 100, or between the electronic device 100 and an external server. It may include one or more modules that enable wireless communication between the devices. Additionally, the wireless communication unit 110 may include one or more modules that connect the electronic device 100 to one or more networks.

This wireless communication unit 110 may include at least one of a broadcast reception module 111, a mobile communication module 112, a wireless Internet module 113, a short-range communication module 114, and a location information module 115. .

The input unit 120 includes a camera 121 or an image input unit for inputting an image signal, a microphone 122 or an audio input unit for inputting an audio signal, and a user input unit 123 for receiving information from a user, for example. , touch keys, push keys (mechanical keys, etc.). Voice data or image data collected by the input unit 120 may be analyzed and processed as a user's control command.

The sensing unit 140 may include one or more sensors for sensing at least one of information within the electronic device, information on the surrounding environment surrounding the electronic device, and user information. For example, the sensing unit 140 includes a proximity sensor (141), an illumination sensor (142), a touch sensor, an acceleration sensor, a magnetic sensor, and a gravity sensor. Sensor (G-sensor), gyroscope sensor, motion sensor, RGB sensor, IR sensor (infrared sensor), fingerprint scan sensor, ultrasonic sensor , optical sensors (e.g., cameras (see 121)), microphones (see 122), battery gauges, environmental sensors (e.g., barometers, hygrometers, thermometers, radiation detection sensors, It may include at least one of a heat detection sensor, a gas detection sensor, etc.) and a chemical sensor (e.g., an electronic nose, a healthcare sensor, a biometric sensor, etc.). Meanwhile, the electronic device disclosed in this specification can utilize information sensed by at least two of these sensors by combining them.

The output unit 150 is for generating output related to vision, hearing, or tactile sense, and includes at least one of a display unit 151, an audio output unit 152, a haptip module 153, and an optical output unit 154. can do. The display unit 151 can implement a touch screen by forming a layered structure or being integrated with the touch sensor. This touch screen functions as a user input unit 123 that provides an input interface between the electronic device 100 and the user, and can simultaneously provide an output interface between the electronic device 100 and the user.

The interface unit 160 serves as a passageway for various types of external devices connected to the electronic device 100. This interface unit 160 connects devices equipped with a wired/wireless headset port, an external charger port, a wired/wireless data port, a memory card port, and an identification module. It may include at least one of a port, an audio input/output (I/O) port, a video input/output (I/O) port, and an earphone port. In response to the external device being connected to the interface unit 160, the electronic device 100 may perform appropriate control related to the connected external device.

Additionally, the memory 170 stores data supporting various functions of the electronic device 100. The memory 170 may store a plurality of application programs (application programs) running on the electronic device 100, data for operating the electronic device 100, and commands. At least some of these applications may be downloaded from an external server via wireless communication. Additionally, at least some of these applications may be present on the electronic device 100 from the time of shipment for basic functions of the electronic device 100 (e.g., incoming and outgoing calls, receiving and sending functions). Meanwhile, the application program may be stored in the memory 170, installed on the electronic device 100, and driven by the control unit 180 to perform an operation (or function) of the electronic device.

In addition to operations related to the application program, the control unit 180 typically controls the overall operation of the electronic device 100. The control unit 180 can provide or process appropriate information or functions to the user by processing signals, data, information, etc. input or output through the components discussed above, or by running an application program stored in the memory 170.

Additionally, the control unit 180 may control at least some of the components examined with FIG. 1 in order to run an application program stored in the memory 170. Furthermore, the control unit 180 may operate at least two of the components included in the electronic device 100 in combination with each other in order to run the application program.

The power supply unit 190 receives external power and internal power under the control of the control unit 180 and supplies power to each component included in the electronic device 100. This power supply unit 190 includes a battery, and the battery may be a built-in battery or a replaceable battery.

At least some of the components may cooperate with each other to implement operation, control, or a control method of an electronic device according to various embodiments described below. Additionally, the operation, control, or control method of the electronic device may be implemented on the electronic device by running at least one application program stored in the memory 170.

In this specification, the electronic device 100 may include a recommendation device, a terminal, a visual coding device, and a convergence type manufacturing device.

Figure 2 is an embodiment to which this specification can be applied.

Referring to Figure 2, a user can communicate with a convergence type manufacturing device through a terminal. For example, a terminal can be connected to a convergence production device through the WEB without a separate application, and users can simultaneously produce 2D and 3D content through the terminal.

The recommendation device includes an AI device 20, which will be described later, and is connected to the user's terminal and a convergence production device, and analyzes the user's tendency using meta information generated from the terminal, platform, and convergence production device, Using the analysis results, you can recommend content, elements, and templates that suit the user.

The recommendation device assigns weight to the content activities used by the user through the platform and analyzes the information of the content created through the convergence production device to extract characteristics and use it as learning data for the recommendation algorithm.

Additionally, the recommendation device can extract real-time characteristics by analyzing information about the content the user is currently working on through a convergence production device.

In addition, the recommender device can generate JSON sentence data using a large-scale language model framework based on the sentence written by the user in the input window through the terminal. For example, a recommender device can extract various keywords by processing sentences entered by the user using a large-scale language model framework.

Using this, the recommendation device can recommend content, elements (Assets), and templates to the convergence production device, and recommend templates and content to the platform. Additionally, the recommendation device can analyze the uploaded file, generate JSON sentence data through a language model framework, and store it in asset meta information.

The convergence type production device receives content creation commands from the terminal through the WEB (S2010). For example, content may include 2D and/or 3D objects.

The convergence production device creates a page to create content (S2020). For example, a page includes pages that can be expressed in 2D or 3D form, and the convergence production device can configure the screen by adding and placing predefined elements (Assets) or templates on each page. In more detail, an event may be registered in the added element, allowing interaction with the content user to be added. This allows users to create immersive and creative content.

Figure 3 is an example of page creation to which this specification can be applied.

Referring to Figure 3, the user can receive a page display screen 300 from the convergence type manufacturing device through the terminal. For example, one piece of content may include one or more pages. In addition, the user can change the shape of the page in 2D or 3D through the layout selection window 310 that can appear on the page display screen 300, and create a virtual space with special functions according to the needs of content users, such as AR mode. It can be added separately. The convergence type production device can register a separate controller 400 depending on the type of page. Additionally, the user can change the size and ratio of the page through the layout selection window 310.

Referring again to FIG. 2, the convergence type manufacturing device registers the controller 400 based on the page (S2030). For example, the convergence production device can register a controller 400 that can control and interact with elements according to the type of page.

Figure 4 is an example of a controller 400 that can be applied to this specification.

Referring to FIG. 4, when the user selects an element through the terminal, the properties appropriate for the element are displayed in the properties window 410, the convergence type production device displays the registered controller 400, and the user selects the controller ( 400), properties can be easily edited using a mouse or touch. For example, a user can enter an accurate value in the property window 410 to modify the property values of an element in detail. The user can perform additional connected functions by adding a tab to the properties window 410. For example, additional linked functions may include the source of the element's creator or media playback information.

Referring again to FIG. 2, the convergence type production device arranges elements on the page (S2040). For example, the convergence type production device may place predefined elements/templates and/or additionally uploaded elements depending on the page type.

Figure 5 is an example of a page list to which this specification can be applied.

Referring to FIG. 5, one content consists of a bundle of several pages (screens), and 2D or 3D screens can be selected according to the user's request.

For example, a page includes (1) page properties, (2) an event list, and (3) a resource list. More specifically, the event list contains information on events assigned to elements, and the resource list contains information on elements added to the page.

Figure 6 is an example of elements to which this specification can be applied.

Referring to FIG. 6, the convergence type production device can provide predefined elements/templates according to the page type to the user through the terminal. Additionally, users can upload and place additional elements.

Referring again to FIG. 2, the convergence type manufacturing device modifies the attribute value of the element (S2050).

For example, when the user selects an element placed on the page display screen 300, the convergence type production device displays the property window 410 corresponding to the element, and the user uses the controller 400 registered based on the page. ), you can modify attribute values through mouse, touch, numerical input, etc.

The convergence type manufacturing device registers an event corresponding to the element (S2060). For example, an event may include a set of “actions” and “results.” More specifically, actions can define conditions under which an event occurs. For example, “action” is a condition for performing a function and may include various types of events or calls, such as keyboard events, mouse/touch events, gesture events, area events, value events, and call events.

Additionally, results may include “function” and “object”.

In more detail, “functions” can define property changes and specific actions to be performed on the “target” that is the purpose of the function when an event is activated, and can define the basic properties of elements such as position, size, rotation, transparency, etc. It can include control functions for media elements such as hiding, playing, stopping, and pausing. Additionally, it may include a camera, GPS, accelerometer, etc. of the user terminal to utilize information from the external environment.

The convergence type production device registers the action, function, and/or object corresponding to the event (S2070). For example, a convergence type production device can register actions, functions, and/or objects based on the attribute values of elements.

When an event for an element is detected, the convergence production device executes the result based on the registered action (S2080).

Figure 7 is an example of event detection of a convergence type manufacturing device to which the present specification can be applied.

Referring to FIG. 7, the convergence type manufacturing device can identify events corresponding to elements, monitor the events, and detect the events. When an event is detected, the convergence production device can check the properties to execute a function corresponding to the event and execute the result based on the function and properties.

Figure 8 is an example of a result execution method to which this specification can be applied.

Referring to Figure 8, the convergence type production device can execute “results” simultaneously, and the lower “results” can be connected to the upper “results” and executed continuously. Additionally, there is no limit to the connection of “results” and the upper “results” can be executed simultaneously. It is also possible to perform continuous functions in the next step using the execution results of "results." Unlike the execution method that is simply driven by a single timeline, this method of executing results provides the user with information regarding the operation of elements and actual programming. It can provide the same environment as the technique and help learn the programming environment naturally.

Figure 9 is an example of element management to which this specification can be applied.

Referring to FIG. 9, 2D-shaped elements and 3D-shaped elements each have different properties, making it difficult for the convergence type manufacturing device to control the elements in the same manner. Therefore, the convergence production device first surrounds 2D-shaped elements and 3D-shaped elements through objects called "basic elements" and expands the "basic elements" into "usage elements" used in the authoring tool of the convergence production device. Manage elements so they can be used. For example, the functionality of a “Usage Element” may be configured to control and use the properties of a “Native Element”. In more detail, in Figure 9, “usable elements” are examples of images, videos, shapes, and 3D models, but if they are in a form that is advantageous for controlling and displaying the characteristics of 2D elements and 3D elements, they can be used as “usable elements”. can be displayed.

10 and 11 are examples of element uploads to which this specification can be applied.

Referring to Figures 10 and 11, the convergence type production device (or platform) loads the uploaded element file through a loader and registers the object of the loaded element file in a "use element" that can correspond to the loader. do. The convergence production device can place uploaded elements using registered “usable elements.”

Referring to FIG. 10, when the uploaded element file is an image file, the convergence type production device can load the image file through an image loader and register the object of the loaded image file as an image element. .

Referring to FIG. 11, when the uploaded element file is a 3D model file, the convergence type production device loads the 3D model file through a 3D loader, creates an object of the loaded 3D model file, and creates an animation. , by adding animation to the created object, the object can be registered as a 3D element.

For example, if the uploaded element file is a video file, the convergence production device can create an HTMLVideoElement, add it to the screen, and then register the object in the "Use Element" to control it.

One content can include multiple pages, and each page can include a “resourceManager” that manages elements and an “eventManager” that manages action events.

Elements placed on the page have separate depths, so the convergence production device can adjust the exposure order of elements on the screen by adjusting the depth according to the user's needs.

For example, when an element is placed in a convergence production device, it registers the element in the resource manager of the page and can be managed through changes in its properties, status, etc., registration, deletion, etc. Resource manager not only manages deployed elements, but also generates events for changes and can call connected functions according to resource changes based on the resource list.

When an element is added, the convergence production device adds the element to the management list to control the element through the controller registered on the page, so that the controller can recognize and manage the element as a controllable element. Elements registered in the controller can have their properties modified or controlled using the user's mouse, gesture, touch, or external controller.

Additionally, the Redo/Undo function may be necessary during the user's control of elements. To this end, the convergence production device records changes in the state of the element by saving the controller's changes in the page's History manager, and upon request, retrieves the element's properties from the list of changes kept in the History manager. You can configure the element state to be restored or updated again by updating the current element properties. Because History manager stores changes for each element, memory problems may occur. Therefore, to prevent this, the convergence type manufacturing device may limit the number of lists of changes stored depending on the situation.

Additionally, the convergence type production device can specify users to share content when creating content and perform the work simultaneously. For example, when content is initially created, the convergence type production device can create a unique channel corresponding to the content. The convergence production device can designate a shared user who can use a unique channel and add that user to the same channel when he or she accesses the content.

Users on the same channel can exchange changes in real time and synchronize authoring data in real time. For this purpose, communication can be done in real time via websocket or webrtc.

However, when a new user accesses unsaved content, the content may be synchronized with the previous version, causing a version difference. To prevent this, when a new user is added to a shared channel and accesses content for the first time, initial synchronization can be performed by performing an overall update on work changes through a specific user among existing users. After this initial synchronization is achieved, the convergence production device can solve the synchronization issue of unsaved content by sharing data about changes in real time.

Figure 12 is a block diagram of an AI device according to an embodiment of the present specification.

The AI device 20 may include an electronic device including an AI module capable of performing AI processing or a server including the AI module. Additionally, the AI device 20 may be included in at least a portion of the electronic device 100 shown in FIG. 1 and may be equipped to perform at least a portion of AI processing.

The AI device 20 may include an AI processor 21, memory 25, and/or a communication unit 27.

The AI device 20 is a computing device capable of learning a neural network, and may be implemented in various electronic devices such as servers, desktop PCs, laptop PCs, tablet PCs, etc.

The AI processor 21 can learn a neural network using a program stored in the memory 25. For example, the AI processor 21 can create a Recurrent Neural Networks model using Tensorflow on the memory 25, using data that can be collected from user terminals and convergence production devices, etc. , these artificial intelligence models can be trained.

For example, a trained artificial intelligence model can include a language model and create sub-models with the following tasks:

1. Content/template recommendation model: Recommend/generate customized content and templates to users using data analyzed from content usage information and content creation information and user characteristic model.

2. Element recommendation model: Recommend customized assets to content creators using user characteristic model and asset meta information

3. User characteristic analysis model: Analyze user characteristics through analysis of user learning interest, participation, immersion, user usage, production data, etc.

Figure 13 is an example of a recommendation method pipeline to which this specification can be applied.

Referring to FIG. 13, the artificial intelligence model 1300 can be learned using data collected from the platform and the convergence type manufacturing device.

For example, the platform can be provided to a terminal through the WEB, and can be connected to a convergence production device to provide a content production environment to users. Additionally, content created by other users can be displayed, and a search function can also be provided for this purpose. Additionally, the platform and convergence production device can provide users with templates that can be used in a content production environment, and can also provide a search function for this.

The learned artificial intelligence model can generate 1) a user characteristic analysis model, 2) a content/template recommendation model, and 3) an element recommendation model. The user characteristics output by the user characteristic analysis model can be used for learning the artificial intelligence model (1300).

In addition, the content/template recommendation model can receive sentences as input from the user, convert them into sentence data in JSON format, and provide automatically generated content and templates to the user through a platform and a convergence production device.

Additionally, the element recommendation model can recommend assets to users through a convergence production device using sentence data in JSON format.

Figure 14 is an example of visual coding to which this specification can be applied.

Referring to FIG. 14, a user can communicate with a visual coding device through a terminal. For example, a convergence type manufacturing device may include a visual coding device.

Additionally, the terminal can be connected to a visual coding device through the WEB without a separate application, and the user can create content through visual coding through the terminal.

The terminal creates a page for visual coding and places elements (asset) on the page (S1410). For example, the terminal may display a list of elements and/or a template containing the elements to the user. Users can select elements from the template to be placed on the page.

Based on the placed elements, the terminal sets a target element that is the target of visual coding (S1420). For example, a user can click to select a target element among elements placed on a page.

The terminal sets user behavior related to interaction with the user (S1430). For example, user action may be a condition for an event of a target element to occur.

Based on user behavior, the terminal sets a result related to the target element (S1440). For example, a result may mean an action performed in relation to a target element when the terminal receives input corresponding to a user action from the user. The result may include a “function” for controlling the size, position, and state value of the target element, an “operation” for calculating variables related to the target element, and a “function page” representing the movement of the page.

The terminal displays the result of the target element based on the user action input (S1450). For example, when a user clicks a target element once, the terminal may move the target element for 1 second or display a screen that has been moved to another page as a result.

Figure 15 is an example of a recommendation device to which this specification can be applied.

Referring to Figure 15, the recommendation device is connected to the user terminal and the convergence production device, learns from the collected data, and provides the predicted output value through the learned artificial intelligence model to the platform and the convergence production device. . For example, a convergence type production device may include an authoring tool for content production.

The recommendation device receives sentences from the user through a convergence production device and/or platform (S1510). For example, a user can input sentences about a specific topic or story into a convergence production device to create content. More specifically, the user can input the sentence “blue sky and grass with a smiling sun” into the convergence type production device. At this time, the input language may mainly be Korean, and the recommendation device may store the input sentence for subsequent processing.

Figure 16 is an example of sentence input to which this specification can be applied.

Referring to FIG. 16, the user can select a chat icon 1610 for entering a sentence on the content creation screen. The convergence type production device can activate the chat function when the chat icon 1610 is input. Through the chat window 1620 displayed through the terminal, the user can input a sentence to create content.

Again, referring to FIG. 15, the recommendation device converts the input sentence into English (S1520). If the input sentence is in a language other than English, the input sentence may be converted to English to ensure the accuracy of the language model included in the recommendation device.

For example, the sentence “Blue sky and grass with a smiling sun” can be converted into the English sentence “Blue sky and grass with a smile.”

The recommendation device converts the sentence into a story-type text through a language model (S1530). More specifically, the template of the language model can be set to command Write a synopsis about the "blue sky and grass with a smile" for the sentence "Blue sky and grass with a smile".

For example, "Blue sky and grass with a smile" means

In the enchanting world of "Blue Sky and Grass with a Smile," we are transported to a serene and picturesque landscape where the beauty of the blue sky and lush green grass becomes a catalyst for transformation. The story revolves around Emily, a young girl grappling with the hardships of life and burdened by sorrow. However, one fateful day, she stumbles upon a hidden meadow, bathed in the golden sunlight, where the sky is a vivid shade of blue and the grass dances with an infectious joy.

As Emily immerses herself in this magical haven, she starts to notice how the vibrant hues of the sky and grass affect her mood. The brilliant blue sky acts as a gentle reminder to look beyond her troubles and find hope, while the cheerful green grass inspires her to embrace the present moment and seek happiness. Slowly but surely, a smile starts to grace Emily's face as she discovers the power of perspective and the healing balm of nature.

With each visit to the meadow, Emily's connection to the blue sky and grass deepens. She begins to cultivate a profound sense of gratitude for the beauty surrounding her, finding solace and strength in the simplicity of life's wonders. The radiant smile that forms on her lips becomes a beacon of optimism, influencing those around her and spreading contagious joy.

Through this poignant journey, “Blue Sky and Grass with a Smile” highlights the transformative power of nature and the profound impact it can have on our emotional well-being. It reminds us to find solace in the smallest of moments, to appreciate the beauty that surrounds us, and to nurture a smile that can illuminate even the darkest of days.

It can be converted into a story-type text. This is to compensate for the fact that the sentences entered by the first user contain insufficient information for content creation, templates, and/or asset recommendation.

The recommendation device converts a story-type text into a plot containing the background, main character, and main elements through a language model (S1540). The recommendation device can use the language model again to analyze text in the form of a story, condense it into one scene including the background, main character, and main elements, and summarize it into a plot. This is to extract key elements of content, and the recommendation device can generate a plot by extracting key elements such as the background, main character, and main elements from the large amount of data included in the story-type text.

For this purpose, the template of the language model can be set to Summarize the plot in three lines, including the background, main characters, and main elements.

For example, the story-type text exemplified in S1530 described above is

In "Blue Sky and Grass with a Smile," Emily, a young girl burdened by sorrow, discovers a hidden meadow where the vibrant blue sky and dancing green grass become catalysts for transformation. Immersed in this magical haven, she learns to find hope, embrace the present, and cultivate gratitude. Through the healing power of nature, Emily's radiant smile becomes a symbol of optimism, inspiring those around her and reminding us to appreciate life's simple wonders.

It can be converted into a plot called.

The recommendation device converts the story into sentence data for content, template, and/or asset recommendation (S1550). If necessary, the recommendation device can translate the plot back into Korean. For example, the plot illustrated in S1540 above is

"In "With the Smile of the Sky and the Grass," Emily, a girl heavy with grief, discovers a hidden meadow where vibrant blue skies and dancing green grass are the starting point of change. As she immerses herself in this magical place, she finds hope and makes sense of the present. "You learn to accept and cultivate gratitude. Through the healing power of nature, Emily's radiant smile becomes a symbol of optimism, inspiring those around her and reminding us to recognize the simple beauty in life." It can be translated as

Additionally, the recommendation device can use a language model to convert the synopsis into sentence data in JSON format based on original words, English words, classification, zero classification, importance, and part of speech.

Table 1 below illustrates sentence data.

{
"original":"Blue sky and grass with a smiling sun"
"sentence": "In With the Smile of Sky and Grass, Emily, a girl heavy with sadness, discovers a hidden meadow where vibrant blue skies and dancing green grass are the starting point of change. Immersing herself in this magical place, she finds hope. "You learn to embrace the present and cultivate gratitude. Through the healing power of nature, Emily's radiant smile becomes a symbol of optimism, inspiring those around her and reminding us to recognize the simple beauty in life.",
"words": [
{
"text": "Sky and",
"english": "sky and",
"classification": "noun",
"english_classification": "noun",
"importance": 3,
"part_of_speech": "NNG"
},
{
"text": "Grass",
"english": "grass with",
"classification": "noun",
"english_classification": "noun",
"importance": 3,
"part_of_speech": "NNG"
},
{
"text": "With a smile",
"english": "smile and",
"classification": "noun",
"english_classification": "noun",
"importance": 3,
"part_of_speech": "NNG"
},
{
"text": "Together",
"english": "with a smile",
"classification": "adverb",
"english_classification": "adverb",
"importance": 2,
"part_of_speech": "MAG"
},
{
"text": "In sadness",
"english": "by sorrow",
"classification": "noun",
"english_classification": "noun",
"importance": 3,
"part_of_speech": "NNG"
},
{
"text": "heavy",
"english": "burdened",
"classification": "adjective",
"english_classification": "adjective",
"importance": 3,
"part_of_speech": "VA"
},
{
"text": "Geonyeoin",
"english": "girl",
"classification": "noun",
"english_classification": "noun",
"importance": 3,
"part_of_speech": "NNG"
},
{
"text": "Emily",
"english": "Emily,",
"classification": "noun",
"english_classification": "noun",
"importance": 3,
"part_of_speech": "NNP"
},
{
"text": "lively",
"english": "vibrant",
"classification": "adjective",
"english_classification": "adjective",
"importance": 3,
"part_of_speech": "VA"
},
{
"text": "blue",
"english": "blue",
"classification": "adjective",
"english_classification": "adjective",
"importance": 3,
"part_of_speech": "VA"
},
{
"text": "Sky and",
"english": "sky",
"classification": "noun",
"english_classification": "noun",
"importance": 3,
"part_of_speech": "NNG"
},
{
"text": "dancing",
"english": "dancing",
"classification": "verb",
"english_classification": "verb",
"importance": 3,
"part_of_speech": "VV+ETM"
},
{
"text": "blue",
"english": "green",
"classification": "adjective",
"english_classification": "adjective",
"importance": 3,
"part_of_speech": "VA"
},
{
"text": "Solution",
"english": "grass",
"classification": "noun",
"english_classification": "noun",
"importance": 3,
"part_of_speech": "NNG"
},
{
"text": "of change",
"english": "transformation",
"classification": "noun",
"english_classification": "noun",
"importance": 3,
"part_of_speech": "NNG"
},
{
"text": "Starting point",
"english": "catalysts for",
"classification": "noun",
"english_classification": "noun",
"importance": 3,
"part_of_speech": "NNG"
},
{
"text": "being",
"english": "become",
"classification": "verb",
"english_classification": "verb",
"importance": 3,
"part_of_speech": "VV+ETM"
},
{
"text": "hidden",
"english": "hidden",
"classification": "adjective",
"english_classification": "adjective",
"importance": 3,
"part_of_speech": "VA"
},
{
"text": "pasture",
"english": "meadow",
"classification": "noun",
"english_classification": "noun",
"importance": 3,
"part_of_speech": "NNG"
},
{
"text": "Discover.",
"english": "discovers",
"classification": "verb",
"english_classification": "verb",
"importance": 3,
"part_of_speech": "VV+EC"
},
{
"text": "this",
"english": "this",
"classification": "pronoun",
"english_classification": "pronoun",
"importance": 1,
"part_of_speech": "MM"
},
{
"text": "Magic",
"english": "magical",
"classification": "noun",
"english_classification": "noun",
"importance": 3,
"part_of_speech": "NNG"
},
{
"text": "same",
"english": "like",
"classification": "adjective",
"english_classification": "adjective",
"importance": 3,
"part_of_speech": "VA"
},
{
"text": "where",
"english": "place",
"classification": "noun",
"english_classification": "noun",
"importance": 2,
"part_of_speech": "NNG"
},
{
"text": "Absorbing,",
"english": "immersing",
"classification": "verb",
"english_classification": "verb",
"importance": 2,
"part_of_speech": "VV+EC"
},
{
"text": "she",
"english": "she",
"classification": "pronoun",
"english_classification": "pronoun",
"importance": 2,
"part_of_speech": "NP"
},
{
"text": "hope",
"english": "hope",
"classification": "noun",
"english_classification": "noun",
"importance": 3,
"part_of_speech": "NNG"
},
{
"text": "looking for",
"english": "finds",
"classification": "verb",
"english_classification": "verb",
"importance": 2,
"part_of_speech": "VV+EC"
},
{
"text": "current",
"english": "the present",
"classification": "noun",
"english_classification": "noun",
"importance": 2,
"part_of_speech": "NNG"
},
{
"text": "Accept",
"english": "accepting",
"classification": "verb",
"english_classification": "verb",
"importance": 2,
"part_of_speech": "VV+EC"
},
{
"text": "Thank you",
"english": "gratitude",
"classification": "noun",
"english_classification": "noun",
"importance": 2,
"part_of_speech": "NNG"
},
{
"text": "Raising",
"english": "cultivating",
"classification": "verb",
"english_classification": "verb",
"importance": 2,
"part_of_speech": "VV+ETM"
},
{
"text": "law",
"english": "the way",
"classification": "noun",
"english_classification": "noun",
"importance": 2,
"part_of_speech": "NNG"
},
{
"text": "Learn",
"english": "learn",
"classification": "verb",
"english_classification": "verb",
"importance": 2,
"part_of_speech": "VV+EC"
},
{
"text": "It works.",
"english": "become",
"classification": "verb",
"english_classification": "verb",
"importance": 2,
"part_of_speech": "VV+EC"
},
{
"text": "Natural",
"english": "nature's",
"classification": "noun",
"english_classification": "noun",
"importance": 3,
"part_of_speech": "NNG"
},
{
"text": "Healing power",
"english": "healing power",
"classification": "noun",
"english_classification": "noun",
"importance": 3,
"part_of_speech": "NNG"
},
{
"text": "via",
"english": "through",
"classification": "investigation",
"english_classification": "particle",
"importance": 1,
"part_of_speech": "JKM"
},
{
"text": "Emily's",
"english": "Emily's",
"classification": "noun",
"english_classification": "noun",
"importance": 2,
"part_of_speech": "NNG"
},
{
"text": "shining",
"english": "shining",
"classification": "adjective",
"english_classification": "adjective",
"importance": 3,
"part_of_speech": "VA"
},
{
"text": "Smile",
"english": "smile",
"classification": "noun",
"english_classification": "noun",
"importance": 2,
"part_of_speech": "NNG"
},
{
"text": "of optimism",
"english": "optimism's",
"classification": "noun",
"english_classification": "noun",
"importance": 3,
"part_of_speech": "NNG"
},
{
"text": "symbol",
"english": "symbol",
"classification": "noun",
"english_classification": "noun",
"importance": 2,
"part_of_speech": "NNG"
},
{
"text": "Become",
"english": "becomes",
"classification": "verb",
"english_classification": "verb",
"importance": 2,
"part_of_speech": "VV+EC"
},
{
"text": "he",
"english": "that",
"classification": "pronoun",
"english_classification": "pronoun",
"importance": 1,
"part_of_speech": "NP"
},
{
"text": "surrounding",
"english": "surrounding",
"classification": "noun",
"english_classification": "noun",
"importance": 2,
"part_of_speech": "NNG"
},
{
"text": "To the people",
"english": "to people",
"classification": "noun",
"english_classification": "noun",
"importance": 2,
"part_of_speech": "NNG+JKO"
},
{
"text": "inspiration",
"english": "inspiration",
"classification": "noun",
"english_classification": "noun",
"importance": 2,
"part_of_speech": "NNG"
},
{
"text": "Give,",
"english": "and",
"classification": "conjunction",
"english_classification": "conjunction",
"importance": 1,
"part_of_speech": "JC"
},
{
"text": "To us",
"english": "to us",
"classification": "pronoun",
"english_classification": "pronoun",
"importance": 2,
"part_of_speech": "NP+JKO"
},
{
"text": "of life",
"english": "life's",
"classification": "noun",
"english_classification": "noun",
"importance": 2,
"part_of_speech": "NNG"
},
{
"text": "simple",
"english": "simple",
"classification": "adjective",
"english_classification": "adjective",
"importance": 3,
"part_of_speech": "VA"
},
{
"text": "Beauty",
"english": "beauty",
"classification": "noun",
"english_classification": "noun",
"importance": 2,
"part_of_speech": "NNG"
},
{
"text": "To understand",
"english": "realize",
"classification": "verb",
"english_classification": "verb",
"importance": 2,
"part_of_speech": "VV+EC"
},
{
"text": "Remind",
"english": "remind",
"classification": "verb",
"english_classification": "verb",
"importance": 2,
"part_of_speech": "VV+EC"
},
{
"text": "I give it to you.",
"english": "us",
"classification": "pronoun",
"english_classification": "pronoun",
"importance": 1,
"part_of_speech": "NP"
}
]
}

Referring to Table 1, the recommendation device can convert the synopsis-type text for the first input sentence into JSON-format data. Data in JSON format can be used as data for content, template, and/or asset recommendations. This JSON format data may include sentence data information such as background, main character, and main elements. The recommendation device recommends content, templates, and/or assets based on the sentence data (S1560).

Figure 17 is an example of a recommended method to which this specification can be applied.

Referring to FIG. 17, the recommendation device can recommend assets, content, and/or templates using not only the above-described sentence data, but also keywords (e.g., “text”) of the sentence data, and user characteristic data. there is. For example, the keyword matching system may include all or part of the artificial intelligence model 1300.

The recommendation device consists of keywords and user characteristic data as input data of the keyword matching system, and can recommend templates/content to the platform and templates/content/assets to the convergence production device.

Figure 18 is an example of user characteristic data generation to which this specification can be applied.

Referring to FIG. 18, the recommendation device may extract asset characteristics and/or content/template characteristics from a database that manages data of a platform and/or a converged production device. The recommendation device may generate user characteristic data from extracted asset characteristics and/or content/template characteristics.

19 and 20 illustrate the collection of characteristic data that may be included in a database to which this disclosure may be applied.

Referring to FIG. 19, the platform can display (1910) the produced content through the convergence type production device. The platform may also display (1920) information about the content. Information about content can be generated based on users' history of using the content through the platform, and can include information such as number of plays, likes, dislikes, saves, progress, retention time, and followers. Information about such content may be included and managed in a database.

Referring to Figure 20, the platform can display (2010) content created by the user for each user. The recommendation device can collect information on components of content created by the user. Unlike information about content produced by other users, this may include information about elements used in production by the user through a convergence production device. For example, information on elements may include information such as the name, description, category, hashtag of the content, meta tags of assets used to create the content, composed actions, and background. Information on these elements can also be included and managed in the database.

Additionally, the recommendation device can collect information about the content the user is currently producing through a convergence production device. For example, information about the content currently being produced includes the name, description, category, hashtags of the content, meta tags of assets used to create the content, composed actions, background, etc. of the elements that make up the content being produced. It may contain information and may be included and managed in a database.

Figures 21 and 22 are examples of information related to content to which this specification can be applied.

Referring to FIG. 21, information about content created or being produced may include the size of the content and the type of content (eg, 2D/3D/VR/AR).

Referring to FIG. 22, information on elements included in content created or being produced can be managed through a separate ID value (uuid), and the database uses the ID value to provide name and tag information of assets. can be managed, and through this, asset characteristics/content characteristics can be extracted.

Figure 23 is an example of asset characteristic extraction to which this specification can be applied.

Referring to FIG. 23, a method in which a recommendation device extracts asset characteristics based on the image of the asset is illustrated in more detail. The image of the asset can be obtained from the database through the ID value of the asset described above. For example, the image of the asset is based on 1) history information of the content the user has used through the platform, 2) information on the components of the content created by the user, and 3) information about the content the user is currently creating. Thus, it can be obtained through a database.

The recommendation device generates an image caption based on the asset image (S2310). Automatic Image Captioning is a technology that combines computer vision and natural language processing. Through this, the recommendation device can automatically generate a description or caption for the image of the asset.

The recommendation device generates text related to the image of the asset based on the image caption (S2320). Image caption generation can perform two main tasks: image recognition and natural language processing. For example, a recommender device uses an image recognition algorithm to detect and recognize objects, scenes, attributes, etc. in the image of the asset. Afterwards, the recommender device can generate text for the image using a natural language processing model. The generated text can be used as a caption to describe the asset's image.

Additionally, the recommender device may process the text to enrich the information in the text. For example, a recommendation device can increase the amount of information in the text through a language model and extract keywords from the processed text.

The recommendation device extracts the mood of the text based on the text (S2330). For example, a recommendation device can extract mood from text or processed text through emotional analysis.

For example, sentiment analysis can refer to technology that recognizes and classifies emotions and emotions in data such as text, images, and voice. The recommendation device can perform text sentiment analysis using Natural Language Processing (NLP) technology. For this purpose, various approaches and algorithms such as emotion dictionary, machine learning, and deep learning can be used.

The recommendation device extracts the color of the asset based on the image of the asset (S2340).

The recommendation device extracts the color atmosphere based on the color of the extracted asset (S2350). For example, a recommendation device can extract the mood of a color using emotion analysis technology. For example, the color mood can include values representing light, medium, and dark, and for this, the ColorThief library can be used.

The recommendation device extracts asset characteristics related to the image of the asset based on the mood of the text, the color of the asset, and the mood of the color (S2360). For example, asset characteristics correspond to the ID value representing the asset, the mood of the text (e.g., positive, negative), the numerical value and color corresponding to the degree of the mood of the color (e.g., light, dark). It may include a numerical value that can be defined using a numerical value corresponding to the degree (for example, brightness). Using the numerical value defined in this way, the asset can be classified.

The characteristics of the extracted assets can be stored and managed through a database.

Figure 24 is an example of content/template characteristic extraction to which this specification can be applied.

Referring to Figure 24, the recommendation device extracts information related to content (S2410). The recommendation device can extract information related to content through a database. For example, information related to content may include information about the content described above in FIGS. 19 and 20, information about components of content created by the user, and/or information about content currently being produced. .

The recommendation device extracts the color and color atmosphere of the image based on the thumbnail image of the content (S2420). For example, a recommendation device can extract the color of a thumbnail image and extract the mood of the color using emotion analysis technology.

The recommendation device extracts the language atmosphere based on information related to the content (S2430). For example, a recommendation device may extract language mood from text-type information (e.g., content name, description, category, etc.) among content-related information using emotion analysis technology.

The recommendation device extracts asset characteristics included in the content (S2450). For example, the characteristics of these assets can be extracted through the operation of FIG. 23 described above, based on information on elements included in content related to the thumbnail.

The recommendation device extracts content characteristics based on the color of the image, color mood, language mood, and asset characteristics included in the content (S2460).

For example, content characteristics include the color of the thumbnail image of the content (e.g., brightness), the mood of the color (e.g., light, dark), the degree of language mood (e.g., positive, negative), and It may include a numerical value that can be defined using a numerical value corresponding to an asset characteristic included in the content. Using the numerical value defined in this way, the content can be classified.

If the content in question is a specific template, the characteristics of the template can also be extracted in the same way. The characteristics of extracted content and templates can be stored and managed through a database.

Again, referring to FIG. 18, the recommendation device may apply weights to the extracted asset and content/template characteristics to generate user characteristic data. For example, weights can be applied using user behavior data that can indicate the user's preference for the asset or content/template, such as the number of plays, likes, and dwell time.

Figure 25 is another embodiment of a recommendation device to which this specification can be applied.

Referring to FIG. 25, the recommendation device inputs keywords (e.g., text field) and user characteristic data included in the sentence data in FIG. 15 described above into the keyword matching system to generate a list of assets, content, or templates. You can.

The recommendation device generates keywords for creating a recommended list of assets, content, or templates based on sentences input from the user (S2510). For example, the recommendation device receives a sentence for recommending an asset, content, or template from the user through the terminal, and generates a keyword for recommending the asset, content, or template based on the operation of FIG. 15. can do.

Based on keywords and user characteristic data, a list of assets, content, and/or templates that can be recommended to the user is generated (S2520).

For example, the recommendation device generates user characteristic data based on history information of content used by the user through the platform, information on components of content created by the user, and information on content currently being created by the user. can do. Additionally, keywords may be associated with language-based data included in information related to assets, content, and/or templates. Based on this correlation, the recommendation device may generate a list of assets, content, and/or templates that can be recommended to the user based on preferences based on user characteristic data.

Table 2 below illustrates a list of assets, content and/or templates that can be recommended to users.

{
"original": "Blue sky and grass with a smiling sun",
"sentence": "In With the Smile of Sky and Grass, Emily, a girl heavy with sadness, discovers a hidden meadow where vibrant blue skies and dancing green grass are the starting point of change. Immersing herself in this magical place, she finds hope. "You learn to embrace the present and cultivate gratitude. Through the healing power of nature, Emily's radiant smile becomes a symbol of optimism, inspiring those around her and reminding us to recognize the simple beauty in life.",
"contentList": [
"a7e884bb-3d4c-46e2-9cf9-640460eb44d0",
"a7e884bb-3d4c-46e2-9cf9-640460eb44d0",
"a7e884bb-3d4c-46e2-9cf9-640460eb44d0",
"a7e884bb-3d4c-46e2-9cf9-640460eb44d0",
...
],
"TemplateList": [
"a7e884bb-3d4c-46e2-9cf9-640460eb44d0",
"a7e884bb-3d4c-46e2-9cf9-640460eb44d0",
"a7e884bb-3d4c-46e2-9cf9-640460eb44d0",
"a7e884bb-3d4c-46e2-9cf9-640460eb44d0",
...
],
"assetList": [
{
"assetIdx": "0b70763b-4e22-43bb-a163-68c80f9b9e37",
"metaData": {
"words": [
{
"text": "Sky and",
"english": "sky and",
"classification": "noun",
"english_classification": "noun",
"importance": 3,
"part_of_speech": "NNG"
},
{
"text": "Grass",
"english": "grass with",
"classification": "noun",
"english_classification": "noun",
"importance": 3,
"part_of_speech": "NNG"
},
{
"text": "With a smile",
"english": "smile and",
"classification": "noun",
"english_classification": "noun",
"importance": 3,
"part_of_speech": "NNG"
}, ...
]
}
},
{
"assetIdx": "0b70763b-4e22-43bb-a163-68c80f9b9e37",
"metaData": {
"words": [
{
"text": "dancing",
"english": "dancing",
"classification": "verb",
"english_classification": "verb",
"importance": 3,
"part_of_speech": "VV+ETM"
},
{
"text": "blue",
"english": "green",
"classification": "adjective",
"english_classification": "adjective",
"importance": 3,
"part_of_speech": "VA"
},
{
"text": "Solution",
"english": "grass",
"classification": "noun",
"english_classification": "noun",
"importance": 3,
"part_of_speech": "NNG"
}, ...
]
}, ...
}
]
}

Referring to Table 2, the list that can be recommended to the user may include a sentence input by the user, a story based on the sentence, a content list, a template list, and/or an asset list, and the asset list is the metadata of the asset. may include. Figure 26 is an example of a platform to which this specification can be applied.

Referring to FIG. 26, the user can search for desired content through a sentence in the search box 2610 on the content main screen displayed through the platform, and the recommendation device provides content based on the keyword and user characteristic data of the input sentence. It can be recommended and displayed (2620). Asset and template recommendations can also be performed in the same way.

Figure 27 is an example of a convergence type manufacturing device to which this specification can be applied.

Referring to FIG. 27, the user can receive a content creation screen from the convergence type production device through the terminal. The user can input (2710) a sentence for content creation on the content creation screen. The recommendation device can recommend and display assets based on the keywords of the input sentence and user characteristic data (2720), and recommended content/templates can be displayed (2740) after the canvas (2730) where the user is currently located. .

Figure 28 is an example of a scene creation method to which this specification can be applied.

Referring to FIG. 28, the recommendation device can automatically generate a scene included in the content using the list of assets, content, and/or templates that can be recommended to the user in Table 2 described above.

The recommendation device generates a list of keywords based on the story (S2810). For example, keywords may include keywords from Table 1 described above (eg, “text”). The recommendation device can identify the intent and components of the sentence by analyzing the “sentence” item through an artificial intelligence model.

Table 3 below illustrates a list of keywords.

{
"original": "Blue sky and grass with a smiling sun",
"sentence": "In With the Smile of Sky and Grass, Emily, a girl heavy with sadness, discovers a hidden meadow where vibrant blue skies and dancing green grass are the starting point of change. Immersing herself in this magical place, she finds hope. "You learn to embrace the present and cultivate gratitude. Through the healing power of nature, Emily's radiant smile becomes a symbol of optimism, inspiring those around her and reminding us to recognize the simple beauty in life.",
"sentenceKeywordList": [
{
"keyword": "sky",
"position": "above",
"size": "large",
"layer": "behind"
},
{
"keyword": "grass",
"position": "below",
"size": "medium",
"layer": "front"
},
...
]
}

Referring to Table 3, the recommendation device uses a large-scale language model to extract keywords from “sentence” and attribute values (e.g., position, size, layer) for each keyword. By predicting, a list of keywords can be created. In more detail, the recommendation device can extract keywords from “sentence”, predict attribute values for each keyword, and create “sentenceKeywordList”.

For example, the recommendation device linguistically analyzes “sentence,” which is a modified version of the input user’s sentence “blue sky and grass with a smiling sun,” and uses the words “sky” and “grass.” Keywords can be extracted. The recommendation device can analyze the context of "sentence", predict attribute values such as "position", "size", and "layer" for each keyword, and record them in "sentenceKeywordList". there is.

For example, positions can be categorized as “top” and “bottom,” sizes can be categorized as “large,” “medium,” and “small,” and the order can be “front,” “back,” and “middle.” "It can be divided into:

This list of keywords can be combined with the recommended asset list to generate data in JSON format.

Table 4 below illustrates the form of data combining a list of keywords and a list of recommended assets.

{
"original": "...",
"sentence": "...",
"sentenceKeywordList": [
{
"keyword": "...",
"position": "...",
"size": "...",
"layer": "..."
},
...
],
"recommendAssetList: [
{
"assetIdx": "...",
"metaData": {
"words": [
{
"text": "...",
"english": "...",
"classification": "...",
"english_classification": "...",
"importance": "...",
"part_of_speech": "..."
},
...
],
}
},
...
]
}

Referring to Table 4, the "assetList" item in Table 2 can be managed by being included in the recommendAssetList item. The recommendation device matches keywords and assets based on the list of keywords and the recommended asset list (S2820). For example, a recommender device can use a language model to compare “keyword” and “text” of the “metadata” item of “recommendAssetList” to match keywords and assets based on linguistic similarity above a certain value. .

Table 5 below illustrates the matched results.

{
"original": "...",
"sentence": "...",
"processedData": [
{
"matchingKeywordData": {
"keyword": "...",
"position": "...",
"size": "...",
"layer": "..."
},
"matchingAssetList": [
{
"assetIdx": "..."
...
},
...
]
},
...
]
}

Referring to Table 5, the recommendation device can manage the matched keywords and assets as pairs in the “processedData” array. The recommendation device predicts the attribute value of the corresponding asset based on the attribute value of the matched keyword ( S2830). For example, the recommendation device uses a numerical prediction model based on the “position”, “size”, and “layer” attribute values of the keywords included in the “matchingKeywordData” item. , the attribute value of the asset matched with the keyword can be predicted.

Figure 29 is an example of attribute value prediction of an asset to which this specification can be applied.

Referring to FIG. 29, the recommendation device can predict the attribute value of the corresponding asset based on the attribute value of the matched keyword.

Additionally, the recommendation device can check information on content in which the corresponding asset is used through the database and extract attribute values of the corresponding asset from the content. In this case, the recommendation device can filter the asset attribute value in the content in which the corresponding asset is used based on the attribute value of the matched keyword, and use the filtered valid data to predict the attribute value of the asset.

Table 6 below illustrates the attribute values of the predicted assets.

{
"data": [
{
"sentenceKeyword": "...",
"assetList": [
{
"assetIdx": "...",
"predictedProperty": {
"x": "...",
"y": "...",
"z": "...",
"depth": "...",
"width": "...",
"height": "...",
...
}
},
...
]
},
...
]
}

Referring to Table 6, the recommendation device uses a numerical prediction model to predict the coordinate value, depth, width, and height within the page in the "predictedProperty" item corresponding to the asset. You can. The recommendation device can utilize these attribute values to express or arrange assets as visual elements. For example, you can perform functions such as placing the asset at a specific location on the page or placing keywords in 3D space using height and depth.

The recommendation device adds the recommended asset to the page based on the recommended asset list (S2840). For example, the recommender device can add high-priority assets from the recommended asset list to the page.

The recommendation device sets the predicted attribute value to the added asset (S2850). Through this, the recommender device can place assets in appropriate positions on the page and create a scene for content creation.

Figure 30 is an example of scene creation to which this specification can be applied.

Referring to FIG. 16 described above, the user selects the chat icon 1610 to input a sentence, and through the displayed chat window 1620, the user can input a sentence to create content.

When a user inputs the sentence “blue sky and grass with a smiling sun”, the recommendation device can generate a list of keywords in the form shown in Table 3 above.

Thereafter, the recommendation device may match keywords and assets based on the list of keywords and the list of recommended assets.

Table 7 below illustrates the matching results based on the sentence “Blue sky and grass with a smiling sun.”

{
"original": "Blue sky and grass with a smiling sun",
"sentence": "In With the Smile of Sky and Grass, Emily, a girl heavy with sadness, discovers a hidden meadow where vibrant blue skies and dancing green grass are the starting point of change. Immersing herself in this magical place, she finds hope. "You learn to embrace the present and cultivate gratitude. Through the healing power of nature, Emily's radiant smile becomes a symbol of optimism, inspiring those around her and reminding us to recognize the simple beauty in life.",
"processedData": [
{
"matchingKeywordData": {
"keyword": "sky",
"position": "above",
"size": "large",
"layer": "behind"
},
"matchingAssetList": [
{
"assetIdx": "0b70763b-4e22-43bb-a163-68c80f9b9e37"
},
{
"assetIdx": "0akb2847-97df-2k77-ks98-sjf83hje6ljg875"
},
...
]
},
{
"matchingKeywordData": {
"keyword": "grass",
"position": "below",
"size": "medium",
"layer": "front"
},
"matchingAssetList": [
{
"assetIdx": "9djg9273-08jf-kf7j-1kgf-9f7nfjs88305hjs8f"
},
{
"assetIdx": "skfg8hk9-3jgh-33l4-22j5-sdfkefekekd99yn"
},
...
]
},
...
]
}

Afterwards, the recommendation device predicts the attribute values of the corresponding assets based on the “matchingKeywordData” item. Table 8 below illustrates attribute value prediction based on the sentence “blue sky and grass with a smiling sun.”

{
"data": [
{
"sentenceKeyword": "sky",
"assetList": [
{
"assetIdx": "0b70763b-4e22-43bb-a163-68c80f9b9e37",
"predictedProperty": {
"x": 30,
"y": 30,
"z": 7,
"depth": 10,
"width": 400,
"height": 265,
...
}
},
{
"assetIdx": "0akb2847-97df-2k77-ks98-sjf83hje6ljg875",
"predictedProperty": {
"x": 125,
"y": 26,
"z": 10,
"depth": 14,
"width": 533,
"height": 387,
...
}
},
...
]
},
{
"sentenceKeyword": "grass",
"assetList": [
{
"assetIdx": "9djg9273-08jf-kf7j-1kgf-9f7nfjs88305hjs8f",
"predictedProperty": {
"x": 264,
"y": 679,
"z": 21,
"depth": 4,
"width": 36,
"height": 48,
...
}
},
{
"assetIdx": "skfg8hk9-3jgh-33l4-22j5-sdfkefekekd99yn",
"predictedProperty": {
"x": 234,
"y": 774,
"z": 1,
"depth": 11,
"width": 39,
"height": 57,
...
}
},
...
]
},
...
]
}

Referring to Table 8, the recommender device can add “Sky (3010)” to the page and set the predicted attribute value for “Sky”. Recommended devices can be added to the scene through the same operation for other assets.

Figure 31 illustrates another embodiment to which this specification can be applied.

Referring to Figure 31, assets that have not been added to the page in the recommended asset list can be displayed on the terminal as a separate list (A), and recommended assets that do not match keywords can also be displayed in a separate list (B). It can be displayed on the terminal.

When replacing an asset that the user has not added to the page, he or she can select another asset by clicking on it in a separate list (A). In this case, the recommendation device adds or exchanges the selected asset with another asset through the above-described operation. And, the property value of the corresponding asset can be set as the predicted property value.

The above-described specification can be implemented as computer-readable code on a program-recorded medium. Computer-readable media includes all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SDD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. This also includes those implemented in the form of carrier waves (e.g., transmission via the Internet). Accordingly, the above detailed description should not be construed as restrictive in all respects and should be considered illustrative. The scope of this specification should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of this specification are included in the scope of this specification.

In addition, although the description has been made above with a focus on services and embodiments, this is only an example and does not limit the present specification, and those skilled in the art will understand that it does not deviate from the essential characteristics of the services and embodiments. It can be seen that various modifications and applications not exemplified above are possible. For example, each component specifically shown in the embodiments can be modified and implemented. And the differences related to these modifications and applications should be construed as being included in the scope of the present specification as defined in the attached claims.

Claims (8)

In a method for a recommendation device to create a scene included in content,
Receiving a sentence from a user to recommend assets for the scene;
Based on the sentence, generating keywords for recommending the asset;
1) History information on content used by the user through the platform, 2) Information on components of content created by the user, and 3) User characteristic data based on information on content currently being created by the user generating a;
generating a list of assets that can be recommended to the user based on the keyword and the user characteristic data;
generating a list of the keywords; as,
The list of keywords includes attribute values for each keyword;
matching the keywords and the assets according to linguistic similarity, based on the list of keywords and the list of assets; and
predicting an attribute value of a corresponding asset based on the attribute value of the matched keyword;
Includes,
The attribute values for each keyword are
Contains position, size, and order values of the keywords,
The attribute value of the predicted asset is
A creation method including coordinate values, depth, horizontal length, and vertical length of the asset.
delete delete According to paragraph 1,
Based on the list of assets, adding the asset to a page for creating the scene; and
setting the added asset as an attribute value of the predicted asset;
Generating method, including.
According to paragraph 1,
The step of generating the user characteristic data is
Obtaining an image of an asset based on history information of the content, information on components of the content created by the user, and information about the content currently being created by the user;
generating an image caption based on the image of the asset;
generating text related to the image of the asset based on the image caption;
extracting the mood of the text based on the text;
extracting a color based on the image of the asset;
extracting the atmosphere of the color based on the extracted color;
extracting asset characteristics related to the image of the asset based on the mood of the text, the color, and the mood of the color; and
generating the user characteristic data based on the asset characteristic;
Generating method, including.
According to clause 5,
extracting a language mood and obtaining a thumbnail image based on history information of the content, information on components of the content created by the user, and information about the content currently being created by the user;
extracting a color of the thumbnail image and an atmosphere of the color based on the thumbnail image;
extracting asset characteristics included in content related to the thumbnail;
extracting content characteristics based on the language mood, the color of the image, the color mood, and asset characteristics included in content related to the thumbnail; and
generating the user characteristic data based on the content characteristics;
A production method further comprising:
According to paragraph 1,
The steps to create the keyword are
Converting the sentence into a story-type text through a language model;
Converting the story-type text into a plot including 1) background, 2) main character, and 3) main elements through the language model;
converting the story into sentence data associated with language-based data included in the information related to the asset through the language model; and
generating the keyword based on the sentence data;
Generating method, including.
In a recommendation device that generates a scene included in content,
Ministry of Communications;
Memory containing language model; and
a processor for functionally controlling the communication unit and the memory; Includes,
The processor is
Receive input from the user to recommend assets for the scene,
Based on the sentence, keywords are generated to recommend the asset,
1) History information on content used by the user through the platform, 2) Information on components of content created by the user, and 3) User characteristic data based on information on content currently being created by the user Create a ,
Based on the keyword and the user characteristic data, generate a list of assets that can be recommended to the user,
Generate a list of keywords, the list of keywords includes attribute values for each keyword, match the keywords and the assets based on linguistic similarity based on the list of keywords and the list of assets, and Based on the attribute value of the matched keyword, the attribute value of the corresponding asset is predicted, wherein the attribute value for each keyword includes the position, size, and order value of the keyword, and the attribute value of the predicted asset is the asset. A recommendation device including coordinate values, depth, horizontal length, and vertical length.