WO2024091084A1

WO2024091084A1 - Reference scene recommendation method and reference scene recommendation device for automatic video generation

Info

Publication number: WO2024091084A1
Application number: PCT/KR2023/016939
Authority: WO
Inventors: 권석면; 김유석
Original assignee: 주식회사 일만백만
Priority date: 2022-10-27
Filing date: 2023-10-27
Publication date: 2024-05-02
Also published as: KR102560610B1

Abstract

A reference scene recommendation method for automatically generating a video, according to one embodiment of the present disclosure, may comprise the steps of: dividing a collected video into units of a scene to generate a plurality of reference scenes; analyzing the plurality of reference scenes to extract feature information and assigning different kinds of tags to the plurality of reference scenes on the basis of the feature information; and storing, in a reference scene database, the plurality of reference scenes to which the tags have been assigned.

Description

Reference scene recommendation method and reference scene recommendation device for automatic video creation

This disclosure relates to a reference scene recommendation method and reference scene recommendation device for automatic video generation. More specifically, by dividing the video into scenes to create multiple reference scenes and assigning tags to each reference scene, when a keyword is received from the automatic video generation device, the reference scene to which the tag corresponding to the keyword is assigned is recommended. This relates to a reference scene recommendation method and reference scene recommendation device for automatic video creation.

If advertisers want to advertise on the Internet, they must separately produce video advertisements, banner advertisements, and flash advertisements.

However, because it takes a lot of time and money to produce advertisements, advertisers with a variety of products (e.g. large corporations, TV home shopping, online shopping malls, etc.) have to produce advertisements limited to some representative products among all products or to specific products. Produces, but is not limited to, membership registration advertisements, brand promotion advertisements, and/or discount advertisements.

However, in the case of online advertising, the target audience of Internet users is diverse, and the conditions of various promotions change from time to time, making it difficult to create online advertising for individual products.

For example, in the case of today's new product, today's closing product, time special product, etc., the conditions of promotion are limited by time, so it is difficult to create advertisements for the product.

The problem that the present disclosure aims to solve is to divide the video into scene units to generate a plurality of reference scenes and assign tags to each reference scene, so that when a keyword is received from an automatic video generation device, the tag corresponding to the keyword is The aim is to provide a reference scene recommendation method and reference scene recommendation device for automatically generating a video that can recommend assigned reference scenes.

The problem to be solved by the present disclosure is not limited to the above-mentioned problems, and problems not mentioned can be clearly understood by those skilled in the art from this specification and the attached drawings. .

A reference scene recommendation method according to an embodiment of the present disclosure includes dividing collected images into scene units to generate a plurality of reference scenes; extracting feature information by analyzing the plurality of reference scenes and assigning different types of tags to the plurality of reference scenes based on this; and storing a plurality of reference scenes to which the tags are assigned in a reference scene database.

The step of allocating different types of tags to the plurality of reference scenes includes extracting feature information of an object included in the reference scene, extracting a feature descriptor expressing the feature information of the object as a vector value, and assigning an object attribute tag to the reference scene according to a feature descriptor; applying the reference scene to a scene type analysis model to extract the type of situation expressed in the reference scene, and assigning a situation attribute tag to the reference scene according to the type of situation; and extracting a highlight portion from the collected video and assigning a highlight attribute tag to a reference scene corresponding to the highlight portion among a plurality of reference scenes stored in the reference scene database.

A reference scene recommendation device according to an embodiment of the present disclosure includes one or more processors; and a memory including instructions configured to cause the one or more processors to execute operations, wherein the operations include dividing the collected images into scenes to generate a plurality of reference scenes. thing; analyzing the plurality of reference scenes to extract feature information and assigning different types of tags to the plurality of reference scenes based on this; And it may include storing a plurality of reference scenes to which the tags are assigned in a reference scene database.

Assigning different types of tags to each of the plurality of reference scenes includes extracting feature information of an object included in the reference scene, extracting a feature descriptor expressing the feature information of the object as a vector value, and extracting the feature assigning object attribute tags to the reference scene according to a descriptor; applying the reference scene to a scene type analysis model to extract the type of situation expressed in the reference scene, and assigning a situation attribute tag to the reference scene according to the type of situation; and extracting a highlight portion from the collected video and assigning a highlight attribute tag to a reference scene corresponding to the highlight portion among a plurality of reference scenes stored in the reference scene database.

The technical solutions of the present disclosure are not limited to the above-described technical solutions, and technical solutions not mentioned are clearly understood by those skilled in the art from this specification and the attached drawings. It could be.

According to the reference scene recommendation method and reference scene recommendation device for automatic video generation according to an embodiment of the present disclosure, the video is divided into scene units to generate a plurality of reference scenes, and tags are assigned to each reference scene to automatically generate the video. When a keyword is received from the generating device, a reference scene to which a tag corresponding to the keyword is assigned can be recommended to the automatic video generating device.

1 is a diagram illustrating an automatic video generation system according to an embodiment of the present disclosure.

Figure 2 is a diagram illustrating an automatic video generation device according to an embodiment of the present disclosure.

Figure 3 is a diagram illustrating a reference scene recommendation device according to an embodiment of the present disclosure.

4 to 7 are diagrams for explaining the operation of a reference scene recommendation device according to an embodiment of the present disclosure.

Figure 8 is a flow chart illustrating a reference scene recommendation method for automatic video generation according to an embodiment of the present disclosure.

The above-described objects, features and advantages of the present disclosure will become more apparent through the following detailed description in conjunction with the accompanying drawings. However, since the present disclosure can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail below.

Like reference numerals throughout the specification in principle refer to the same elements. In addition, components with the same function within the scope of the same idea shown in the drawings of each embodiment will be described using the same reference numerals, and overlapping descriptions thereof will be omitted.

If it is determined that a detailed description of a known function or configuration related to the present disclosure may unnecessarily obscure the gist of the present disclosure, the detailed description will be omitted. In addition, numbers (eg, first, second, etc.) used in the description of this specification are merely identifiers to distinguish one component from another component.

In addition, the suffixes “module” and “part” for components used in the following examples are given or used interchangeably only for the ease of writing the specification, and do not have distinct meanings or roles in themselves.

In the following examples, singular terms include plural terms unless the context clearly dictates otherwise.

In the following embodiments, terms such as include or have mean the presence of features or components described in the specification, and do not exclude in advance the possibility of adding one or more other features or components.

In the drawings, the sizes of components may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are arbitrarily shown for convenience of explanation, and the present disclosure is not necessarily limited to what is shown.

If an embodiment can be implemented differently, the order of specific processes may be performed differently from the order described. For example, two processes described in succession may be performed substantially simultaneously, or may proceed in an order opposite to that in which they are described.

In the following embodiments, when components are connected, this includes not only the case where the components are directly connected, but also the case where the components are indirectly connected by intervening between the components.

For example, in this specification, when components, etc. are said to be electrically connected, this includes not only cases where the components are directly electrically connected, but also cases where components, etc. are interposed and indirectly electrically connected.

Hereinafter, a reference scene recommendation method and a reference scene recommendation device for automatically generating a video according to an exemplary embodiment of the present disclosure will be described with reference to FIGS. 1 to 8.

Referring to FIG. 1 , the automatic video generation system may include an automatic video generation device 200, a reference scene recommendation device 300, one or more customer terminals 400, and one or more user terminals 500.

Customer terminal 400 may refer to an electronic device used by customers such as advertisers. The user terminal 500 may refer to an electronic device used by general users other than advertisers. The customer can input the video generation reference information needed to automatically generate the video into the customer terminal 400, and the customer terminal 400 can transmit the video generation reference information input by the customer to the automatic video creation device 200. there is. According to an embodiment, the image generation reference information may be a keyword in word units.

The automatic video generation device 200 can automatically generate videos, such as advertising videos, according to customer requests. Specifically, when video generation reference information is received from the customer terminal 400, the automatic video creation device 200 may generate a script using the received video generation reference information and a pre-generated script database.

According to an embodiment, the script database may store one or more attributes related to a keyword and text matching each attribute. Specifically, one or more properties related to a keyword include object properties of the object corresponding to the keyword, screen properties of the scene matching the object, situation properties of the scene matching the object, and highlight properties of the scene matching the object. there is. Accordingly, the automatic video generation device 200 may generate a script of a reference scene using text that matches an attribute determined based on user behavior information using customer-related content among one or more attributes related to a keyword.

Afterwards, the automatic video generation device 200 may generate a scenario consisting of a reference scene based on the script.

Furthermore, the automatic video creation device 200 can extract keywords from the script. More specifically, the automatic video generation device 200 can extract words from the text of the script based on spaces. And, based on a database of frequency values for each word created in advance, the frequency values of the extracted words can be measured.

Then, the automatic video generation device 200 may generate tokens by performing morphological analysis on each of the extracted words. According to an embodiment, a token may include a pair of words and morpheme values, and may be assigned a label indicating a frequency value. For example, the automatic video generating device 200 has (frequency value: 1000, (word, morpheme value)), (frequency value: 234, (word, morpheme)), (frequency value: 2541, (word, morpheme) ), and (frequency value: 2516, (word, morpheme)) can be created.

Thereafter, the automatic video generating device 200 may assign different weights to each token according to the word of each token and/or the label of each token. According to one embodiment, the automatic video generation device 200 determines the type of language that implements the words in the token (e.g., English, Chinese, Korean, etc.), the position of the words within the text of the script, and/or the characters assigned to the token. Depending on the frequency value indicated by the label, different weights can be assigned to each token. A more detailed explanation of assigning different weights to each token is as follows.

First, the automatic video generation device 200 may calculate the first weight using the total number of tokens generated from the text of the script and the order of each token.

Specifically, the automatic video generation device 200 calculates the order of the current token based on the total number of tokens generated from the text of the script and an important value predetermined according to the type of language. The first weight can be calculated. For example, if the total number of tokens is 12 and the current token order is 4th, 12 can be assumed to be '1' and 1 can be divided by 4 to calculate '0.25'. And the first weight can be calculated by reflecting the important value predetermined according to the type of language in the value calculated in this way. According to an embodiment, the significant value may change depending on the order of the current token. Specifically, if an important word is a language that appears at the end of a sentence, the important value reflected may also increase as the order of the current token increases. If the important word is a language that appears at the beginning of the sentence, the important value reflected will decrease as the order of the current token increases.

Next, the automatic video generating device 200 creates a second weight for the current token using the frequency value indicated by the label of the current token, the frequency value indicated by the label of the previous token, and the frequency value indicated by the label of the next token. can be calculated.

Finally, the automatic video generating device 200 may assign a final weight to the current token using the first weight and the second weight. Then, keywords consisting of tokens with final weights can be extracted.

The automatic video generation device 200 may provide the reference scene recommendation device 300 with a reference scene recommendation request message including keywords composed of tokens with different weights. Additionally, a reference scene may be received from the reference scene recommendation device 300.

Thereafter, the automatic video generation device 200 may generate an image by combining the received reference scene and pre-generated environmental data. To this end, the automatic video generation device 200 may select sound data according to a scenario and convert text data corresponding to the scenario into voice data. And, the automatic video generation device 200 can generate an AI actor according to the above scenario.

Meanwhile, the reference scene recommendation device 300 can collect images to automatically generate videos according to customer requests and build a reference scene database based on the collected images. In addition, when the reference scene recommendation device 300 receives a reference scene recommendation request message from the automatic video generation device 200, it refers to a reference scene to which a tag that is the same or similar to a keyword included in the reference scene recommendation request message is assigned. It can be extracted from the scene database and provided to the automatic video creation device 200.

Specifically, the reference scene recommendation device 300 may collect images (eg, videos). Then, the collected video can be decoded to obtain the frames that make up the video, and then the frames can be sampled at playback time intervals.

Thereafter, the reference scene recommendation device 300 may list the sampled frames in the order of playback time and calculate the degree of similarity between adjacent frames. When the similarity is calculated for all the listed frames, the reference scene recommendation device 300 groups the frames based on the similarity, thereby generating a plurality of reference scenes divided by scene.

According to one embodiment, the reference scene recommendation device 300 may perform feature matching on adjacent frames to calculate similarity between adjacent frames. Specifically, the reference scene recommendation device 300 compares the keypoints between adjacent frames and, if the similarity is greater than the reference value, groups the frames into one scene to create one reference scene. . If, as a result of comparing feature points between adjacent frames, the similarity is less than the standard value, it can be determined that the scene has been switched, and different reference images can be generated by grouping the corresponding frames into different scenes.

According to another embodiment, the reference scene recommendation device 300 may extract objects for each listed frame and then determine whether to change the scene based on a change in the number of extracted objects. Additionally, a reference scene can be created based on the point in time when the number of extracted objects changes or the point in time when the number of extracted objects changes beyond the standard value.

According to another embodiment, the reference scene recommendation device 300 determines whether the background has changed based on a change in pixel value between pixels in adjacent frames, and determines whether or not there is a scene change based on the determination result. You can judge. Next, a reference scene can be created based on the point in time when the background changes.

According to another embodiment, the reference scene recommendation device 300 may determine whether to switch scenes based on changes in the content of audio data and/or subtitle data constituting the video. Additionally, a reference scene can be created based on the point in time when new content appears in the audio data and/or subtitle data.

According to another embodiment, the reference image recommendation device 300 may extract objects for each listed frame and then determine whether a scene change occurs based on a change in the type of the extracted object. Additionally, a reference scene can be created based on the point in time when a previously extracted object disappears and/or when a new object appears.

When a plurality of reference scenes divided into scene units are created through the above-described process, the reference scene recommendation device 300 can analyze the plurality of reference scenes and extract characteristic information of the reference scene. And, depending on the extracted feature information, different types of tags can be assigned to each reference scene. For example, depending on the extracted feature information, one of an object attribute tag, a screen attribute tag, a situation attribute tag, and a highlight attribute tag can be assigned. Hereinafter, the process of allocating an object attribute tag, a process of allocating a screen attribute tag, and a process of assigning a situation attribute tag will be described in turn.

In order to assign an object attribute tag, the reference scene recommendation device 300 may detect a feature area of an object in the reference scene (Interest Point Detection). The feature area refers to the main area from which a feature descriptor that describes the characteristics of an object is extracted. Feature descriptors may also be referred to as descriptors, feature vectors, or vector values, and may be used to determine whether objects are identical or similar.

For example, the feature area is the contour included in the object, corners such as corners among the contours, blobs that are distinct from the surrounding area, areas that are invariant or covariant according to the transformation of the reference scene data, and/or the surrounding brightness. May contain poles with dark or bright features. The feature area may target a patch (piece) of the reference scene or the entire reference scene.

After detecting the feature area of an object in the reference scene, the reference scene recommendation device 300 may extract feature information of the object from the detected feature area. Additionally, a feature descriptor expressing the extracted feature information as a vector value can be extracted. And object attribute tags can be assigned to the reference scene according to the feature descriptor.

To assign a screen attribute tag, the reference scene recommendation device 300 may detect a feature area of the reference scene. And the feature information of the reference scene can be extracted from the feature area of the detected reference scene. Additionally, a feature descriptor expressing the extracted feature information as a vector value can be extracted. And screen attribute tags can be assigned to the reference scene according to the feature descriptor.

The above-mentioned feature descriptor may be calculated using the location of the feature area, brightness, color, sharpness, gradient, scale and/or pattern information of the feature area in the reference scene. For example, the feature descriptor may calculate the brightness value, brightness change value, and/or distribution value of the feature area by converting them into vectors.

Furthermore, the feature descriptor is not only a local descriptor based on the feature area as described above, but also a global descriptor, frequency descriptor, binary descriptor, or neural network descriptor. It can also be expressed as

The global descriptor can convert the brightness, color, sharpness, gradient, scale, and/or pattern information of the entire reference scene, each area where the reference scene is divided by an arbitrary standard, or each feature area into vector values. there is. The frequency descriptor can convert the number of times pre-classified feature descriptors are included in a reference scene and/or the number of times they include global features such as a conventionally defined color table into a vector value. A binary descriptor can be used by extracting in bits whether each descriptor is included and/or whether the size of each element value constituting the descriptor is larger or smaller than a specific value, and then converting it to an integer type. A neural network descriptor can extract image information used for learning or classification from the layers of a neural network.

To assign a situation attribute tag, the reference scene recommendation device 300 may apply the reference scene to a scene type analysis model. A scene type analysis model may refer to a model learned to receive a scene as input and output the scene type. Additionally, the scene type may refer to the type of situation being expressed in the scene. When the type of situation expressed in the reference scene is extracted through the scene type analysis model, the reference scene recommendation device 300 may assign a situation attribute tag to the reference scene according to the type of the extracted situation.

According to an embodiment, the reference scene recommendation device 300 may build a scene type analysis model as a CNN (Convolution Neural Network) model, which is one of the deep learning models, and learn the above-described data set. At this time, the CNN model can be designed to include two convolutional layers, a relu layer, a max pooling layer, and one fully connected layer. Furthermore, the reference scene recommendation device 300 uses the RCNN technique to construct a feature sequence in the map order of the convolution feature maps calculated from the CNN model, and then converts each feature sequence into a long and short term. It can be learned by applying to memory networks (LSTM; Long Short Term Memory networks).

In order to assign a highlight attribute tag, the reference scene recommendation device 300 may extract a highlight portion from the image. The highlight portion may refer to the section containing the most important information in the video. For example, if the content of the video consists of four sections of 'Before', 'Before', and 'Before', the section corresponding to 'Before' may be considered the highlight section. Highlights can be extracted manually or automatically. When the highlight portion of the video is extracted, the reference scene recommendation device 300 may assign a highlight attribute tag to the reference scene corresponding to the highlight portion.

After assigning tags to each of a plurality of reference scenes according to the above-described method, upon receiving a reference scene recommendation request message from the automatic video generation device 200, the reference scene recommendation device 300 includes a tag in the reference scene recommendation request message. A reference scene to which a tag identical or similar to the existing keyword is assigned can be extracted from the reference scene database and provided to the automatic video generation device 200.

Specifically, upon receiving a reference scene recommendation request message from the automatic video generation device 200, the reference video data recommendation device 300 may extract a keyword from the reference scene recommendation request message and extract tokens constituting the keyword. there is. Next, a tag that matches the morpheme value of the token can be selected from among a plurality of tags assigned to the reference scene. And if the selected tag and the word in the token match, the reference scene to which the tag is assigned can be extracted from the reference scene database.

For example, when the morpheme value of the token is a noun, the reference scene recommendation device 300 may select an object attribute tag from a plurality of tags assigned to the reference scene. And if the object attribute tag and the word in the token match, the reference scene to which the corresponding tag is assigned can be extracted from the reference scene database and provided to the automatic video generation device 200.

As another example, when the morpheme value of the token is an adjective, the reference scene recommendation device 300 may select a screen attribute tag and a situation attribute tag from a plurality of tags assigned to the reference scene. And if the screen attribute tag and the word in the token match, and the situation attribute tag and the word in the token match, the reference scene to which the corresponding tag is assigned is extracted from the reference scene database and provided to the automatic video creation device 200. You can.

Meanwhile, the reference scene recommendation device 300 calculates the similarity ratio between each of a plurality of tags assigned to the reference image and the word of the token, targeting a reference scene to which a tag that does not match the morpheme value of the token is assigned. You can. Additionally, reference scenes to which tags with a similarity ratio greater than a certain ratio are assigned can be extracted from the reference scene database and provided to the automatic video generation device 200.

Specifically, the reference scene recommendation device 300 may compare the characters constituting the tag assigned to the reference scene and the characters constituting the word of the token to calculate the number of matching characters. Also, by comparing the number of strings corresponding to the tag and the number of strings corresponding to the word of the token, a longer number of strings can be selected. Additionally, a similarity ratio representing the ratio of the number of matching characters to the number of selected strings can be calculated. In addition, reference scenes to which tags with a similarity ratio greater than a certain ratio are assigned can be extracted from the reference scene database and provided to the automatic video generation device 200.

The automatic video generation device 200 and/or the reference scene recommendation device 300 as described above may be implemented as included in, for example, a web service providing server. The web service providing server can provide various contents to the user terminal 500. The type of content provided to the user terminal 500 may vary depending on the type of application used by the user terminal 500 to access the web service providing server. This web service providing server may be implemented as an online shopping mall server or a search engine server.

The customer terminal 400 may include an application for accessing a web service providing server. Accordingly, when the application is selected and executed by the customer, the customer terminal 400 can access the automatic video generating device 200 through the application. Thereafter, when the customer inputs video generation reference information into the customer terminal 400, the customer terminal 400 may request automatic video generation by providing the input video generation reference information to the automatic video generation device 200.

Like the customer terminal 400, the user terminal 500 may include an application for accessing a web service providing server. Accordingly, when the application is selected and executed by the user, the user terminal 500 can access the web service providing server through the application.

The user terminal 500 can display a web page provided from a web service providing server through an application. A web page may include a screen loaded on an electronic device and/or content within the screen so that it can be immediately displayed on the screen according to a user's scroll input. For example, while a web page is displayed on an application of the user terminal 500, the entire application execution screen that extends horizontally or vertically and is displayed as the user scrolls may be included in the concept of a web page. Additionally, the camera roll screen can also be included in the concept of a web page.

Meanwhile, the user terminal 500 may include an application (eg, software, neural network model, etc.) for analyzing user interests. Accordingly, the user terminal 500 may collect and store log records and/or engagement records and determine the user's interests by analyzing the log records and/or engagement records through an application for user interest analysis.

According to one embodiment, the user terminal 500 may extract content by analyzing log records and/or engagement records stored in the user terminal 500, and create a label indicating the type of extracted content. It can be extracted. Log records may be created by recording events that occur while the operating system or software of the user terminal 500 is running. Engagement records can be created by recording a set of committed actions that result in a user becoming interested, participating, and engaging. User behavior information includes not only actions such as the user viewing content through a web browser, the user creating a 'like' tag on content through social networks, and the user viewing images or text on the homepage. , it can also include the object of these actions, the time when these actions occurred, and the time these actions were maintained. A label indicating the type of extracted content may indicate, for example, whether the extracted content corresponds to the user's interests or not. A label indicating the type of extracted content may be extracted by analyzing log records and/or engagement records, or may be extracted from labels stored in advance.

According to another embodiment, the user terminal 500 may be equipped with a crawler, a parser, and an indexer, through which web pages viewed by the user may be collected. Additionally, the item information (e.g., image, item name, and item price) included in the collected web pages can be accessed to extract content and a label indicating the type of content. Specifically, the crawler can collect data related to item information by collecting a list of web addresses that users browse, checking websites, and tracking links. The parser can interpret web pages collected during the crawling process and extract item information such as images, item prices, and item names included in the page. The indexer can index the location and meaning of the extracted item information.

Referring to FIG. 2, the automatic video generation device 200 includes a script creation unit 210, a scenario creation unit 220, a keyword extraction unit 230, a reference scene transmission/reception unit 240, and an environment data creation unit 250. and an image synthesis unit 260.

When image generation reference information including keywords in word units is received from the customer terminal 400, the script generator 210 may generate a script using the received image generation reference information and a pre-generated script database.

Specifically, the script generator 210 searches the script database for keywords included in the image generation reference information, and then generates object properties of the object corresponding to the searched keyword, screen properties of the scene matching the object, and scenes matching the object. A script can be created using text that matches the attributes determined based on the user's behavior information using content related to the customer among the highlight attributes of the scene matching the situation attributes and objects.

The scenario generator 220 may generate a scenario composed of a standard scene based on the script generated by the script generator 210. According to embodiments, the scenario may further include sound effects and/or atmosphere in addition to the reference scene.

The keyword extraction unit 230 may extract keywords from the script generated by the script creation unit 210. More specifically, the keyword extractor 230 may extract words from the text of the script based on spaces. And, based on a database of frequency values for each word created in advance, the frequency values of the extracted words can be measured.

Afterwards, the keyword extraction unit 230 may generate a token by performing morphological analysis on each of the extracted words. According to an embodiment, a token may include a pair of words and morpheme values, and may be assigned a label indicating a frequency value. For example, the keyword extraction unit 230 has (frequency value: 1000, (word, morpheme value)), (frequency value: 234, (word, morpheme)), (frequency value: 2541, (word, morpheme)) Tokens such as , and (frequency value: 2516, (word, morpheme)) can be generated.

Thereafter, the keyword extractor 230 may assign different weights to each token according to the word and/or label of each token. According to one embodiment, the keyword extraction unit 230 determines the type of language (e.g., English, Chinese, Korean, etc.) that implements the word in the token, the position of the word within the text of the script, and/or the label assigned to the token. Depending on the frequency value indicated, different weights can be assigned to each token. A more detailed explanation of assigning different weights to each token is as follows.

First, the keyword extractor 230 may calculate the first weight using the total number of tokens generated from the text of the script and the order of each token.

Specifically, the keyword extraction unit 230 quantifies the order of the current token based on the total number of tokens generated from the text of the script and an important value predetermined according to the type of language, and provides information on the current token. 1 Weight can be calculated. For example, if the total number of tokens is 12 and the token order is 4th, the keyword extractor 230 may assume 12 as '1' and divide 1 by 4 to calculate '0.25'. And the first weight can be calculated by reflecting the important value predetermined according to the type of language in the value calculated in this way. According to an embodiment, the significant value may change depending on the order of the current token. Specifically, if an important word is a language that appears at the end of a sentence, the important value reflected may also increase as the order of the current token increases. If the important word is a language that appears at the beginning of the sentence, the important value reflected will decrease as the order of the current token increases.

Next, the keyword extractor 230 may calculate the second weight using the frequency value indicated by the label of the current token, the frequency value indicated by the label of the previous token, and the frequency value indicated by the label of the next token. .

Finally, the keyword extractor 230 may assign a final weight to the current token using the first weight and the second weight. Then, keywords consisting of tokens with final weights can be extracted.

The reference scene transceiver 240 provides a reference scene recommendation request message containing keywords composed of tokens with different weights to the reference scene recommendation device 300, and selects the reference scene from the reference scene recommendation device 300. You can receive it.

The environmental data generator 250 may select sound data according to the scenario. And text data corresponding to the above scenario can be converted into voice data. Furthermore, an AI actor can be created according to the above scenario.

The image synthesis unit 260 may generate an image by combining the reference scene received by the reference scene transmission/reception unit 240 and the environment data generated by the environment data generation unit 250.

Figure 3 is a diagram of a reference scene recommendation device according to an embodiment of the present disclosure.

Referring to FIG. 3, the reference scene recommendation device 300 may build a reference scene database based on the collected images. Additionally, when receiving a reference scene recommendation request message from the automatic video generation device 200, the reference scene recommendation device 300 refers to a reference scene to which a tag that is the same or similar to a keyword included in the reference scene recommendation request message is assigned. It can be extracted from the scene database and provided to the automatic video creation device 200. To this end, the reference scene recommendation device 300 may include an image segmentation unit 310, a tag allocation unit 320, a reference scene database 330, and a reference scene recommendation unit 340.

The image segmentation unit 310 may collect images (eg, videos). Then, the collected video can be decoded to obtain the frames that make up the video, and then the frames can be sampled at playback time intervals.

Thereafter, the image segmentation unit 310 may arrange the sampled frames in the order of playback time and calculate the degree of similarity between adjacent frames. When the similarity is calculated for all the listed frames, the reference scene recommendation device 300 groups the frames based on the similarity, thereby generating a plurality of reference scenes divided by scene.

According to one embodiment, the image segmentation unit 310 may perform feature matching on adjacent frames to calculate the degree of similarity between adjacent frames. Specifically, the image segmentation unit 310 may compare keypoints between adjacent frames and, if the similarity is greater than or equal to a reference value, generate one reference scene by grouping the corresponding frames into one scene. If, as a result of comparing feature points between adjacent frames, the similarity is less than the standard value, it can be determined that the scene has been switched, and different reference images can be generated by grouping the corresponding frames into different scenes.

According to another embodiment, the image segmentation unit 310 may extract objects for each listed frame and then determine whether to change the scene based on a change in the number of extracted objects. Additionally, a reference scene can be created based on the point in time when the number of extracted objects changes or the point in time when the number of extracted objects changes beyond the standard value.

According to another embodiment, the image segmentation unit 310 determines whether the background changes based on the change in pixel value between pixels of the same position among pixels of adjacent frames, and determines whether there is a scene change based on the determination result. can do. Next, a reference scene can be created based on the point in time when the background changes.

According to another embodiment, the video segmentation unit 310 may determine whether to change the scene based on a change in the content of the audio data and/or subtitle data constituting the video. Additionally, a reference scene can be created based on the point in time when new content appears in the audio data and/or subtitle data.

According to another embodiment, the image segmentation unit 310 may extract objects for each listed frame and then determine the beginning of a scene change based on a change in the type of the extracted object. Additionally, a reference scene can be created based on the point in time when a previously extracted object disappears and/or when a new object appears.

The tag allocator 320 may analyze a plurality of reference scenes and extract characteristic information of the reference scenes. And, depending on the extracted feature information, different types of tags can be assigned to each reference scene. For example, depending on the extracted feature information, one of an object attribute tag, a screen attribute tag, a situation attribute tag, and a highlight attribute tag can be assigned.

In order to assign an object attribute tag, the tag allocator 320 may detect a characteristic area of an object in a reference scene (Interest Point Detection). The feature area refers to the main area from which a feature descriptor that describes the characteristics of an object is extracted. Feature descriptors may also be referred to as descriptors, feature vectors, or vector values, and may be used to determine whether objects are identical or similar.

For example, feature areas include the contours of the object, corners such as corners among the contours, blobs that are distinct from the surrounding area, areas that are invariant or covariant depending on the deformation of the reference scene, and/or are darker than the surrounding brightness. Or it may contain poles with bright features. The feature area may target a patch (piece) of the reference scene or the entire reference scene.

After detecting the feature area of the object in the reference scene, the tag allocator 320 may extract feature information of the object from the detected feature area. Additionally, a feature descriptor expressing the extracted feature information as a vector value can be extracted. And object attribute tags can be assigned to the reference scene according to the feature descriptor.

To assign a screen attribute tag, the tag allocator 320 may detect a feature area of a reference scene. And the feature information of the reference scene can be extracted from the feature area of the detected reference scene. Additionally, a feature descriptor expressing the extracted feature information as a vector value can be extracted. And screen attribute tags can be assigned to the reference scene according to the feature descriptor.

The above-described feature descriptor may be calculated using the location of the feature area, brightness, color, sharpness, gradient, scale and/or pattern information of the feature area in the reference scene. For example, the feature descriptor may calculate the brightness value, brightness change value, and/or distribution value of the feature area by converting them into vectors.

To assign a situation attribute tag, the tag allocation unit 320 may apply the reference scene to the scene type analysis model. A scene type analysis model may refer to a model learned to receive a scene as input and output the scene type. Additionally, the scene type may refer to the type of situation being expressed in the scene. When the type of situation expressed in the reference scene is extracted through the scene type analysis model, the tag allocation unit 320 may assign a situation attribute tag to the reference scene according to the type of the extracted situation.

According to an embodiment, the tag allocator 320 may build a scene type analysis model as a CNN (Convolution Neural Network) model, which is one of the deep learning models, and learn the above-described data set. At this time, the CNN model can be designed to include two convolutional layers, a relu layer, a max pooling layer, and one fully connected layer. Furthermore, the tag allocation unit 320 uses the RCNN technique to construct a feature sequence in the map order of the convolution feature maps calculated from the CNN, and then stores each feature sequence in a long-short-term memory network. It can be learned by substituting for (LSTM; Long Short Term Memory networks).

In order to assign a highlight attribute tag, the tag allocation unit 320 may extract the highlight portion from the video. The highlight portion may refer to the section containing the most important information in the video. For example, if the content of the video consists of four sections of 'Before', 'Before', and 'Before', the section corresponding to 'Before' may be considered the highlight section. Highlights can be extracted manually or automatically. When the highlight portion of the video is extracted, the tag allocation unit 320 may assign a highlight attribute tag to the reference scene corresponding to the highlight portion.

Reference scenes to which tags are assigned by the tag allocation unit 320 may be stored in the reference scene database 330. According to an embodiment, the reference scene database 330 may store the start time of the reference scene, the end time of the reference scene, and one or more tags assigned to the reference scene in a table format.

When receiving a reference scene recommendation request message from the automatic video generating device 200, the reference scene recommendation unit 340 may extract a keyword from the reference scene recommendation request message. And the tokens that make up the keyword can be extracted. Next, a tag that matches the morpheme value of the token can be selected from among a plurality of tags assigned to the reference scene. And if the selected tag and the word in the token match, the reference scene to which the tag is assigned can be extracted from the reference scene database.

.

Meanwhile, the reference scene recommendation unit 340 targets a reference image to which a tag that does not match the morpheme value of the token is assigned, and calculates the similarity ratio between each of the plurality of tags assigned to the reference image and the word of the token. You can. Additionally, reference scenes to which tags with a similarity ratio greater than a certain ratio are assigned can be extracted from the reference scene database and provided to the automatic video generation device 200.

Specifically, the reference scene recommendation unit 340 may compare the characters constituting the tag assigned to the reference scene and the characters constituting the word of the token to calculate the number of matching characters. Also, by comparing the number of strings corresponding to the tag and the number of strings corresponding to the word of the token, a longer number of strings can be selected. Additionally, a similarity ratio representing the ratio of the number of matching characters to the number of selected strings can be calculated. Additionally, a reference image to which a tag with a similarity ratio greater than a certain rate is assigned can be extracted from the reference image database and provided to the automatic video generation device 200.

Referring to FIGS. 4 to 7 , the reference scene recommendation device 300 may collect an image 410. The collected image 410 may be provided to the image segmentation unit. The image division unit may divide the input image into scenes to create a plurality of reference scenes (420_1 to 420_4). A plurality of reference scenes may be input to the tag allocation unit. The tag allocation unit may assign tags to each reference scene (420_1 to 420_4). Reference scenes 420_1 to 420_4 to which tags are assigned may be stored in the reference scene database 430.

According to an embodiment, the image segmentation unit may decode the input image 410 to obtain frames constituting the image, and then sample the frames at playback time intervals.

Thereafter, the image segmentation unit may calculate the similarity between adjacent frames among the sampled frames and group the frames based on the similarity, thereby generating a plurality of reference scenes divided on a scene basis.

In addition, the tag allocation unit can analyze a plurality of reference scenes (420_1 to 420_4) to extract characteristic information of each reference scene and assign different types of tags to each reference scene (420_1 to 420_1) according to the extracted specific information. there is. For example, the tag allocation unit may allocate one of an object attribute tag, a screen attribute tag, a situation attribute tag, and a highlight attribute tag, according to the extracted feature information.

In order to assign an object attribute tag, the tag allocation unit may detect the feature area of the object in the reference scene and extract feature information of the object from the detected feature area. Additionally, a feature descriptor expressing the extracted feature information as a vector value can be extracted. And object attribute tags can be assigned to the reference scene according to the feature descriptor.

For example, as shown in FIG. 6(a), the tag allocator may analyze the reference scene 420_3 and detect the feature area of the object (Interest Point Detection). And as shown in FIG. 6(b), the object and its characteristic information can be extracted from the detected feature area. Afterwards, the tag allocation unit can extract the feature information of the object by expressing the feature information of the object as a vector value. Next, the tag allocation unit may allocate an object attribute tag to the reference scene 420_3 according to the characteristic information of the object, as shown in FIG. 6(c)d.

Referring to FIG. 8, the reference scene recommendation device 300 may collect an image and then divide the collected image into scenes to generate a plurality of reference scenes (S810).

The reference scene recommendation device 300 may extract feature information by analyzing a plurality of reference scenes and then assign different types of tags to the plurality of reference scenes based on this (S820).

The reference scene recommendation device 300 may store a reference scene to which a tag is assigned in a reference scene database (S830).

When the reference scene recommendation device 300 receives a reference scene recommendation request message from the automatic video generation device 200, the reference scene recommendation device 200 extracts a reference scene from the reference scene database based on the received reference scene recommendation request message. ) can be provided (S840).

According to one embodiment, step S840 includes the reference scene recommendation device 300 receiving a reference scene recommendation request message from the automatic video generation device 200, and extracting keywords included in the reference scene recommendation request message. Step, extracting the tokens constituting the keyword, selecting a tag that matches the morpheme value of the token among a plurality of tags assigned to the reference scene, if the selected tag matches the word of the token, the corresponding tag is It may include extracting an assigned reference scene from a reference scene database, and providing the extracted reference scene to the automatic video generating device 200.

According to another embodiment, the step S840 includes the reference scene recommendation device 300 receiving a reference scene recommendation request message from the automatic video generation device 200, and extracting keywords included in the reference scene recommendation request message. Step, extracting tokens constituting a keyword, targeting a reference image to which a tag that does not match the morpheme value of the token among a plurality of tags assigned to the reference scene is assigned, a plurality of tags assigned to the reference image A step of calculating a similarity ratio between each tag and a word in the talk, extracting a reference scene assigned to a tag with a similarity ratio of a certain ratio or more from the reference scene database, and converting the extracted reference scene to the automatic video generation device 200. It may include providing steps.

More specifically, calculating the similarity ratio includes calculating the number of matching characters by comparing the characters constituting the tag assigned to the reference scene and the characters constituting the word of the token, and calculating the number of matching characters. Comparing the number of strings and the number of strings corresponding to the word of the token, selecting a greater number of strings, calculating a similarity ratio indicating the ratio of the number of matching characters to the number of selected strings, the similarity ratio is specified It may include extracting a reference scene to which a tag greater than or equal to the ratio is assigned from a reference scene database, and providing the extracted reference scene to the automatic video generating device 200 .

Above, with reference to FIGS. 1 to 8 , a reference scene recommendation method and a reference scene recommendation device for automatically generating a video according to an embodiment of the present disclosure have been described.

According to an embodiment of the present disclosure, programs for various operations of the reference scene recommendation device 300 may be stored in the memory of the reference scene recommendation device 300. The processor of the reference scene recommendation device 300 may load and execute a program stored in the memory. The processor may be implemented as an application processor (AP), central processing unit (CPU), microcontroller unit (MCU), or similar devices, depending on hardware, software, or a combination thereof. At this time, hardware may be provided in the form of an electronic circuit that processes electrical signals to perform a control function, and software may be provided in the form of a program or code that drives the hardware circuit.

Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium that stores instructions executable by a computer. Instructions may be stored in the form of program code, and when executed by a processor, may create program modules to perform operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

Computer-readable recording media include all types of recording media storing instructions that can be decoded by a computer. For example, there may be read only memory (ROM), random access memory (RAM), magnetic tape, magnetic disk, flash memory, optical data storage, etc.

Additionally, computer-readable recording media may be provided in the form of non-transitory storage media. Here, 'non-transitory storage medium' only means that it is a tangible device and does not contain signals (e.g. electromagnetic waves). This term refers to cases where data is semi-permanently stored in a storage medium and temporary storage media. It does not distinguish between cases where it is stored as . For example, a 'non-transitory storage medium' may include a buffer where data is temporarily stored.

According to one embodiment, methods according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable recording medium (e.g. compact disc read only memory (CD-ROM)) or via an application store (e.g. Play StoreTM) or on two user devices (e.g. It can be distributed directly between smartphones (e.g. smartphones) or distributed online (e.g. downloaded or uploaded). In the case of online distribution, at least a portion of a computer program product (e.g., a downloadable app) is stored at least temporarily on a machine-readable recording medium, such as the memory of a manufacturer's server, an application store's server, or a relay server. It can be stored or created temporarily.

The features, structures, effects, etc. described in the embodiments above are included in at least one embodiment of the present disclosure and are not necessarily limited to only one embodiment. Furthermore, the features, structures, effects, etc. illustrated in each embodiment can be combined or modified and implemented in other embodiments by a person with ordinary knowledge in the field to which the embodiments belong. Accordingly, contents related to such combinations and modifications should be construed as being included in the scope of the present disclosure.

In addition, although the above description focuses on the embodiment, this is only an example and does not limit the present disclosure, and those skilled in the art will be able to understand the above without departing from the essential characteristics of the present embodiment. You will see that various modifications and applications not illustrated are possible. In other words, each component specifically shown in the embodiment can be modified and implemented. And these variations and differences related to application should be construed as being included in the scope of the present disclosure as defined in the attached claims.

The reference scene recommendation method and reference scene recommendation device described above can be applied to the video production field.

Claims

In the reference scene recommendation method for automatically generating a video executed on a reference scene recommendation device,

Generating a plurality of reference scenes by dividing the collected images into scenes;

extracting feature information by analyzing the plurality of reference scenes and then assigning different types of tags to the plurality of reference scenes based on this; and

Including storing a plurality of reference scenes to which the tags are assigned in a reference scene database,

The step of assigning different types of tags to each of the plurality of reference scenes includes:

Extracting feature information of an object included in the reference scene, extracting a feature descriptor expressing the feature information of the object as a vector value, and assigning an object attribute tag to the reference scene according to the feature descriptor;

applying the reference scene to a scene type analysis model to extract the type of situation expressed in the reference scene, and assigning a situation attribute tag to the reference scene according to the type of situation; and

Comprising the step of extracting a highlight portion from the collected video and assigning a highlight attribute tag to a reference scene corresponding to the highlight portion among a plurality of reference scenes stored in the reference scene database.

How to recommend reference scenes.
According to paragraph 1,

Receiving a reference scene recommendation request message including keywords composed of tokens with different weights from an automatic video generating device;

extracting a reference scene to which a tag matching a morpheme value of the token is assigned from among a plurality of tags of the reference scene data; and

If the tag of the extracted reference scene matches the word of the token, further comprising providing the extracted reference scene to the automatic video generating device,

How to recommend reference scenes.
According to paragraph 1,

Receiving a reference scene recommendation request message including keywords composed of tokens to which different weights are assigned from an automatic video generating device;

extracting a reference scene to which a tag that does not match the morpheme value of the token is assigned among the plurality of tags of the reference scene data;

calculating a similarity ratio between each of a plurality of tags assigned to the extracted reference scene and the word of the token; and

Further comprising providing a reference scene to which a tag having a similarity ratio of a certain ratio or more is assigned to the automatic video generating device,

How to recommend reference scenes.
According to paragraph 3,

The step of calculating the similarity ratio is,

Comparing letters constituting a plurality of tags assigned to the extracted reference scene and letters constituting a word of the token to calculate the number of matching letters;

Comparing the number of strings corresponding to the plurality of tags and the number of strings corresponding to words of the token, and selecting a longer number of strings; and

Comprising the step of calculating the similarity ratio, which represents the ratio of the number of matching characters to the number of selected strings,

How to recommend reference scenes.
In the reference scene recommendation device,

One or more processors; and

A memory containing instructions configured to cause the one or more processors to execute operations.

The above operations are:

Creating a plurality of reference scenes by dividing the collected images into scene units;

analyzing the plurality of reference scenes to extract feature information and assigning different types of tags to the plurality of reference scenes based on this; and

Including storing a plurality of reference scenes to which the tags are assigned in a reference scene database,

Assigning different types of tags to the plurality of reference scenes,

extracting feature information of an object included in the reference scene, extracting a feature descriptor expressing the feature information of the object as a vector value, and assigning an object attribute tag to the reference scene according to the feature descriptor;

applying the reference scene to a scene type analysis model to extract the type of situation expressed in the reference scene, and assigning a situation attribute tag to the reference scene according to the type of situation; and

Including extracting a highlight portion from the collected video and assigning a highlight attribute tag to a reference scene corresponding to the highlight portion among a plurality of reference scenes stored in the reference scene database.

Reference scene recommendation device.
According to clause 5,

The operation is,

Receiving a reference scene recommendation request message from the automatic video creation device containing keywords consisting of tokens with different weights;

Extracting a reference scene to which a tag matching the morpheme value of the token is assigned among a plurality of tags of the reference scene data,

If the tag of the extracted reference scene matches the word of the token, further comprising providing the extracted reference scene to the automatic video generating device,

Reference scene recommendation device.
According to clause 6,

The operation is,

Receiving a reference scene recommendation request message including keywords composed of the tokens with different weights from the automatic video generating device,

Extracting a reference scene assigned a tag that does not match the morpheme value of the token from among the plurality of tags in the reference scene data,

Calculating a similarity ratio between each of a plurality of tags assigned to the extracted reference scene and the word of the token,

Further comprising providing, to the automatic video generating device, a reference scene to which a tag whose similarity rate is greater than or equal to a certain rate is assigned.

Reference scene recommendation device.
In clause 7,

To calculate the similarity ratio,

Comparing letters constituting a plurality of tags assigned to the extracted reference scene and letters constituting a word of the token to calculate the number of matching letters;

Comparing the number of strings corresponding to the plurality of tags and the number of strings corresponding to words of the token and selecting a longer number of strings, and

Including calculating the similarity ratio, which represents the ratio of the number of matching characters to the number of selected strings,

Reference scene recommendation device.