KR102459775B1 - Automatic editing method, apparatus and system for artificial intelligence-based vidieo content production - Google Patents

Abstract

An apparatus according to an embodiment receives a production file including a video clip for producing content, an audio file, and subtitle text corresponding to the audio file from a user terminal of a user, creates a content style using the production file, applies the content style to the production file to create a content draft, transmits the content style and the content draft to the user terminal, provides the user terminal with an interface for editing the content style, and applies the content style edited by the interface to the production file to create a final content.

Description

AUTOMATIC EDITING METHOD, APPARATUS AND SYSTEM FOR ARTIFICIAL INTELLIGENCE-BASED VIDIEO CONTENT PRODUCTION}

The following embodiments relate to a technology for providing an automatic editing method for artificial intelligence-based video content production.

As offline activities are restricted and online activities are active due to the impact of COVID-19, more and more people are watching video content. In addition, there is an increasing trend in the number of people not only watching video content, but also planning and producing video content and uploading it to a video platform to generate revenue.

In order to produce video content, the user must plan the subject of the video content, cut and edit the length and sequence of the captured video accordingly, and perform cumbersome tasks such as the process of adding visual elements by inserting subtitles and effects. do.

Accordingly, there is a need to develop a technology for automatically editing video content by generating a content style including a subtitle style, a cut editing style, and a color style from a content production file.

Republic of Korea Patent Publication No. 10-2018-0038318 (published on April 16, 2018) Republic of Korea Patent Publication No. 10-2020-0099427 (published on August 24, 2020) Republic of Korea Patent Publication No. 10-2019-0105533 (published on September 17, 2019) Republic of Korea Patent Publication No. 10-2020-0071031 (published on June 18, 2020)

Embodiments are intended to provide an automatic editing method for video content production based on artificial intelligence.

Embodiments seek to create a subtitle style from an audio file and subtitle text.

Embodiments intend to create a cut editing style and a color style for video content production.

According to one embodiment, there is provided a method performed by an apparatus, comprising: receiving a production file including a video clip, an audio file, and subtitle text corresponding to the audio file for producing content from a user terminal of a user; generating a content style using the production file; creating a content draft by applying the content style to the production file; sending the content style and the content draft to the user terminal; providing an interface capable of editing the content style to the user terminal; and applying the content style edited by the interface to the production file to generate final content.

The content style includes a caption style, a cut edit style, and a color style, the caption style is generated from the audio file and caption text, the cut edit style is generated from the video clip, and the color style is the caption text can be generated from

The generating of the subtitle style may include: comparing the intensity of a sound in the audio file with a preset first reference intensity; When the intensity of the sound is less than the first reference intensity, setting the subtitle text corresponding to the sound less than the first reference intensity as a first closed caption style set to a first font, a first size, and a first color; ; comparing the sound intensity with a preset second reference intensity when the intensity of the sound is greater than the first reference intensity; setting a caption text corresponding to a sound lower than the second reference intensity as a second closed caption style set to a second font, a second size, and a second color when the intensity of the sound is lower than the second reference intensity; ; determining whether the loudness of the sound greater than the second reference intensity is maintained for a preset reference period when the intensity of the sound is greater than the second reference intensity; When the intensity of the sound is maintained for the reference period, the subtitle text corresponding to the sound equal to or greater than the second reference intensity is set as a third closed caption style set to a third font, a third size, and a third color during the reference period to do; setting, as an effect caption style, a caption text corresponding to a sound equal to or greater than the second reference intensity during the reference period when the volume of the sound is not maintained for the reference period; and generating the caption style based on the first caption style, the second caption style, the third caption style, and the effect caption style.

The generating of the cut-editing style may include: extracting a background from the video clip according to whether a pattern of a predefined similarity range is repeated within the entire video clip beyond a predefined criterion; obtaining a main object of the video clip by separating the extracted background from the video clip; dividing the video clip into a plurality of clips; For each of the plurality of clips, if the movement of the main object is greater than or equal to a preset reference change value, the clip is classified as a dynamic clip. classifying as; setting a first cut-editing style for applying a first time weight to the dynamic clip; setting a second cut editing style in which a second time weight smaller than the first time weight is applied to the static clip; and generating the cut-editing style based on the first cut-editing style and the second cut-editing style.

The generating of the color style may include: extracting, as main keywords, words mentioned more than a preset reference number from the subtitle text; classifying the main keyword as a positive keyword or a negative keyword; setting the video clip as a first color style when the main keyword is classified as a positive keyword; setting the video clip as a second color style when the main keyword is classified as a negative keyword; and generating the color style based on the first color style and the second color style.

A method performed by the device, the method comprising: obtaining, for a channel to which the user uploads the content during a preset reference period, the number of visits to the channel and the visit maintenance period for each country; generating a visit score for each country based on the number of visits and the visit maintenance period; obtaining, by country, a content viewing time for viewing the content uploaded to the channel during the reference period; generating a content viewing score for each country based on the content viewing time; calculating a channel interest level for each country by adding up the country-specific visit score and the country-specific content viewing score; acquiring information about the main age group that watched the content for each country; comparing the main age group information with preset target age group information; assigning weights to countries in which the main age group information and the target age group information match, updating and storing the channel interest level for each country; setting the country with the highest channel interest as a target country by arranging the channel interest for each country in the order of highest; and setting a language corresponding to the target country as an additional subtitle language.

In the method performed by the apparatus, the method comprising: extracting a video image of a first time point, which is a time point at which a last video clip among the video clips, appears as a first image in the final content; dividing an area occupied by a background in the first image into a first area, and extracting a 1-1 image by dividing a portion having the first area in the first image; if it is confirmed that there is a performer in the 1-1 image, dividing an area occupied by the performer in the 1-1 image into a second region; As a result of replaying and analyzing the final content in reverse chronological order from the first time point, if it is confirmed that there is no performer in the second area at the 1-1 time point, the video image of the 1-1 time point in the final content extracting as a second image; extracting a 2-1 image by dividing a portion having the second region from the second image; and replacing the portion having the second region in the 1-1 image with the 2-1 image.

The apparatus according to an embodiment may be controlled by a computer program stored in the medium to execute the method of any one of the above-described methods in combination with hardware.

Embodiments may provide an automatic editing method for video content production based on artificial intelligence.

Embodiments may create a subtitle style from an audio file and subtitle text.

Embodiments may generate a cut-editing style and a color style for video content production.

1 is a diagram for explaining the configuration of a system according to an embodiment.
2 is a flowchart for explaining a process of providing an automatic editing method for producing an AI-based video content according to an embodiment.
3 is a flowchart illustrating a process of generating a caption style according to an exemplary embodiment.
4 is a flowchart illustrating a process of creating a cut editing style according to an exemplary embodiment.
5 is a flowchart illustrating a process of generating a color style according to an exemplary embodiment.
6 is a flowchart illustrating a process of setting an additional subtitle language according to an exemplary embodiment.
7 is a diagram for explaining learning of a neural network according to an embodiment.
8 is a flowchart illustrating a process of deleting a performer from a video image of a first viewpoint according to an exemplary embodiment.
FIG. 9 is a view for explaining a 1-1 image in which a region with a performer is replaced with another image according to an exemplary embodiment.
10 is an exemplary diagram of a configuration of an apparatus according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, since various changes may be made to the embodiments, the scope of the patent application is not limited or limited by these embodiments. It should be understood that all modifications, equivalents and substitutes for the embodiments are included in the scope of the rights.

Specific structural or functional descriptions of the embodiments are disclosed for purposes of illustration only, and may be changed and implemented in various forms. Accordingly, the embodiments are not limited to a specific disclosure form, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical spirit.

Although terms such as first or second may be used to describe various elements, these terms should be interpreted only for the purpose of distinguishing one element from another. For example, a first component may be termed a second component, and similarly, a second component may also be termed a first component.

When a component is referred to as being “connected” to another component, it may be directly connected or connected to the other component, but it should be understood that another component may exist in between.

Terms used in the examples are used for the purpose of description only, and should not be construed as limiting. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present specification, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It is to be understood that this does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiment belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

In addition, in the description with reference to the accompanying drawings, the same components are assigned the same reference numerals regardless of the reference numerals, and the overlapping description thereof will be omitted. In the description of the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the embodiment, the detailed description thereof will be omitted.

The embodiments may be implemented in various types of products, such as personal computers, laptop computers, tablet computers, smart phones, televisions, smart home appliances, intelligent cars, kiosks, wearable devices, and the like.

In an embodiment, the artificial intelligence (AI) system is a computer system that implements human-level intelligence, and is a system in which a machine learns and makes decisions on its own, unlike an existing rule-based smart system. The more the AI system is used, the better the recognition rate and the more accurate understanding of user preferences, and the existing rule-based smart systems are gradually being replaced by deep learning-based AI systems.

Artificial intelligence technology consists of machine learning and element technologies using machine learning. Machine learning is an algorithm technology that categorizes/learns characteristics of input data by itself, and element technology uses machine learning algorithms such as deep learning to simulate functions such as cognition and judgment of the human brain. It consists of technical fields such as understanding, reasoning/prediction, knowledge expression, and motion control.

The various fields where artificial intelligence technology is applied are as follows. Linguistic understanding is a technology for recognizing and applying/processing human language/text, and includes natural language processing, machine translation, dialogue system, question and answer, and speech recognition/synthesis. Visual understanding is a technology for recognizing and processing objects like human vision, and includes object recognition, object tracking, image search, human recognition, scene understanding, spatial understanding, image improvement, and the like. Inferential prediction is a technology for logically reasoning and predicting by judging information, and includes knowledge/probability-based reasoning, optimization prediction, preference-based planning, and recommendation. Knowledge expression is a technology that automatically processes human experience information into knowledge data, and includes knowledge construction (data generation/classification) and knowledge management (data utilization). Motion control is a technology for controlling autonomous driving of a vehicle and movement of a robot, and includes motion control (navigation, collision, driving), manipulation control (action control), and the like.

In general, in order to apply the machine learning algorithm to real life, learning is performed in the Trial and Error method due to the characteristics of the basic methodology of machine learning. In particular, deep learning requires hundreds of thousands of iterations. It is impossible to execute this in the actual physical external environment, so instead, the actual physical external environment is implemented on a computer and learning is performed through simulation.

1 is a diagram for explaining the configuration of a system according to an embodiment.

Referring to FIG. 1 , a system according to an embodiment may include a user terminal 10 and a device 30 capable of communicating with each other through a communication network.

First, the communication network may be configured regardless of its communication mode, such as wired and wireless, and may be implemented in various forms so that communication between a server and a server and communication between a server and a terminal are performed.

The user terminal 10 may be a terminal used by a user who wants to produce image content according to the present invention. The user terminal 10 may be a desktop computer, a notebook computer, a tablet, a smart phone, or the like. For example, as shown in FIG. 1 , the user terminal 10 may be a smartphone, and may be employed differently according to embodiments.

The user terminal 10 may be configured to perform all or part of an arithmetic function, a storage/referencing function, an input/output function, and a control function of a typical computer. The user terminal 10 may be configured to communicate with the device 30 in a wired or wireless manner.

The user terminal 10 is connected to a web page operated by a person or organization providing a service using the device 30, or an application developed and distributed by a person or organization providing a service using the device 30. can be installed. The user terminal 10 may be linked with the device 30 through a web page or an application.

The user terminal 10 may access the device 30 through a web page or an application provided by the device 30 .

Expressions in the singular in the claims may be understood to include the plural. For example, a user in a claim may refer to one user or more than one user.

The device 30 may be a self-server owned by a person or organization that provides services using the device 30, a cloud server, or a peer-to-peer (p2p) set of distributed nodes. may be The device 30 may be configured to perform all or part of an arithmetic function, a storage/referencing function, an input/output function, and a control function of a typical computer.

The device 30 may be configured to communicate with the user terminal 10 by wire or wireless, and may control the operation of the user terminal 10 and control which information to display on the screen of the user terminal 10 . .

Meanwhile, although only the user terminal 10 is illustrated in FIG. 1 for convenience of explanation, the number of terminals may vary according to embodiments. As long as the processing capacity of the device 30 allows, the number of terminals is not particularly limited.

The device 30 may provide an automatic editing method for artificial intelligence-based video content production. According to an embodiment, a database may be provided in the device 30 , and the database is not limited thereto, and a database may be configured separately from the device 30 . The device 30 may include a number of artificial neural networks for performing machine learning algorithms.

2 is a flowchart for explaining a process of providing an automatic editing method for producing an AI-based video content according to an embodiment.

Referring to FIG. 2 , first, in step S201 , the device 30 may receive a production file for producing content from the user terminal 10 . In this case, the production file is a file used to produce content, and may include, for example, a video clip, an audio file, and subtitle text corresponding to the audio file.

In step S202 , the device 30 may generate a content style using the production file. According to an embodiment, the device 30 may generate a content style by using a video clip, an audio file, and caption text corresponding to the audio file. In this case, the content style may include a caption style for the font, size and color of the subtitle, a cut editing style for the time of the video clip, and a color style for the color of the video clip.

A detailed description of a process of generating a subtitle style, a process of generating a cut editing style, and a process of generating a color style will be described later with reference to FIGS. 3, 4, and 5, respectively.

In step S203 , the device 30 may create a content draft by applying a content style to the production file.

Specifically, the device 30 generates a content style including a caption style, a cut-editing style, and a color style with respect to the video clip, the audio file, and the caption text corresponding to the audio file, and stores the generated content style in the production file. Can be applied to create draft content.

In step S204 , the device 30 may transmit the content style and content draft to the user terminal 10 .

In step S205 , the device 30 may provide the user terminal 10 with an interface capable of editing the content style.

According to an embodiment, the device 30 may generate an interface capable of editing a content style, and transmit the generated interface to the user terminal 10 . The interface may include a page generated to enable editing of a subtitle style for the font, size and color of the subtitle, a cut editing style for the time of the video clip, and a color style for the color of the video clip.

In step S206 , the device 30 may generate the final content by applying the content style edited by the interface to the production file.

According to an embodiment, the device 30 may transmit an interface capable of editing the content style to the user terminal 10 , and receive the content style edited by the interface from the user terminal 10 , and receive the edited content style from the user terminal 10 . You can apply content styles to production files to create the final content.

3 is a flowchart illustrating a process of generating a caption style according to an exemplary embodiment.

Referring to FIG. 3 , first, in step S301 , the device 30 may compare the intensity of a sound in the audio file with a preset first reference intensity. In this case, the first reference intensity may be set differently depending on the embodiment.

In step S302 , when the sound intensity is lower than the first reference intensity, in step S303 , the device 30 may set the subtitle text corresponding to the sound lower than the first reference intensity as the first closed caption style. In this case, the first word caption style is mainly set to be displayed at the bottom of the video clip, and as a caption input according to speech sound, the caption text may be set to a first font, a first size, and a first color.

For example, when the first word caption style is set to Gulim, 10pt, and black, and the caption text is set as the first closed caption style, the device 30 may set the caption text to Gulim, 12pt, and black.

In step S302 , if the sound intensity is greater than the first reference intensity, in step S304 , the device 30 may compare the sound intensity with a preset second reference intensity. In this case, the second reference intensity may be set differently according to embodiments.

In step S305 , when the sound intensity is less than the second reference intensity, the device 30 may set the subtitle text corresponding to the sound less than the second reference intensity as the second closed caption style. In this case, the second word caption style is mainly set to be displayed at the bottom of the video clip, and as a caption input according to speech sound, the caption text may be set to a second font, a second size, and a second color.

For example, when the second closed caption style is set to arch font, 12pt, and blue, and the subtitle text is set to the second closed caption style, the device 30 may set the subtitle text to archery, 12pt, and blue.

In step S306 , if the sound intensity is greater than the second reference intensity, the device 30 may determine whether the loudness greater than the second reference intensity is maintained for a preset reference period. In this case, the reference period may be set differently depending on the embodiment.

In operation S307 , if the sound intensity is maintained for the reference period, the device 30 may set the subtitle text corresponding to the sound having the second reference intensity or higher during the reference period as the third closed caption style. In this case, the third word caption style is mainly set to be displayed at the bottom of the video clip, and as a caption input according to speech sound, the caption text may be set to a third font, a third size, and a third color.

For example, when the third closed caption style is set to a bold font, 14 pt, and green, and the subtitle text is set to the third closed caption style, the device 30 may set the subtitle text to a raised font, 14 pt, or green.

In step S308 , if the sound intensity is not maintained for the reference period, the device 30 may set the caption text corresponding to the sound equal to or greater than the second reference intensity level as the effect caption style during the reference period. In this case, the effect caption style may be a caption style mainly set to be displayed in the middle of a video clip, and used to add fun elements of content, such as onomatopoeia, mimetic words, exclamation words, and sound effects.

For example, the effect subtitle style may include, but is not limited to, font, size, color, animation effect, and the like of subtitle text.

In step S309 , the device 30 may generate a caption style based on the first word caption style, the second word caption style, the third word caption style, and the effect caption style.

According to one embodiment, the subtitle style may be created from an audio file and subtitle text.

4 is a flowchart illustrating a process of creating a cut editing style according to an exemplary embodiment.

Referring to FIG. 4 , first, in step S401 , the device 30 may extract a background from the video clip. According to an embodiment, the device 30 may extract a background from a video clip according to whether a pattern of a predefined similarity range is repeated beyond a predefined criterion within the entire video clip.

In step S402 , the device 30 may obtain the main object of the video clip by separating the extracted background from the video clip.

In step S403 , the device 30 may divide the video clip into a plurality of clips. For example, if the video clip has a total of 100 seconds, the device 30 may divide the video clip by 10 seconds to divide the video clip into a plurality of 10 clips.

In step S404 , the device 30 may compare the movement of the main object with a preset reference change value for each of the plurality of clips. In this case, the reference change value may be set differently depending on the embodiment.

In step S405 , when the movement of the main object is equal to or greater than the reference change value, the device 30 may classify the corresponding clip as a dynamic clip.

In this case, the dynamic clip may refer to an image clip classified by determining that the movement of the main object is changing or dynamic when the movement of the main object is equal to or greater than the reference change value for each of the plurality of image clips.

In step S406 , the device 30 may set a first cut editing style for applying a first time weight to the dynamic clip. In this case, the first time weight may be set differently depending on the embodiment.

In step S407 , when the movement of the main object is equal to or less than the reference change value, the apparatus 30 may classify the corresponding clip as a static clip.

In this case, the static clip may mean an image clip classified by determining that the movement of the main object is continuously maintained without changing when the movement of the main object with respect to each of the plurality of image clips is less than or equal to the reference change value.

In step S408 , the device 30 may set a second cut editing style for applying a second time weight to the static clip. In this case, the second time weight may be set to be smaller than the first time weight, and may be set differently according to embodiments.

According to an embodiment, the device 30 may set a first cut-editing style in which a first time weight is applied to a dynamic clip, and set a second cut-editing style in which a second time weight is applied to a static clip. That is, the device 30 can reduce the boredom of video content and create a dynamic and interesting cut-editing style by increasing the time weight of the dynamic clip and decreasing the time weight of the static clip. .

In step S409 , the device 30 may generate a cut edit style based on the first cut edit style and the second cut edit style.

According to an embodiment, the cut-editing style may be generated from a video clip.

5 is a flowchart illustrating a process of generating a color style according to an exemplary embodiment.

Referring to FIG. 5 , first, in step S501 , the device 30 may extract a word mentioned more than a reference number from the subtitle text as a main keyword. In this case, the reference number may be set differently depending on the embodiment.

In step S502, the device 30 may classify the main keyword as a positive keyword or a negative keyword.

According to an embodiment, words classified as negative or positive words may be pre-stored in the database of the device 30 , and when the extracted main keyword is a word classified as a positive word, the device 30 affirms the main keyword It may be classified as a keyword, and when the extracted main keyword is a word classified as a negative word, the device 30 may classify the main keyword as a negative keyword.

In step S503 , when the main keyword is classified as a positive keyword, the device 30 may set the video clip as the first color style.

According to an embodiment, when the main keyword is classified as a positive keyword and the first color style is a warm color, the device 30 may adjust the color temperature so that the video clip has a warm color.

In step S504 , when the main keyword is classified as a negative keyword, the device 30 may set the video clip as the second color style.

According to an embodiment, when the main keyword is classified as a negative keyword and the second color style is a cool color, the device 30 may adjust the color temperature so that the video clip has a cool color.

In operation S505 , the device 30 may generate a color style based on the first color style and the second color style.

According to an embodiment, the color style may be generated from subtitle text.

6 is a flowchart illustrating a process of setting an additional subtitle language according to an exemplary embodiment.

Referring to FIG. 6 , first, in step S601 , the device 30 may acquire the number of visits and the visit maintenance period of channels for each country with respect to a channel through which a user uploads content during a preset reference period. In this case, the reference period may be set differently depending on the embodiment.

According to an embodiment, the device 30 may obtain channel information including a channel name, number of subscribers, inquiry history and visit history of a channel to which the user uploads content, and the number of visits and visits to the channel by country maintenance period can be obtained. For example, if the reference period is 3 months, the device 30 may obtain a visit retention period for how many visits to a channel by country for 3 months and how many visits are maintained for 3 months. .

In step S602 , the device 30 may generate a visit score for each country based on the number of visits and the visit maintenance period.

For example, if the reference period is 3 months, the number of visits to country A is 60, and the visit maintenance period is 30 days, the device 30 may generate a visit score of country A as 90, and country B If the number of visits is 55 and the visit maintenance period is 20 days, the device 30 may generate a visit score of country B as 75 points.

In step S603 , the device 30 may acquire the content viewing time for viewing the content uploaded to the channel for each country during the reference period.

In step S604 , the device 30 may generate a content viewing score for each country based on the content viewing time.

For example, if the reference period is 3 months and country A's content viewing time is 30 hours, the device 30 may generate a visit score of 30 points for country A, and country B's content viewing time is 25 In the case of time, the device 30 may generate the content viewing score of country B as 25 points.

In step S605 , the device 30 may calculate the channel interest for each country by adding up the visit score for each country and the content viewing score for each country.

For example, if country A's visit score is 90 and the content viewing score is 30, the device 30 may calculate the channel interest of country A as 120 points, country B's visit score is 75, and the content When the viewing score is 25, the device 30 may calculate the channel interest of country B as 100 points.

In step S606 , the device 30 may obtain information about the main age group that watched the content for each country.

For example, the device 30 may acquire main age group information of country A in their 20s and main age group information of country B in their 50s.

In step S607 , the device 30 may compare the main age group information and the preset target age group information. In this case, the target age group information may mean an age group set as a target when the user creates content, and may be set differently according to embodiments.

In step S608 , the device 30 may update and store the channel interest for each country by giving weights to countries in which the main age group information and the target age group information match.

For example, when the main age group information of country A and the target age group information match in their 20s, the device 30 may update and store the channel interest level by weighting the channel interest level of country A. In this case, the weight may be set differently depending on the embodiment.

In step S609 , the device 30 may list the channel interest by country in the order of highest, and set the country having the highest channel interest as the target country.

In step S610 , the device 30 may set the language corresponding to the target country as the additional subtitle language. For example, if the target country is country A, the device 30 may set a language corresponding to country A as an additional subtitle language.

7 is a diagram for explaining learning of a neural network according to an embodiment.

As shown in FIG. 7 , the device 30 may train the neural network 123 to form a content style from a production file received from the user terminal 10 . Also, the device 30 may train the neural network 123 with a history from production files including video clips, audio files, and subtitle text to the formation of final content. According to an embodiment, the device 30 may be a separate entity other than the server, but is not limited thereto.

The neural network 123 includes an input layer 121 to which training samples are input and an output layer 125 to output training outputs, and may be trained based on a difference between the training outputs and labels. Here, the labels may be defined based on a video clip, an audio file, a production file including subtitle text, and a content style corresponding to the production file. The neural network 123 is connected to a group of a plurality of nodes, and is defined by weights between the connected nodes and an activation function that activates the nodes.

The device 30 may train the neural network 123 by using a Gradient Decent (GD) technique or a Stochastic Gradient Descent (SGD) technique. The device 30 may use a loss function designed by the outputs and labels of the neural network 123 .

The device 30 may calculate the training error using a predefined loss function. The loss function may be predefined with a label, an output, and a parameter as input variables, where the parameter may be set by weights in the second artificial neural network 200 . For example, the loss function may be designed in a Mean Square Error (MSE) form, an entropy form, or the like, and various techniques or methods may be employed in an embodiment in which the loss function is designed.

The apparatus 30 may find weights affecting the training error using a backpropagation technique. Here, the weights are relationships between nodes in the neural network 123 . The apparatus 30 may use the SGD technique using labels and outputs to optimize the weights found through the backpropagation technique. For example, the apparatus 30 may update the weights of the loss function defined based on the labels, outputs, and weights using the SGD technique.

According to an embodiment, the device 30 may generate training input signals based on training video clips, training audio files and training caption texts. Device 30 may generate output signals from a training input signal.

According to an embodiment, the neural network 123 may be trained by minimizing training errors generated based on differences in training video clips, training audio files, training caption texts, output signals, and training content styles. .

8 is a flowchart for explaining a process of deleting a performer from a video image of a first viewpoint according to an embodiment, and FIG. 9 is a 1-1 image in which a region with a performer is replaced with another image according to an embodiment. It is a drawing for explaining.

Referring to FIG. 8 , first, in step S801 , the device 30 may extract, as a first image, a video image of a first point of view, which is a point at which the last video clip among video clips in the final content. That is, the device 30 may capture the image of the first viewpoint in the final content, and extract the captured image as the first image.

In step S802 , the device 30 may extract a 1-1 image by dividing a portion having the first region in the first image. The device 30 may classify an area occupied by the background in the first image as the first area.

In step S803 , the device 30 may check whether it is recognized that there is a performer in the 1-1 image. That is, the device 30 may check whether a human-shaped object is recognized in the 1-1 image other than the background on the 1-1 image, and may determine whether a performer is present in the 1-1 image.

If it is confirmed in step S803 that there is a performer in the 1-1 image, in step S804 , the device 30 may classify an area occupied by the performer in the 1-1 image as a second region. In this case, the device 30 may distinguish the second region from the 1-1 image through the same method as the method for distinguishing the first region from the first image.

In step S805 , the device 30 may analyze the final content by playing it back in reverse chronological order from the first time point. At this time, since the final content is reproduced and analyzed in reverse chronological order, the performer moves in reverse with respect to the movement line. That is, if the final content is played backwards, the performer moves in the opposite direction.

In step S806 , the device 30 reproduces and analyzes the final content in reverse chronological order from the first time point, and as a result, it can be confirmed that there are no performers in the second area at the 1-1 time point.

That is, the final content is played back in the order of the 1-1 time point and the first time point, and the device 30 reproduces the final content in the reverse chronological order from the first time point to monitor the reverse movement of the performer in the second area. In addition, a time point at which it is confirmed that the performer within the second area leaves the second area may be set as the 1-1 time point.

In step S807 , the device 30 may extract the video image of the 1-1 view from the final content as the second image. That is, the device 30 may capture the image of the 1-1 view from the final content, and extract the captured image as the second image.

In step S808 , the device 30 may extract a 2-1 image by dividing a portion having the second region in the second image.

In step S809 , the device 30 may replace the portion having the second region in the 1-1 image with the 2-1 image.

That is, as shown in (a) of FIG. 9 , the device 30 can confirm that there is a performer in the 1-1 image 700 , and the area occupied by the performer in the 1-1 image 700 . can be divided into a second region. In this case, the device 30 may set the second area so that the second area is divided into a rectangular shape.

Thereafter, the device 30 deletes the portion having the second region from the 1-1 image, inserts the 2-1 image at the deleted position, and combines the 1-1 image with the 2-1 image. A 1-1 image may be generated. That is, in the 1-1 image, the part with the second area is replaced with the 2-1 image, the performer in the 1-1 image is deleted, and a background is added in the deleted place, and the 1-1 image In , the performer may no longer be recognized.

10 is an exemplary diagram of a configuration of an apparatus 30 according to an embodiment.

Device 30 according to one embodiment includes a processor 31 and a memory 32 . The device 30 according to an embodiment may be the above-described server or terminal. The processor may include at least one of the devices described above with reference to FIGS. 1 to 9 , or may perform at least one method described above with reference to FIGS. 1 to 9 . The memory 32 may store information related to the above-described method or may store a program in which the above-described method is implemented. Memory 32 may be volatile memory or non-volatile memory.

The processor 31 may execute a program and control the device 30 . The code of the program executed by the processor 31 may be stored in the memory 32 . The device 30 may be connected to an external device (eg, a personal computer or a network) through an input/output device (not shown) and exchange data.

The embodiments described above may be implemented by a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the apparatus, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Software may comprise a computer program, code, instructions, or a combination of one or more of these, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

As described above, although the embodiments have been described with reference to the limited drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (3)

A method performed by an apparatus comprising:
Receiving a production file including a video clip, an audio file, and subtitle text corresponding to the audio file for producing content from a user terminal of a user;
generating a content style using the production file;
creating a content draft by applying the content style to the production file;
sending the content style and the content draft to the user terminal;
providing an interface capable of editing the content style to the user terminal; and
Including; applying the content style edited by the interface to the production file to generate the final content;
The content style is
Includes subtitle style, cut editing style and color style,
The subtitle style is
generated from the audio file and subtitle text;
The cut editing style is,
generated from the video clip;
The color style is
generated from the subtitle text;
The step of creating the cut editing style includes:
extracting a background from a video clip according to whether a pattern of a predefined similarity range is repeated beyond a predefined criterion within the entire video clip;
obtaining a main object of the video clip by separating the extracted background from the video clip;
dividing the video clip into a plurality of clips;
For each of the plurality of clips, if the movement of the main object is greater than or equal to a preset reference change value, the clip is classified as a dynamic clip. classifying as;
setting a first cut-editing style for applying a first time weight to the dynamic clip;
setting a second cut editing style in which a second time weight smaller than the first time weight is applied to the static clip; and
generating the cut-editing style based on the first cut-editing style and the second cut-editing style;
including,
The step of creating the color style includes:
extracting words mentioned more than a preset reference number from the subtitle text as main keywords;
classifying the main keyword as a positive keyword or a negative keyword;
setting the video clip as a first color style when the main keyword is classified as a positive keyword;
setting the video clip as a second color style when the main keyword is classified as a negative keyword; and
generating the color style based on the first color style and the second color style;
containing
Automatic editing method for creating video contents based on artificial intelligence. According to claim 1,
The step of generating the subtitle style includes:
comparing the intensity of the sound in the audio file with a preset first reference intensity;
When the intensity of the sound is less than the first reference intensity, setting the subtitle text corresponding to the sound less than the first reference intensity as a first closed caption style set to a first font, a first size, and a first color; ;
comparing the sound intensity with a preset second reference intensity when the intensity of the sound is greater than the first reference intensity;
setting a caption text corresponding to a sound lower than the second reference intensity as a second closed caption style set to a second font, a second size, and a second color when the intensity of the sound is lower than the second reference intensity; ;
determining whether the loudness of the sound greater than the second reference intensity is maintained for a preset reference period when the intensity of the sound is greater than the second reference intensity;
When the intensity of the sound is maintained for the reference period, the subtitle text corresponding to the sound equal to or greater than the second reference intensity is set as a third closed caption style set to a third font, a third size, and a third color during the reference period to do;
setting, as an effect caption style, a caption text corresponding to a sound equal to or greater than the second reference intensity during the reference period when the volume of the sound is not maintained for the reference period; and
generating the caption style based on the first caption style, the second caption style, the third caption style, and the effect caption style;
containing
Automatic editing method for creating video contents based on artificial intelligence. delete

Images