KR102281298B1 - System and method for video synthesis based on artificial intelligence - Google Patents

Abstract

A method for synthesizing a video, according to an exemplary embodiment of the present invention, may comprise the steps of: obtaining input data that at least partially defines a video; extracting a series of instructions and content data to be included as part of the video from the input data based on predefined rules; generating structured intermediate data from content data and a series of instructions, based on at least one machine learning model learned by a plurality of instruction samples; and generating the video based on the intermediate data. Rules may include rules for identifying content data and a set of instructions. A user can easily synthesize the video.

Description

SYSTEM AND METHOD FOR VIDEO SYNTHESIS BASED ON ARTIFICIAL INTELLIGENCE

The technical idea of the present invention relates to video synthesis, and more particularly, to a system and method for video synthesis based on artificial intelligence.

Due to the development of wired and wireless communication technology, high-capacity data transmission is possible, and the data storage capacity is increased due to the development of storage technology. is increasing significantly. While acquiring information through video provides high convenience compared to acquiring information from text or images, the producer of information, that is, the producer of video, may require more effort than text or images to produce a video. there is.

The technical idea of the present invention provides a system and method for video synthesis based on artificial intelligence that allows a user to easily synthesize a video.

In order to achieve the above object, a method for synthesizing a video according to an aspect of the technical idea of the present invention includes the steps of obtaining input data that at least partially defines a video, based on predefined rules, extracting a series of instructions from the input data and content data to be included as part of a moving picture, based on at least one machine learning model learned by a plurality of instruction samples, a structured intermediate from the content data and a series of instructions generating data, and generating a moving picture based on the intermediate data, and the rules may include rules for identifying content data and a set of instructions.

According to an exemplary embodiment of the present invention, extracting the set of instructions and content data may include extracting a text-based instruction based on rules, wherein extracting the text-based instruction includes: extracting a first instruction for instructing insertion of content, extracting a second instruction for controlling a flow of a moving picture, and extracting a third instruction written in natural language; may include at least one of

According to an exemplary embodiment of the present invention, the generating of the intermediate data comprises: when the first instruction corresponds to the insertion of the subtitle, dividing the subtitle into at least one part, and generating the intermediate data including the at least one part generating, when the first instruction corresponds to the insertion of an image, obtaining information about the position and/or size of the image, and generating intermediate data including the obtained information, and the first instruction is When it corresponds to insertion, the method may include at least one of determining at least one of a voice, a background sound, and an effect sound, and generating intermediate data including information on the determined sound.

According to an exemplary embodiment of the present invention, generating the intermediate data may include, in response to the second instruction, generating timing information of a series of frames, and generating intermediate data including the timing information. there is.

According to an exemplary embodiment of the present invention, generating the intermediate data may include, in response to the third instruction, generating at least one of the first instruction and the second instruction by processing the natural language.

According to an exemplary embodiment of the present invention, extracting the set of instructions and content data may include extracting an action-based instruction based on rules, wherein extracting the action-based instruction includes: extracting a shape included in the action layer, recognizing a symbol and/or language from the shape, based on at least one machine learning model, and generating at least one text-based instruction based on the symbol and/or language It may include the step of generating.

According to an exemplary embodiment of the present disclosure, extracting the set of instructions and content data may include extracting an action-based instruction based on rules, wherein extracting the action-based instruction includes: based on the at least one machine learning model, identifying an action-based indication and extracting a shape from the identified action-based indication, based on the at least one machine learning model, recognizing a symbol and/or language from the shape; , and generating at least one text-based indication based on the symbol and/or language.

According to an exemplary embodiment of the present invention, obtaining the input data may include providing an input interface to the user, and receiving the input data through the input interface, and providing the input interface. corresponds to a scene that is a part of the playback time of the moving picture, displaying a scene area in which content data corresponding to the scene and a series of instructions are displayed, and a flow including at least a part of the series of scenes ) displaying the region.

According to an exemplary embodiment of the present invention, extracting the series of instructions and content data may include extracting a fourth instruction indicating a timing-dependent effect through the flow region.

According to an aspect of the present invention, a method for synthesizing a video includes providing a first video to multiple users, receiving a first feedback on the first video from multiple users, and a first video Receiving first input data that at least partially modifies from at least some of multiple users, synthesizing a second video by modifying the first video based on the first input data, and converting the second video to multiple users It may include the step of providing to them.

According to an exemplary embodiment of the present invention, a method for synthesizing a video includes: receiving second feedback on a second video from multiple users; The method may further include receiving from at least some of users, synthesizing a third video by modifying the second video based on the second input data, and providing the third video to multiple users.

According to the system and method according to the technical idea of the present invention, it is possible for a user to intuitively input information required for video generation, and accordingly, it is possible to not be limited to video generation through shooting, and the convenience of video generation can be improved. can

In addition, according to the system and method according to the technical idea of the present invention, a user can use a natural language input as well as a structured input based on rules, and thus anyone can easily create a moving picture.

In addition, according to the system and method according to the technical idea of the present invention, it is possible to edit a video by multiple users, and collective generation of a video by multiple users can be utilized in various business models.

Effects that can be obtained in the embodiments of the present invention are not limited to the effects mentioned above, and other effects not mentioned are common in the art to which the present invention belongs from the description of the embodiments of the present invention below. It can be clearly derived and understood by those with knowledge. That is, unintended effects of carrying out the present invention may also be derived by those of ordinary skill in the art from the embodiments of the present invention.

1 is a block diagram showing a video synthesis system according to an exemplary embodiment of the present invention.
Fig. 2 is a flowchart showing an example of a method for synthesizing a moving picture according to an exemplary embodiment of the present invention.
3 is a flowchart illustrating an example of a method for synthesizing a moving picture according to an exemplary embodiment of the present invention.
4A shows an example of a display provided to a user for video synthesis according to an exemplary embodiment of the present invention, and FIG. 4B shows an exemplary frame of a video synthesized according to an exemplary embodiment of the present invention.
5A shows an example of a display provided to a user for video synthesis according to an exemplary embodiment of the present invention, and FIG. 5B shows exemplary frames of a video synthesized according to an exemplary embodiment of the present invention.
6 shows an example of a display provided to a user for video synthesis according to an exemplary embodiment of the present invention.
7 is a flowchart illustrating an example of a method for synthesizing a video according to an exemplary embodiment of the present disclosure.
8 shows an example of a display provided to a user for video synthesis according to an exemplary embodiment of the present invention.
9A and 9B are flowcharts illustrating examples of a method for synthesizing a moving picture according to exemplary embodiments of the present invention.
10 shows an example of a display provided to a user for video synthesis according to an exemplary embodiment of the present invention.
11 is a block diagram illustrating an example in which a video synthesis system according to an exemplary embodiment of the present disclosure is used.
Fig. 12 is a flowchart showing an example of the operation of the video synthesis system of Fig. 11 according to an exemplary embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The embodiments of the present invention are provided to more completely explain the present invention to those of ordinary skill in the art. Since the present invention can have various changes and can have various forms, specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the present invention to the specific disclosed form, it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention. In describing each figure, like reference numerals are used for like elements. In the accompanying drawings, the dimensions of the structures are enlarged or reduced than the actual size for clarity of the present invention.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It is to be understood that it does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings in the context of the related art, and unless explicitly defined in the present application, they are not interpreted in an ideal or excessively formal meaning. .

In the drawings and description below, a component indicated or described as one block may be a hardware block or a software block. For example, each of the components may be an independent hardware block that transmits and receives signals with each other, or may be a software block executed by at least one processor.

Fig. 1 is a block diagram showing a video synthesis system 10 according to an exemplary embodiment of the present invention. As shown in FIG. 1 , the video synthesis system 10 may communicate with a user 20 , and may connect to a network 32 or communicate with a data store 34 . In some embodiments, the video synthesis system 10 may be a stationary computing system, such as a server, desktop computer, or the like, or a mobile computing system, such as a laptop computer, mobile phone, or the like.

The video synthesis system 10 may synthesize and provide the video according to the input data IN provided by the user 20 . As will be described later with reference to the drawings, the user 20 may provide input data IN that can easily define a moving picture to the moving picture synthesizing system 10, and the moving picture synthesizing system 10 may use artificial intelligence ( Based on artificial intelligence (AI), video can be synthesized by processing input data (IN). Accordingly, the user 20 can intuitively input information required for video generation into the video synthesis system 10 , and the convenience of video generation can be improved. In addition, the user 20 can use not only the rule-based structured input data IN, but also the natural language input as the input data IN, so that anyone can easily produce a video without limitation of the group to which the user 20 belongs. can In addition, as will be described later with reference to FIGS. 11 and 12 , the video synthesis system 10 may enable video editing by multiple users, and collect videos by multiple users in various business models. It may enable creation to be utilized. As shown in FIG. 1 , the video synthesis system 10 includes a user interface 11 , an input processing unit 13 , an intermediate data generation unit 15 , a video generation unit 17 , and at least one machine learning model 19 . ) may be included.

The user interface 11 may provide communication with the user 20 . For example, as shown in FIG. 1 , the user interface 11 may receive input data IN from the user 20 , and provide the received input data IN to the input processing unit 13 . can In addition, the user interface 11 may provide the response RES provided from the input processing unit 13 and/or the video data VDO provided from the video generation unit 17 to the user 20 as output data OUT. there is. In some embodiments, the user 20 may provide input data IN to the video synthesis system 10 through an input device connected to the video synthesis system 10 , such as a keyboard, mouse, touch pad, etc. , the output data OUT may be received through an output device connected to the video synthesis system 10 , for example, a monitor, a printer, a speaker, or the like. Also, in some embodiments, the user 20 may access the video synthesis system 10 through his/her terminal, for example, a smart phone, a tablet PC, or the like, and may provide input data IN or output data through the terminal. (OUT) may be received.

The input processing unit 13 may receive the input data IN from the user interface 11 , and may provide a response RES to the user interface 11 by processing the input data IN. When the input data IN is received, the input processing unit 13 may generate a response RES to provide a display reflecting the input data IN or to request additional input data IN, and the user interface 11 . ) through the user 20 can be provided. The input data IN may refer to data that at least partially defines a video provided by the user 20 . For example, the input data IN may include information about content included as a part of a moving picture, for example, a moving picture, subtitles, photos, sound, etc. It may also include information about

The input processing unit 13 can extract, from the input data IN, the content data CNT about the content and the instruction DIR about the guide, and convert the content data CNT and the instruction DIR to the intermediate data generating unit (15) can be provided. Also, the input processing unit 13 may extract content data CNT corresponding to a plurality of contents and an instruction DIR corresponding to a series of instructions from the input data IN. In some embodiments, as shown by dotted lines in FIG. 1 , the input processing unit 13 is configured to perform content data CNT and instructions DIR from input data IN based on at least one machine learning model 19 . may be extracted, and the at least one machine learning model 19 may be in a state learned by a plurality of input data samples.

The intermediate data generating unit 15 may receive the content data CNT and the instruction DIR from the input processing unit 13 , and intermediate data INT from the content data CNT and the instruction DIR. can create The intermediate data INT is structured data, and may be generated by combining the content data CNT and the instruction DIR, and may have a structure recognizable by the video generating unit 17 . In some embodiments, as shown by a dotted line in FIG. 1 , the intermediate data generation unit 15 generates intermediate data ( INT), and the at least one machine learning model 19 may be in a state learned by a plurality of instruction samples.

The intermediate data INT may define a segment as at least a part of a moving picture. For example, the intermediate data INT may include a path to an image, sound, other moving picture, etc., to define a segment, and may include a pointer to data stored in a memory. In addition, the intermediate data INT may define a voice synthesized by text-to-speech (TTS), a voice of the user 20, and the like as sound. In some embodiments, the intermediate data INT may include not only the above-described structured data, but also unstructured data, such as embeddings and/or latent vectors. Embedding and latency vectors are a kind of machine learning model, and may refer to intermediate results generated by an artificial neural network processing an input. For example, the intermediate data generating unit 15 may generate an embedding and/or a latency vector from the content data (CNT) and the instruction (DIR), and the intermediate data (INT) including the embedding and/or the latency vector. ) can be created. The embedding and/or latency vector included in the intermediate data INT may be provided to the artificial neural network by the video generating unit 17 , and output data of the artificial neural network may be utilized when generating the video.

In some embodiments, the intermediate data generator 15 may generate new intermediate data INT based on previously generated intermediate data INT. That is, the intermediate data generating unit 15 may use many resources to generate the intermediate data INT from the content data CNT and the instruction DIR, and accordingly, the intermediate data generating unit 15 may use the instruction DIR. ), it is possible to determine whether to modify the already generated intermediate data INT or add data to the already generated intermediate data INT. For example, when a modified instruction DIR is received according to the input data IN, the intermediate data generating unit 15 needs to perform the voice synthesis again, combine other images, or create and attach a new segment. can decide whether to

The video generator 17 may receive the intermediate data INT from the intermediate data generator 15 , and may generate video data VDO by synthesizing the video based on the intermediate data INT. For example, the moving picture generator 17 may generate a segment by synthesizing an image or sound defined by the intermediate data INT, and may generate a moving picture by combining the segments. In some embodiments, the video generating unit 17 may detect an effect that needs to be reflected in a series of segments from the intermediate data INT, and may apply the detected effect to the series of segments in common. In some embodiments, as shown by a dotted line in FIG. 1 , the video generating unit 17 may generate video data VDO from the intermediate data INT based on at least one machine learning model 19 . In addition, at least one machine learning model 19 may be in a state learned by a plurality of intermediate data samples. Further, in some embodiments, the video generating unit 17 may connect to a network 32 such as, for example, the Internet, and search the network 32 by performing a search based on a guide included in the intermediate data INT. The data received through this may be used to generate moving picture data (VDO). In addition, the video generating unit 17 may access the data storage 34, obtain necessary data from the data storage 34 based on the guide included in the intermediate data INT, and store the obtained data. It can be used to create video data (VDO). In some embodiments, different from that shown in FIG. 1 , data store 34 may be included in video synthesis system 10 .

In some embodiments, the video generating unit 17 may provide a preview of the video to the user 20 as output data OUT through a user interface. For example, the input processing unit 13 receives the instruction DIR from the natural language instruction, "Subsequent scenes are in black and white like the video effect of the movie sin city, but use only red" included in the input data IN. When generating and the intermediate data generating unit 15 generates the intermediate data INT based on the instruction DIR, the moving picture generating unit 17 requires a computing resource with a high effect commonly applied to a series of segments. It may be determined that this is the case, and by showing some segments to which such an effect is applied to the user 20 in advance, a confirmation (eg, yes or no) of whether to apply may be received from the user 20 .

At least one machine learning model 19 may be in a trained state by a plurality of samples, and may be any artificial intelligence model. In addition, in some embodiments, at least one machine learning model 19 may be continuously learned while video synthesis is performed by the video synthesis system 10 according to the input data IN of the user 20 . . For example, the at least one machine learning model 19 may be an artificial neural network, a decision tree, a support vector machine, a regression analysis, a Bayesian network, or a genetic algorithm. It may be a model based on, etc. In this specification, the at least one machine learning model 19 will be mainly described with reference to an artificial neural network as shown in FIG. 1 , but it is noted that the exemplary embodiments of the present disclosure are not limited thereto. Artificial neural networks include, as non-limiting examples, Convolution Neural Network (CNN), Region with Convolution Neural Network (R-CNN), Region Proposal Network (RPN), Recurrent Neural Network (RNN), Stacking-based deep neural (S-DNN). Network), S-SDNN (State-Space Dynamic Neural Network), Deconvolution Network, DBN (Deep Belief Network), RBM (Restricted Boltzmann Machine), Fully Convolutional Network, LSTM (Long Short-Term Memory) Network, Classification Network, etc. may include

In some embodiments, at least one machine learning model 19 is a machine learning model used by the input processor 13 , a machine learning model used by the intermediate data generator 15 , and a video generator 17 . It may include at least one of the machine learning models used by The video synthesis system 10 may, in some embodiments, include dedicated hardware designed to implement the at least one machine learning model 19 , such as a neural processing unit (NPU), a graphic processing unit (GPU), or the like. and, in some embodiments, may include a software block that implements at least one machine learning model 19 by being executed on a general processor. Also, in some embodiments, different from that shown in FIG. 1 , at least one machine learning model 19 may be implemented outside of the video synthesis system 10 , and the video synthesis system 10 includes at least one It is also possible to access the machine learning model 19 of

Fig. 2 is a flowchart showing an example of a method for synthesizing a moving picture according to an exemplary embodiment of the present invention. As shown in FIG. 2 , the method for synthesizing a moving picture may include a plurality of steps S200 , S400 , S600 , and S800 . In some embodiments, the method of FIG. 2 may be performed by the video synthesis system 10 of FIG. 1 , which will be described below with reference to FIG. 1 .

Referring to FIG. 2 , an operation of acquiring input data IN may be performed in step S200 . For example, the user interface 11 may receive the input data IN from the user 20 and provide the input data IN to the input processing unit 13 . Also, the video synthesis system 10 may acquire the input data IN by accessing the storage in which the input data IN is stored. As described above with reference to FIG. 1 , the input data IN may refer to data that at least partially defines a moving picture.

In step S400, an operation of extracting the content data (CNT) and the instruction (DIR) may be performed. In some embodiments, the input processing unit 13 may extract the content data CNT and the instruction DIR from the input data IN based on predefined rules. For example, rules for identifying content data (CNT) and instruction (DIR) (eg, grammars defining instructions) may be predefined, and the user 20 may write according to the predefined rules. The input data IN including the guide may be provided, and accordingly, the input processing unit 13 may extract an instruction DIR corresponding to the guide based on predefined rules, and the input data IN Content data (CNT) can be extracted from the remaining part of In addition, in some embodiments, the input processing unit 13 may extract the content data (CNT) and the instruction (DIR) based on the at least one machine learning model (19). An example of step S400 will be described later with reference to FIGS. 3, 9A, 9B, and the like.

In operation S600, an operation of generating the intermediate data INT may be performed. For example, as described above with reference to FIG. 1 , the intermediate data generating unit 15 generates structured intermediate data INT by combining the content data CNT and the instruction DIR extracted in step S400 . can do. Also, in some embodiments, the intermediate data generator 15 may generate the intermediate data INT based on at least one machine learning model 19 learned by a plurality of instruction samples.

In step S800, an operation of generating a moving picture may be performed. For example, the moving picture generator 17 may generate the moving picture data VDO based on the intermediate data INT generated in step S600 . In some embodiments, the video generating unit 17 may be based on at least one machine learning model 19, may be connected to the network 32, and may be configured to generate video data (VDO), a data storage ( 34) can also be accessed.

3 is a flowchart illustrating an example of a method for synthesizing a moving picture according to an exemplary embodiment of the present invention. Specifically, the flowchart of FIG. 3 shows step S420 included in step S400 of FIG. 2 . As described above with reference to FIG. 2 , an operation of extracting the content data CNT and the instruction DIR may be performed in step S400 of FIG. 2 . In some embodiments, step S400 of FIG. 2 may include step S420 of FIG. 3 extracting a text-based instruction. As shown in FIG. 3 , step S420 may include a plurality of steps S422 , S424 , and S426 . In some embodiments, different from that shown in FIG. 3 , step S420 may include only some of the plurality of steps S422 , S424 , and S426 . In some embodiments, step S420 may be performed by the input processing unit 13 of FIG. 1 , and FIG. 3 will be described below with reference to FIGS. 1 and 2 .

Referring to FIG. 3 , an operation of extracting an instruction for inserting content may be performed in step S422. For example, the input processing unit 13 may extract an instruction corresponding to insertion of at least one of content, such as a moving picture, subtitle, image, and sound, and in this specification, the instruction to insert the content is referred to as a first instruction. can be As will be described later, the intermediate data generating unit 15 may generate the intermediate data INT according to the content inserted according to the first instruction.

In some embodiments, upon receiving the first instruction corresponding to the insertion of the caption, the intermediate data generating unit 15 may divide the caption into at least one part, and the intermediate data ( INT) can be created. For example, the intermediate data generating unit 15 may divide the subtitles based on the size of characters and/or the size of the screen, and by providing the subtitles to at least one machine learning model 19 , a section for human reading Subtitles may be divided based on The intermediate data generator 15 may generate intermediate data INT including information for distinguishing the divided parts of the subtitle, and accordingly, the subtitle is divided into the moving picture as will be described later with reference to FIG. 4B and the like. may be included.

In some embodiments, upon receiving the first instruction corresponding to the insertion of the image, the intermediate data generating unit 15 may obtain information about the position and/or size of the image, including the obtained information. Intermediate data (INT) can be generated. For example, the intermediate data generating unit 15 may obtain information on the location and/or size directly input by the user 20 with respect to the image, and may obtain information on the location and/or size from the image placed by the user 20 . It is also possible to obtain information about the size. The intermediate data generating unit 15 may generate intermediate data INT including information on the position and/or size of the image, and accordingly, as will be described later with reference to FIG. 4B and the like, the image is included in the moving picture. may be included.

In some embodiments, upon receiving the first instruction corresponding to the insertion of the sound, the intermediate data generating unit 15 may determine at least one of a voice, a background sound, and an effect sound, and include information about the determined sound. Intermediate data (INT) can be generated. A moving picture may include various sounds, and the sound may be divided into a voice, a background sound, and an effect sound. The intermediate data generator 15 may determine a sound according to an instruction written by the user 20 , or may determine a sound based on at least one machine learning model 19 . The intermediate data generating unit 15 may include, in the intermediate data INT, information (eg, information for reading characters included in the input data) for outputting the corresponding part as a sound when the sound is determined to be voice, and , information about the background sound obtained from the network 32 and/or the data storage 34 when the sound is determined as a background sound may be included in the intermediate data INT, and when the sound is determined as the background sound, the network 32 and/or information about sound effects obtained from the data storage 34 may be included in the intermediate data INT.

Referring back to FIG. 3 , in step S424, an operation of extracting an instruction for controlling a flow of a video may be performed. The flow of the video may refer to the timing of frames constituting the video. In this specification, an instruction for controlling the flow of a video may be referred to as a second instruction. In response to the second instruction, the intermediate data generator 15 may generate timing information of a series of frames and generate intermediate data INT including the timing information.

In step S426, an operation of extracting the natural language instruction may be performed. The user 20 may include instructions written in natural language as well as instructions defined by predefined rules in the input data IN, and the input processing unit 13 provides natural language instructions that are not defined by predefined rules. can be extracted. In this specification, an instruction written in a natural language may be referred to as a third instruction, and an example of an operation of processing the natural language instruction will be described later with reference to FIG. 7 and the like.

4A shows an example of a display provided to a user for video synthesis according to an exemplary embodiment of the present invention, and FIG. 4B shows an exemplary frame of a video synthesized according to an exemplary embodiment of the present invention. In some embodiments, the user interface 11 may provide the user 20 with an input interface as shown in FIG. 4A , and may receive input data IN through the input interface. That is, the step S200 of obtaining the input data of FIG. 2 may include providing an input interface to the user 20 and receiving the input data IN through the input interface. For example, the user interface 11 may display the display of FIG. 4A through a display device (eg, a monitor) connected to the video synthesis system 10 , and may display the display of FIG. 4A on the display device of the terminal of the user 20 . It is also possible to display the display of Also, in some embodiments, the display of FIG. 4A may be displayed by an application running on the terminal of the user 20 . Hereinafter, FIGS. 4A and 4B will be described with reference to FIG. 1 .

Referring to FIG. 4A , the display may include first to fourth regions R1 to R4 . The first area R1 may provide tools that the user 20 can use to generate the input data IN. For example, as shown in FIG. 4A , the first area R1 includes a storage tool C11 for storing input data IN created up to now, and a new file tool C12 for creating a new video. may include Further, the first region R1 includes a subtitle tool C21 for inserting subtitles and/or text-based instructions, an image tool C22 for inserting an image, and a sound tool C23 for inserting a sound. can do. In addition, the first region R1 may include a preview tool C30 for viewing a video generated by input data IN created up to now. Also, the first region R1 may include a speech balloon tool C41 and a writing tool C42 for adding a guide for an object of a moving picture. In some embodiments, different from that shown in FIG. 4A , the user interface 11 may provide the user 20 with only some of the tools shown in FIG. 4A , and may provide additional tools not shown in FIG. 4A . It may be further provided to the user 20 .

The second area R2 may display a series of scenes to the user, and may be referred to as a flow area in this specification. For example, as shown in FIG. 4A , the second region R2 may display the first to fifth scenes S01 to S05 to the user 20 . In the present specification, a scene may refer to a configuration corresponding to a portion of a playback time of a moving picture, and may include at least one frame. For example, as shown in FIG. 4A , the second region R2 has the first to fifth scenes S01 to S05 together with the start times of each of the first to fifth scenes S01 to S05 (eg, 25.50 of the second scene S02 may be displayed. In addition, the user 20 may input a guide for timing-dependent effects in the second region R2, and FIG. 6 is an example of a guide input by the user 20 in the second region R2. It will be described later with reference.

The third area R3 may display a scene selected from a series of scenes displayed in the second area R2 , and may be referred to as a scene area. The user 20 may input content and a guide for defining a corresponding scene in the third area R3. For example, as shown in FIG. 4A , the user 20 may insert an image in the fourth scene S04 and may input text under the image. In FIG. 4A, "#Government photo description" starting with "#" may indicate a comment for the user 20's own understanding, and may not be used for video synthesis. In FIG. 4A, "Qing Dynasty (omitted)" is a subtitle, and in some embodiments, the input processing unit 13 may extract text input as an instruction for inserting a subtitle, and extract the text into content data ( CNT) can be produced. Referring to FIG. 4B , in a frame corresponding to the fourth scene S04 of FIG. 4A , an image and a caption inserted in the third region R3 of FIG. 4A may be displayed. In some embodiments, as will be described later with reference to FIG. 10 , the user 20 may input not only subtitles but also lines of characters appearing in a video into the third area.

Referring back to FIG. 4A , the user 20 may input an additional instruction for content or guides corresponding to a scene displayed in the third area R3 into the fourth area R4. For example, the user 20, "img location = center, width = 70%", "img location = left 220, ratio = 30%", etc., such as an indication of the location and / or size of the image in the fourth area (R4) can be entered. In some embodiments, in the fourth area R4 , information and guides regarding a portion currently selected by the user 20 in the third area R3 , for example, images, subtitles, instructions, annotations, and the like may be displayed.

5A shows an example of a display provided to a user for video synthesis according to an exemplary embodiment of the present invention, and FIG. 5B shows exemplary frames of a video synthesized according to an exemplary embodiment of the present invention. In some embodiments, the video synthesis system 10 of FIG. 1 may generate a figure of a person reading the text input by the user 20 and include it in the video. In the description of FIG. 5A , content overlapping with the description of FIG. 4A will be omitted.

Referring to FIG. 5A , the display may include first to fourth regions R1 to R4. In the third region R3, a person for reading “Gukchae after the Qing Dynasty (omitted)” by “voc/cap:” may be included in the moving picture, and the same text may be displayed as subtitles. For example, as shown in the upper part of FIG. 5B, a person reading "The Qing Dynasty's economic collapse was (omitted) induced," may be included in the video. Also, the timing at which the image inserted into the third region R3 is included in the moving picture may be determined by “img: (omitted):img” in FIG. 5A . For example, as shown in the middle of FIG. 5B , an image inserted in the third area R3 may be displayed while "Xinhai Revolution has been (omitted)" is read and displayed as subtitles. Then, as shown in the lower part of Fig. 5B, "After that, the government bonds ..." can be read again by a person and displayed as subtitles.

6 shows an example of a display provided to a user for video synthesis according to an exemplary embodiment of the present invention. 6 , the display may include first to fourth regions R1 to R4. In some embodiments, the user 20 may input a guide for a timing-dependent effect in the second region R2 . In the description of FIG. 6 , content overlapping with the descriptions of FIGS. 4A and 5A will be omitted.

Referring to FIG. 6 , the second area R2 may display a series of scenes S01 to S05. Among the series of scenes S01 to S05, the second scene S02 may be composed of an image, and the third scene S03 may be composed of a moving picture. The user 20 may input a guide for timing-dependent effects in the second region R2 . That is, the step S400 of extracting the content data and the instruction in FIG. 2 includes the step of extracting an instruction (which may be referred to as a fourth instruction in this specification) indicating a timing-dependent effect through the second region R2. may include For example, the user 20 may input a guide for a start time of a scene, an order of scenes, an appearance effect of the scenes, and the like, by right-clicking the mouse in the second area R2 . Also, the user 20 may input a guide for a background sound, a duration, a scene change effect, etc. with respect to the second scene S02 composed of an image. In addition, the user 20 may input a guide for the replay speed, whether to mute, the scene change effect, etc. with respect to the third scene S03 composed of a moving picture.

7 is a flowchart illustrating an example of a method for synthesizing a video according to an exemplary embodiment of the present disclosure. Specifically, FIG. 7 shows an example of step S600 of FIG. 2 , and an operation of generating the intermediate data INT may be performed in step S600 ′ of FIG. 7 . As shown in FIG. 7 , step S600 ′ may include a plurality of steps S620 , S640 , S660 , and S680 . Hereinafter, FIG. 7 will be described with reference to FIGS. 1 and 2 .

Referring to FIG. 7 , an operation of determining whether to indicate a natural language may be performed in step S620 . For example, as described above with reference to FIG. 3 , the natural language instruction and the third instruction may be extracted in step S400 of FIG. 2 . For example, the user 20 may write and input the guides described above in natural language with reference to FIGS. 4A , 5A and 6 , and the natural language instruction may be extracted by the input processing unit 13 . As shown in FIG. 7 , when the non-natural language instruction is extracted, step S640 may be subsequently performed, and an operation of generating intermediate data INT according to the instruction may be performed in step S640 . On the other hand, when the natural language instruction is extracted, step S660 may be subsequently performed.

In step S660, natural language processing may be performed. Natural language processing extracts the meaning contained in natural language through morpheme analysis and lexical analysis, and is used in various fields such as artificial intelligence speakers and question and answer systems. The intermediate data generator 15 may process the natural language of the natural language instruction based on the at least one machine learning model 19 , and thus extract the intention of the user 20 included in the natural language instruction.

In step S680, an operation of generating a non-natural language instruction may be performed. For example, the intermediate data generator 15 may generate at least one non-natural language instruction according to the intention of the natural language instruction extracted in step S660. An example in which at least one non-natural language indication is generated will be described later with reference to FIG. 8 . Following step S680, step S640 may be performed, and thus intermediate data INT may be generated according to the non-natural language instruction generated in step S680.

8 shows an example of a display provided to a user for video synthesis according to an exemplary embodiment of the present invention. As illustrated in FIG. 8 , the display may include first to fourth regions R1 to R4 . As described above with reference to FIG. 7 , the user 20 of FIG. 1 may input a natural language instruction, and the input processing unit 13 and the intermediate data generating unit 15 may process the natural language instruction. Hereinafter, FIG. 8 will be described with reference to FIG. 1 .

Referring to FIG. 8 , in the third region R3 , the user 20 may input a natural language instruction “// Read with emphasis on large and large amounts”. The input processing unit 13 may extract "// read with emphasis on a large amount" from the input data IN as a natural language instruction, and may provide it to the intermediate data generation unit 15 . The intermediate data generating unit 15 can extract the intention of the natural language instruction by natural language processing "// read with emphasis on large and large amounts", and based on the extracted intention, for example, "Qing Dynasty (omitted) accent: large" It is possible to generate at least one non-natural language indication "accent", such as :accent (omitted).

Also, referring to FIG. 8 , in the third region R3 , the user 20 may input a natural language instruction “// disappear while reducing the image”. The input processing unit 13 may extract “// disappear while reducing the image” as a natural language instruction from the input data IN, and may provide it to the intermediate data generating unit 15 . The intermediate data generating unit 15 may extract the intention of the natural language instruction by natural language processing of "// disappear while reducing the image", and based on the extracted intention, for example, "img: shrink & disappear :img" At least one non-natural language directive "img:", "shrink", "disappear", ":img" may be generated.

9A and 9B are flowcharts illustrating examples of a method for synthesizing a video according to exemplary embodiments of the present invention, and FIG. 10 is a display provided to a user for video synthesis according to an exemplary embodiment of the present invention. shows an example of Specifically, the flowcharts of FIGS. 9A and 9B show steps S440a and S440b included in step S400 of FIG. 2 , respectively. As described above with reference to FIG. 2 , an operation of extracting the content data CNT and the instruction DIR may be performed in step S400 of FIG. 2 . In some embodiments, step S400 of FIG. 2 may include steps S440a and/or step S440b of FIGS. 9A and 9B of extracting an action-based instruction. In some embodiments, steps S440a and S440b may be performed by the input processing unit 13 of FIG. 1 , and FIGS. 9A and 9B will be described below with reference to FIGS. 1 and 2 , FIGS. 9A and 9B . Contents overlapping each other in the description of FIG. 9B will be omitted.

Referring to FIG. 9A , step S440a may include a plurality of steps S442a, S444a, and S446a. In step S442a, an operation of extracting a shape included in the action layer may be performed. The action layer may refer to a layer including an action-based instruction in the third region R3, and in some embodiments, instructions input through the speech balloon tool C41 or the writing tool C42 may be included in the action layer. there is. For example, as shown in FIG. 10 , the user 20 may input “male in his twenties” and “male in his 50s” on the image of the third region R3 through the speech balloon tool C41. . Also, the user 20 may input “A” and “B” on the image through the writing tool C42 . The input processing unit 13 may extract shapes from the action layer including the action-based instructions input by the speech balloon tool C41 and the writing tool C42, and accordingly, two speech balloons and two handwritten characters are extracted. can be

In step S444a, an operation of recognizing a symbol and/or a language from the shape may be performed. For example, the input processing unit 13 may extract “male in 20s” and “male in 50s” from the speech bubbles extracted in step S442a, and extract “A” and “B” written on the people. can In addition, although not shown in FIG. 10 , the user 20 may input a symbol such as an arrow or a pig's tail in the action layer, and the input processing unit 13 may recognize a symbol such as an arrow and a pig's tail.

In step S446a, an operation of generating a text-based indication may be performed. For example, the input processing unit 13 may generate a text-based instruction from the symbol and/or language recognized in step S444a. For example, the input processing unit 13 may recognize objects included in the image, ie, people, based on the at least one machine learning model 19 , and “A” and “B” written on the recognized people From ", two people can be identified, respectively. Accordingly, the input processing unit 13 may generate text-based instructions specifying the identifier of the left person as “A” and the identifier of the right person as “B”, and to be transmitted to the intermediate data generating unit 15 . can Further, the input processing unit 13 may generate text-based instructions specifying attributes of two persons from “a male in his twenties” and “a man in his 50s” written in speech bubbles indicating the recognized persons, and the intermediate data generating unit (15) can be forwarded. Accordingly, the intermediate data generating unit 15 may generate the intermediate data INT for reproducing the lines under the image with the voices of a “male in his 50s” and a “male in his twenties”, respectively.

The action-based instruction is not limited to the above-described examples. For example, the action-based instruction may include instructions for camerawork, instructions for actions of characters appearing in the video, such as shaking the head or closing eyes, and the like. Text-based instructions can be generated using a library for dimensional modeling. Accordingly, 3D data for rendering may be generated as intermediate data, and a moving picture may be synthesized.

Referring to FIG. 9B , step S440b may include a plurality of steps S442b, S444b, and S446b. In step S442b, an operation of identifying an action-based indication and extracting a shape may be performed. For example, the input processing unit 13 may identify the action-based instruction included in the input data IN based on the at least one machine learning model 19 . In general, at least one machine learning model 19 may also be in a state of being trained by a plurality of sample continuity samples, just as a human can distinguish content and continuity described with the content, such as symbols, text, etc. can Accordingly, the input processing unit 13 includes the action-based instruction included in the input data IN without an indicator (eg, an action layer) indicating that the user 20 is an action-based instruction in at least one machine learning model 19 . can be identified based on The input processing unit 13 may extract a shape from the identified action-based instruction as described above with reference to FIG. 9A . Then, an operation of recognizing a symbol and/or a language from the shape may be performed in step S444b, and an operation of generating a text-based indication may be performed in step S446b.

11 is a block diagram illustrating an example in which a video synthesis system according to an exemplary embodiment of the present disclosure is used, and FIG. 12 is an example of the operation of the video synthesis system 40 of FIG. 11 according to an exemplary embodiment of the present disclosure. is a flowchart showing In some embodiments, as shown in FIG. 11 , video synthesis system 40 may be used for multiple users 70 via network 60 .

Referring to FIG. 11 , the video synthesis system 40 may communicate with the video storage server 50 . For example, the video synthesis system 40 may store the generated video in the video storage server 50 and read the video stored in the video storage server 50 . In addition, the video synthesis system 40 may provide an input interface for video synthesis to multiple users 70 through a network 60 such as the Internet, and may receive input data from multiple users 70 . there is. In addition, the video synthesis system 40 may provide a video synthesized through the network 60 to the multiple users 70 , and may receive feedback on the provided video from the multiple users 70 . As described above with reference to FIG. 1 , each of the multiple users 70 may access the video synthesis system 40 through the network 60 using their terminal, for example, a personal computer, a mobile phone, or the like.

Referring to FIG. 12 , the method of operating the video synthesis system 40 of FIG. 11 may include a plurality of steps S20 , S40 , S60 , and S80 . First, in step S20 , an operation of providing a video to multiple users 70 may be performed. For example, the video synthesis system 40 reads the video stored in the video storage server 50 in response to the request of the multiple users 70 and provides it to the multiple users 70 through the network 60 . can

In step S40, an operation of receiving and providing feedback may be performed. For example, the video synthesis system 40 may receive feedback on the video provided in step S20 from the multiple users 70, and may provide feedback to the multiple users 70 by posting the received feedback. there is. Accordingly, the multi-users 70 may check the feedback provided by the user as well as the feedback provided by other users for the video. In some embodiments, the feedback may include comments from multiple users 70 , and may include ratings (or rewards) from multiple users 70 on the video provided in step S20 .

In operation S60, an operation of receiving input data may be performed, and in operation S80, an operation of synthesizing a moving picture may be performed. Step S20 may be performed again following step S80, and accordingly, the plurality of steps S20, S40, S60, and S80 may be repeated.

For example, the video synthesis system 40 may provide an input interface to multiple users 70 through the network 60 as described above with reference to the drawings, and the multiple users 70 may use the input interface It is possible to receive input data input through the network 60 through the network 60 . As described above with reference to FIG. 1 and the like, the input data is data defining at least a part of a moving picture, and the input data received in step S60 includes the content of modifying the moving picture provided to the multiple users 70 in step S20. can do. Multiple users 70 may feel an interest or a sense of achievement by looking at the feedback provided by themselves in step S40 as well as feedback of other users, and in step S20 for greater pleasure or satisfaction or to obtain a response from other users. Input data for correcting the provided video may be provided to the video synthesis system 40 through the network 60 .

As described above with reference to the drawings, the video synthesis system 40 allows multiple users 70 to intuitively input information required for video generation, thereby improving the convenience of video creation and natural language input. By supporting it, anyone can easily create a video. Accordingly, multiple users 70 can easily participate in video synthesis, and as the video is gradually processed by the multiple users 70, feedback on the video of the processing process as well as users involved in the editing of the video is provided. Many users may access the video synthesizing system 40 , such as users who provide, and users who are interested by checking feedback. Through this, the video synthesis system 40 may be utilized in various business models, such as advertisements.

Exemplary embodiments have been disclosed in the drawings and specification as described above. Although the embodiments have been described using specific terms in the present specification, these are only used for the purpose of explaining the technical idea of the present invention and not used to limit the meaning or the scope of the present invention described in the claims . Therefore, it will be understood by those skilled in the art that various modifications and equivalent other embodiments are possible therefrom. Accordingly, the true technical protection scope of the present invention should be defined by the technical spirit of the appended claims.

Claims (11)

obtaining input data that at least partially defines a moving picture;
extracting, from the input data, content data about content to be included as part of a video and a series of instructions about a guide used for video synthesis, based on predefined rules;
generating structured intermediate data from the content data and a series of instructions based on at least one machine learning model learned by a plurality of instruction samples; and
Comprising the step of generating a video based on the intermediate data,
the rules include rules for identifying the content data and the set of instructions;
wherein extracting the series of instructions and content data includes extracting text-based instructions based on the rules;
Extracting the text-based instruction comprises:
extracting a first instruction indicating insertion of content;
extracting a second instruction for controlling the flow of the video; and
extracting a third instruction written in natural language; Method for synthesizing a video, characterized in that it comprises at least one of. delete The method according to claim 1,
The step of generating the intermediate data includes:
dividing the subtitle into at least one part and generating the intermediate data including the at least one part when the first instruction corresponds to the insertion of the subtitle;
when the first instruction corresponds to the insertion of an image, obtaining information on the position and/or size of the image, and generating the intermediate data including the obtained information; and
determining at least one of a voice, a background sound, and an effect sound when the first instruction corresponds to insertion of a sound, and generating the intermediate data including information on the determined sound; A method for synthesizing a video, comprising at least one of. The method according to claim 1,
The step of generating the intermediate data includes:
and generating timing information of a series of frames in response to the second instruction, and generating intermediate data including the timing information. The method according to claim 1,
The generating of the intermediate data comprises generating at least one of a first instruction and a second instruction by processing the natural language in response to the third instruction. The method according to claim 1,
extracting the series of instructions and content data includes extracting an action-based instruction as at least one of the series of instructions based on the rules;
The step of extracting the action-based instruction is,
extracting a shape included in the action layer;
recognizing a sign and/or language from the shape based on the at least one machine learning model; and
based on the symbol and/or the language, generating at least one text-based indication. The method according to claim 1,
extracting the series of instructions and content data includes extracting an action-based instruction as at least one of the series of instructions based on the rules;
The step of extracting the action-based instruction is,
identifying an action-based instruction based on the at least one machine learning model, and extracting a shape from the identified action-based instruction;
recognizing a sign and/or language from the shape based on the at least one machine learning model; and
based on the symbol and/or the language, generating at least one text-based indication. The method according to claim 1,
The step of obtaining the input data includes:
providing an input interface to a user; and
receiving the input data via the input interface;
Providing the input interface comprises:
displaying a scene area corresponding to a scene that is part of the reproduction time of the moving picture, and in which the content data and a series of instructions corresponding to the scene are displayed; and
A method for synthesizing a moving picture comprising the step of displaying a flow area comprising at least a portion of a series of scenes. 9. The method of claim 8,
and extracting the series of instructions and content data includes extracting a fourth instruction indicating a timing-dependent effect through the flow area. A method of synthesizing a video performed by a video synthesis system accessed by multiple users, the method comprising:
providing a first video to the multiple users;
receiving a first feedback on the first video from the multiple users;
receiving first input data for at least partially modifying the first video from at least some of the multiple users;
synthesizing a second video by modifying the first video based on the first input data;
providing the second video to the multiple users;
receiving a second feedback on the second video from the multiple users;
receiving second input data for at least partially modifying the second video from at least some of the multiple users;
synthesizing a third video by modifying the second video based on the second input data; and
and providing the third video to the multiple users. delete

Images