KR102661488B1 - Servers, systems, methods and programs that provide special effect creation and 3d model creation services using generative ai models - Google Patents

Abstract

According to an embodiment of the present invention, a server providing special effect generation and 3D model generation services using a generative AI model is disclosed. The server includes an object segmentation unit that determines objects included in an input image using an image segmentation model; A video generator that generates a special effects video using a video generation model; and a synthesis unit that designates a special effect input area in the original video using a user-specified area received from the user terminal and synthesizes the special effect video in the special effect input area of the original video.

Description

Servers, systems, methods and programs that provide special effect synthesis and 3D model creation services using generative AI models {SERVERS, SYSTEMS, METHODS AND PROGRAMS THAT PROVIDE SPECIAL EFFECT CREATION AND 3D MODEL CREATION SERVICES USING GENERATIVE AI MODELS}

The present invention relates to a server, system, method, and program that provides special effect generation and 3D model generation services using generative AI models.

Unless otherwise indicated herein, the material described in this section is not prior art to the claims of this application, and is not admitted to be prior art by inclusion in this section.

Removing and replacing objects in the film industry is one of the most time-consuming and expensive tasks. Because objects were removed and replaced manually for each countless frame, a lot of time and manpower were consumed, and the quality of removal and replacement varied depending on the skill of the worker.

When removing and replacing objects using inpainting models and outpainting models using deep learning, work speed is dramatically improved through automation and quality consistency is maintained. In particular, the introduction of inpainting models and outpainting models into the film industry is accelerating because costs are greatly reduced compared to existing methods.

Republic of Korea Patent Publication No. 10-2486300 (2023.01.04) Republic of Korea Patent Publication No. 10-2023-0133059 (2023.09.19) Republic of Korea Patent Publication No. 10-2021-0144294 (2021.11.30) Republic of Korea Patent Publication No. 10-2338913 (2021.12.08)

The present invention provides a server, system, method, and program that provides a service for generating images or videos with special effects applied using generative AI, and a service for creating special effects and 3D models using a generative AI model. It is for work purposes.

The purpose of the present invention is to provide a server, system, method, and program that provides a 3D modeling service using generative AI, a special effect generation using a generative AI model, and a 3D model creation service.

One aspect of the present invention to achieve the above object is to provide a server that provides special effect generation and 3D model generation services using a generative AI model.

The server includes an object segmentation unit that determines objects included in an input image using an image segmentation model; A video generator that generates a special effects video using a video generation model; and a synthesis unit that designates a special effect input area in the original video using a user-specified area received from the user terminal and synthesizes the special effect video in the special effect input area of the original video.

In addition, the object division unit divides each of the plurality of frames included in the special effect video into a special effect object area and a background area using an image segmentation model and a video object segmentation model, and divides each of the plurality of frames included in the special effect video into a special effect object area and a background area. Remove the background area.

In addition, the synthesis unit designates a user-specified area as a special effect input area of the first frame of the original video, and the object division unit uses an image segmentation model to create at least one special effect input area included in the first frame of the original video. Identify the object, determine one of at least one object as the target object, create a mask matching the target object, and target each remaining frame of the original video using the mask matching the target object and the video object segmentation model. Separated into an object area and a background area, the synthesis unit creates a special effect input area for each remaining frame of the original video using the target object area of each remaining frame of the original video.

In addition, the synthesis unit performs 1:1 matching between each of the plurality of frames of the original video and each of the plurality of frames of the special effects video, and each of the plurality of frames of the special effects video is placed in the special effect input area of each of the plurality of frames of the original video. synthesize.

In addition, the server uses each of the plurality of frames included in the special effects video and an image generation model to match each of the plurality of frames included in the special effects video and generates special effect objects included in each of the plurality of frames in various directions. an image generator that generates a plurality of modeling images shown from angles; and a 3D generation unit that generates a 3D model matching a plurality of modeling images using the 3D generation model.

In addition, the video generator specifies a marker creation area on one of the plurality of modeling images, and the object segmentation unit marks the remaining modeling images among the plurality of modeling images using the marker creation area and the video object segmentation model. It is divided into a creation area and a background area, and the server includes an inpainting unit that creates a marker in the marker creation area of each of the plurality of modeling images using an inpainting model.

According to an embodiment of the present invention, a composite video can be generated by combining a continuous original video sequence with a special effects video sequence created using generative AI.

According to an embodiment of the present invention, a plurality of modeling images can be generated using generative AI, and a 3D model can be created using the generated modeling images.

Figure 1 is an outline diagram of a system for providing special effect synthesis and 3D model creation services using a generative AI model according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating functional modules of the service providing server according to FIG. 1.
FIG. 3 is a flowchart illustrating a process in which the service providing server according to FIG. 1 detects a dynamic object in a video frame.
Figure 4 is a diagram illustrating a process in which an image segmentation model creates a mask in an area that matches a target object in a frame.
Figure 5 is a flowchart showing a process in which a video object segmentation model receives the mask of the first frame of the video and the remaining frames of the video and detects dynamic objects included in the remaining frames.
Figure 6 is a flowchart showing an example of a specific process in step S200 of Figure 3.
Figure 7 is a diagram showing the process in which the image segmentation model is involved during the operation of the video object segmentation model.
Figure 8 is a diagram illustrating a process in which an inpainting model removes a dynamic object detected in a video and restores the removed part.
Figure 9 is a diagram illustrating a process in which an inpainting model removes a dynamic object detected in a video and restores the removed part.
FIG. 10 is a flowchart illustrating a process in which the service providing server according to FIG. 1 synthesizes a special effects video with the original video.
FIG. 11 is a flowchart showing the specific process of step S310 of FIG. 10.
FIG. 12 is a flowchart showing the specific process of step S320 of FIG. 10.
Figure 13 is a diagram conceptually showing the process of creating a special effects video.
FIG. 14 is a flowchart showing a process in which the service providing server according to FIG. 1 creates a 3D model.
FIG. 15 is a flowchart illustrating an example of step S410 of FIG. 14.
FIG. 16 is a flowchart illustrating an example of step S410 of FIG. 14.
FIG. 17 is a flowchart showing the specific process of step S420 of FIG. 14.
FIG. 18 is a diagram illustrating a plurality of modeling images in which markers are created in a marker creation area using an inpainting model.
Figure 19 is a diagram illustrating a UV map as an example.
FIG. 20 is a diagram illustrating the hardware configuration of the service providing server according to FIG. 1.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all changes, equivalents, and substitutes included in the spirit and technical scope of the present invention. While describing each drawing, similar reference numerals are used for similar components.

Terms such as first, second, A, and B may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, a first component may be named a second component without departing from the scope of the present invention, and similarly, the second component may also be named a first component. The term and/or includes any of a plurality of related stated items or a combination of a plurality of related stated items.

When a component is said to be "connected" or "connected" to another component, it is understood that it may be directly connected to or connected to the other component, but that other components may exist in between. It should be. On the other hand, when it is mentioned that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between.

The terms used in this application are only used to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by a person of ordinary skill in the technical field to which the present invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and unless explicitly defined in the present application, should not be interpreted in an ideal or excessively formal sense. No.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the attached drawings.

Figure 1 is an outline diagram of a system for providing special effect synthesis and 3D model creation services using a generative AI model according to an embodiment of the present invention.

Referring to FIG. 1, a system for providing dynamic object tracking, inpainting, and outpainting services includes a service providing server 100, a user terminal 200, and an artificial neural network model server 300.

The service providing server 100, the user terminal 200, and the artificial neural network model server 300 are connected to each other through a network.

In addition, this network includes, for example, a plurality of access networks (not shown) and a core network (not shown), and may be configured to include an external network, such as an Internet network (not shown). Here, the access network (not shown) is an access network that performs wired and wireless communication with the service providing server 100, the user terminal 200, and the artificial neural network model server 300, for example, BS (Base Station), BTS (Base Station) It can be implemented with multiple base stations such as Transceiver Station, NodeB, eNodeB, etc., and base station controllers such as BSC (Base Station Controller) and RNC (Radio Network Controller). In addition, as described above, the digital signal processing unit and the wireless signal processing unit integrated in the base station are divided into a digital unit (hereinafter referred to as DU) and a radio unit (hereinafter referred to as RU), respectively. , it can be configured by installing multiple RUs (not shown) in multiple areas and connecting multiple RUs (not shown) with a centralized DU (not shown).

In addition, the core network (not shown) that constitutes the mobile network together with the access network (not shown) serves to connect the access network (not shown) with an external network, such as an Internet network (not shown).

As described above, this core network (not shown) is a network system that performs major functions for mobile communication services such as mobility control and switching between access networks (not shown), and is a network system that performs circuit switching or packet switching. switching) and manages and controls packet flow within the mobile network. In addition, the core network (not shown) manages mobility between frequencies and plays a role in linking traffic within the access network (not shown) and core network (not shown) with other networks, such as the Internet network (not shown). It may be possible. This core network (not shown) further includes Serving GateWay (SGW), PDN GateWay (PGW), Mobile Switching Center (MSC), Home Location Register (HLR), Mobile Mobility Entity (MME), and Home Subscriber Server (HSS). It may be configured to include.

In addition, the Internet network (not shown) refers to a general public communication network in which information is exchanged according to the TCP/IP protocol, that is, a public network, and includes a service providing server 100, a user terminal 200, and an artificial neural network model server. The information provided from 300 can be provided to the network through the core network (not shown) and the access network (not shown), and conversely, the information provided through the network can be provided through the core network (not shown) and the access network (not shown). It may also be provided through the service provision server 100, user terminal 200, and artificial neural network model server 300. However, it is not limited to this, and the service providing server 100 may be implemented integrally with the core network (not shown).

In addition, in addition to the above-mentioned communication methods, it may include all other types of communication methods that are widely known or will be developed in the future.

The service providing server 100 may be a server that provides a web page, app page, program, or application that provides dynamic object tracking, inpainting, and outpainting services.

The user terminal 200 refers to a user terminal that wants to use dynamic object tracking, inpainting, and outpainting services.

The user terminal 200 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, a laptop equipped with a navigation system and a web browser, a desktop, a laptop, etc. At this time, the user terminal 200 may be implemented as a terminal that can connect to a remote server or terminal through a network. The user terminal 200 is, for example, a wireless communication device that guarantees portability and mobility, and includes navigation, personal communication system (PCS), global system for mobile communications (GSM), personal digital cellular (PDC), and personal communication system (PHS). Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) terminal, It may include all types of handheld wireless communication devices such as smartphones, smartpads, tablet PCs, etc.

The artificial neural network model server 300 may include an image segmentation model, a video object segmentation model, a resolution conversion model, an inpainting model, and an outpainting model.

Image segmentation models are learned to classify each pixel of an image into a specific semantic category to segment various objects and areas. An image segmentation model can be created through learning using a training dataset in which a segmentation mask is created by labeling the correct answer for each pixel of the image. A variety of known algorithms such as U-Net, FCN (Fully Convolutional Network), SegNet, DeepLab, and Mask R-CNN can be used to learn the image segmentation model, and the model is adjusted by adjusting the weights through gradient descent using a loss function. It can be learned. In one embodiment, the Segment Anything (Kirillov, Alexander, et al. arXiv preprint arXiv:2304.02643, 2023) method may be used to create an image segmentation model.

A video object segmentation model can be created through learning using a training dataset in which a mask, which is the image of the target object, is labeled in each consecutive frame. To learn a video object segmentation model, various known algorithms can be used, such as 3D Convolutional Neural Networks (3D CNNs), 2D CNNs with temporal information modeling, or a model combining RNN and LSTM. For example, the Mask R-CNN for Video method and the You can. A video object segmentation model can be learned by adjusting the weights using gradient descent using a set loss function.

The inpainting model is trained to restore missing or damaged parts. The inpainting model is created through learning using training data that labels missing or damaged images and normal correct images. To learn the inpainting model, an autoencoder, a convolutional neural network (CNN)-based model, or a GAN (Generative Adversarial Network)-based algorithm can be used. For example, the Stable Diffusion XL (Stability AI, GitHub Repository: Stable Diffusion XL) method can be used to create an inpainting model. The inpainting model can be learned by adjusting the weights using gradient descent using a set loss function.

The outpainting model is learned to extend the image beyond the boundaries of the existing image. Using the outpainting model, the image can be expanded to a wider field of view or environment. The outpainting model can be learned using the original image, expanded image, reduced image, and original image as training data. In one embodiment, algorithms such as Convolutional Neural Networks (CNNs), Generative Adversarial Networks (GANs), Attention Mechanisms and Transformers, and Recurrent Neural Networks (RNNs) may be used to learn the inpainting model. The inpainting model can be learned by adjusting the weights to minimize the set loss function.

The resolution conversion model is learned to convert low-resolution images into high-resolution images. The resolution conversion model can be created by generating training data by labeling a high-resolution image and a low-resolution image obtained by downsampling the high-resolution image, and learning using the generated training data. In one embodiment, algorithms such as Convolutional Neural Networks (CNNs), Generative Adversarial Networks (GANs), Deep Residual Networks (ResNet), Attention Mechanisms, Transformer Models, etc. may be used to learn the resolution transformation model. The resolution conversion model can be learned by adjusting the weights to minimize the set loss function.

The image generation model is created by preprocessing the collected training image data and training using the preprocessed image data. In one embodiment, the image generation model may be created using algorithms such as Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), Stable diffusion, Auto-Regressive Models (e.g., PixelRNN, PixelCNN). However, it is not limited to this, and various known algorithms can be used. During the training process, parameters are set to minimize the loss function defined for the set algorithm, and this can improve the performance of the image generation model.

A video generation model can be created by preparing images and video sequences related to the images as training data and training using the training data. In one embodiment, a video generation model may be created by preparing text and video sequences related to the text as training data and training using the training data. In one embodiment, a video generation model may be created by preparing text and images and video sequences related to the text and images as training data, and training using the training data. In one embodiment, the image generation model may be created using algorithms such as Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), Recurrent Neural Networks, 3D Convolutional Neural Networks, Recurrent Neural Networks, and 3D Convolutional Neural Networks. However, it is not limited to this, and various known algorithms can be used. During the training process, parameters are set to minimize the loss function defined for the set algorithm, which can improve the performance of the video generation model.

The 3D generation model is a model that performs feature point extraction and matching, 3D reconstruction, texture mapping, refining and optimization. It refers to a model that creates a 3D model using multi-angle modeling images, Structure from Motion (SfM), Multi-view Algorithms such as Stereo (MVS) and Photogrammetry can be used to build 3D generation models. However, it is not limited to this and various known algorithms can be used. For example, to create 3D models, software such as Agisoft Metashape and RealityCapture can be used.

In the illustrated embodiment, the artificial neural network model server 300 is shown as a separate server from the service providing server 100, but the service providing server 100 and the artificial neural network model server 300 are not limited thereto. can be formed.

FIG. 2 is a block diagram illustrating functional modules of the service providing server 100 according to FIG. 1.

The service providing server 100 includes an object division unit 101, an inpainting unit 102, an image creation unit 103, a video creation unit 104, a synthesis unit 105, and a 3D creation unit 106. do.

When the service providing server 100 transmits an application, program, app page, web page, etc. that provides dynamic object tracking and inpainting services to the user terminal 200, the user terminal 200 performs dynamic object tracking and inpainting. You can install or open applications, programs, app pages, web pages, etc. that provide painting services. Additionally, a service program may be run on the user terminal 200 using a script executed in a web browser. Here, a web browser is a program that allows the use of web (WWW: World Wide Web) services and refers to a program that receives and displays hypertext written in HTML (Hyper Text Mark-up Language), for example, Netscape. , Explorer, Chrome, etc. Additionally, an application refers to an application on a terminal and includes, for example, an app running on a mobile terminal (smartphone).

FIG. 3 is a flowchart illustrating a process in which the service providing server according to FIG. 1 detects a dynamic object in a video frame.

Referring to FIG. 3, the object segmentation unit 101 generates a mask representing the area of the target object included in the first frame of the video using an image segmentation model (S100).

Figure 4 is a diagram illustrating a process in which an image segmentation model creates a mask in an area that matches a target object in a frame.

The object division unit 101 may provide the user terminal 200 with a user interface that can designate a region of interest (ROI) within a frame.

When a region of interest is input through the user interface, the object segmentation unit 101 inputs the portion of the frame that matches the region of interest into the image segmentation model, and the image segmentation model identifies the target object included in the region of interest and determines the target object. Separate the area from the rest of the background area.

The object segmentation unit 101 provides the first frame of the video to the user terminal 200, receives the region of interest of the first frame from the user terminal 200, and generates a portion matching the region of interest of the first frame into an image segmentation model. and obtain a mask that matches the target object included in the first frame generated by the image segmentation model.

Referring again to FIG. 3, the object segmentation unit 101 detects target objects included in the remaining frames of the video using the mask generated in the first frame and the video object segmentation model (S200).

Figure 5 is a flowchart showing a process in which a video object segmentation model receives the mask of the first frame of the video and the remaining frames of the video and detects dynamic objects included in the remaining frames.

The object segmentation unit 101 inputs the mask generated from the first frame and the remaining frames of the video into the video object segmentation model, and the video object segmentation model divides the remaining frames into the target object area and the remaining frames using the mask generated from the first frame. Divide into background areas. The object division unit 101 obtains a mask, which is the area of the target object, for each remaining frame.

Figure 6 is a flowchart showing an example of a specific process in step S200 of Figure 3.

When all remaining frames are separated into the target object area and the remaining background area using the video object segmentation model, the accuracy of detecting the target object area decreases as the video playback time becomes longer and the number of frames becomes relatively larger. It can be.

If the image segmentation model intervenes in the process of the video object segmentation model detecting the target object in the remaining frames, it is possible to prevent a decrease in the accuracy of detecting the area of the target object even when the number of frames increases.

Referring to FIG. 6, the object segmentation unit 101 detects target objects included in a preset reference number of frames using a mask and a video object segmentation model generated in the first frame (S210).

The object segmentation unit 101 provides the mask generated from the first frame to the video object segmentation model, and uses the video object segmentation model to separate a preset standard number of frames after the first frame into the target object area and the remaining background area. do. The object division unit 101 obtains a mask, which is the area of the target object, for each remaining frame.

When mask generation for a preset reference number of frames is completed, the object division unit 101 generates a mask matching the target object included in the next frame (S220).

In one embodiment, the object segmentation unit 101 provides the user terminal 200 with a user interface for specifying a region of interest (ROI) within a frame, and divides the information received from the user terminal 200 using an image segmentation model. A mask that matches the target object included in the region of interest can be created.

In one embodiment, the object segmentation unit 101 creates a region of interest of a preset size including the target object of the frame last processed by the video object segmentation model, and images the portion matching the region of interest in the next frame. As input to the segmentation model, the image segmentation model identifies the target object included in the region of interest and separates the area of the target object from the remaining background area.

The object segmentation unit 101 uses an image segmentation model to generate a mask that is an area of the target object included in the next frame of the frame last processed by the video object segmentation model.

When mask generation using the image segmentation model is completed, the object segmentation unit 101 detects target objects included in a preset reference number of frames among the remaining frames (S230). For target object detection, a mask and a video object segmentation model generated from the frame following the last frame processed by video object segmentation are used.

When detection of the target object included in the preset reference number of frames is completed, the object division unit 101 determines whether there are unprocessed frames (S240).

If there are unprocessed frames, the object division unit 101 performs steps S220 and S230 again.

If there are no unprocessed frames, the object segmentation unit 101 ends target object detection.

Figure 7 is a diagram showing the process in which the image segmentation model is involved during the operation of the video object segmentation model.

In the illustrated embodiment, the image segmentation model intervenes in the middle of the process of the video object segmentation model detecting the target object in the remaining frames, generates a mask of the target object, and provides the generated mask to the video object segmentation model. In other words, since the frames are divided and processed, a decrease in the accuracy of target object detection can be prevented even when the number of frames increases.

However, when processing frames by dividing them, the time required for calculation and resource consumption may increase compared to continuously processing all frames except the first frame using the video object segmentation model.

Therefore, it is important to appropriately set the maximum number of frames that the video object segmentation model can process while maintaining target object detection accuracy. If you set the optimal number of frames that can be processed continuously while maintaining detection accuracy, you can minimize the increase in computation time and resource consumption and improve the detection accuracy of videos with long playback times.

In one embodiment, the object segmentation unit 101 may determine the number of frames that the video object segmentation model sequentially processes at a time using a frame number determination model. The frame number determination model can be created through learning using training data generated by labeling the number of frames with the tracked object, the number of video frames processed, and the detection accuracy. In one embodiment, accuracy may include Intersection over Union (IoU), Pixel Accuracy, Boundary F1 Score, Temporal Consistency, etc. In one embodiment, various known algorithms such as Artificial Neural Networks (ANN), Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM), Random Forest, and Xgboost may be used for learning. In one embodiment, the detection object, number of frames, and accuracy may be matched and stored in the database of the service providing server 100. The object division unit 101 searches for the accuracy that matches the detection object and the number of frames, then inputs the detection object, the number of frames, and the accuracy as input values to the frame number determination model, and obtains the corresponding number of frames as the result value. You can. The object segmentation unit 101 may set the number of frames obtained from the frame number determination model as the standard number of frames continuously processed by the video object segmentation model.

Figures 8 and 9 are diagrams illustrating a process in which an inpainting model removes a dynamic object detected in a video and restores the removed part.

The inpainting unit 102 may provide a frame and a frame mask as input values to the inpainting model, and obtain a frame from which the mask portion has been removed and restored from the inpainting model.

The inpainting unit 102 may provide a frame and a frame mask as input values to the inpainting model, remove the background excluding the mask portion, and obtain a frame with the background restored from the inpainting model.

The inpainting unit 102 may provide the user terminal 200 with a user interface through which a user can input a prompt for requesting removal and restoration of the mask or background.

The inpainting unit 102 receives a user prompt provided by the user from the user terminal 200. The inpainting unit 102 uses a user prompt to generate an input prompt to be input to the inpainting model, and provides the input prompt and negative prompt along with a frame and mask to the inpainting model. The inpainting model receives frames and masks for input prompts and negative prompts, generates output frames, and provides them to the inpainting unit 102. In one embodiment, the inpainting unit 102 may generate an input prompt that is the same as the user prompt.

The inpainting unit 102 provides an output frame to the user terminal 200.

For example, if “remove it” is input to the inpainting model as an input prompt along with the input frame and mask, the inpainting model may remove the mask portion and generate an output frame in which the removed portion is restored.

For example, if "remove it with tree" is input to an inpainting model as an input prompt along with an input frame and a mask, the inpainting model will generate an output frame with the masked portion removed and a tree created in the removed portion. You can.

In one embodiment, the inpainting unit 102 may input a negative prompt along with the input prompt to improve image creation performance.

In one embodiment, the inpainting model can be trained to avoid output images labeled for negative prompts when negative prompts are input using training data that labels the prompts used for negative output images as negative prompts.

In one embodiment, the inpainting unit 102 may use a negative word generation model to obtain a negative word to improve the quality of the output image and generate a negative prompt using the negative word. The negative word generation model can be trained through machine learning using training data generated by labeling objects included in the frame, objects removed from the frame, and negative wording. For example, if the objects included in the frame are "car", "window", "human", "balcony", "pillar", and "entrance door", and the object to be removed is "human", then "car", Preprocess "window", "human", "balcony", "pillar", "entrance door", "human", negative wording "cartoon", "cgi", "render", "illustration", "painting", Training data can be created by labeling “drawing” and “front door,” and a negative word generation model can be created through learning using the training data. For example, various known deep learning algorithms such as ANN (Artificial Neural Networks), RNN (Recurrent Neural Networks), and LSTM (Long Short-Term Memory) can be used for learning.

In one embodiment, the inpainting unit 102 inputs the first frame of the video into an image segmentation model to obtain all objects included in the first frame, and all objects included in the first frame and the removed objects are converted into a negative word generation model. It is input as an input value, and a negative word can be obtained as a result value from the negative word generation model. The inpainting unit 102 may generate a negative prompt by listing the obtained negative words. For example, if the negative words are "cartoon", "cgi", "render", "illustration", "painting", and "drawing", use "cartoon, cgi, render, illustration, painting, drawing" as the negative prompt. Create. The inpainting unit 102 may input a video frame, each video frame's mask, input prompt, and negative prompt into an inpainting model, and obtain an output frame corresponding to each video frame from the inpainting model. The inpainting unit 102 provides the user terminal 200 with an output frame generated in correspondence with each video frame.

FIG. 10 is a flowchart showing a process in which the service providing server according to FIG. 1 synthesizes a special effect video with the original video.

Referring to FIG. 10, the synthesis unit 105 designates a special effect input area in the original video using the area received from the user terminal 200 (S310).

The synthesis unit 105 provides the user terminal 200 with a user interface through which original video to be used for combining special effects can be input. The synthesis unit 105 provides the user terminal 200 with a user interface that allows designating an area to synthesize special effects in the original video. The synthesis unit 105 may receive an original video from the user terminal 200, and may receive an area in which a special effect is to be synthesized for at least one of a plurality of frames included in the original video.

The synthesis unit 105 can designate a special effect input area in the original video using the area where the special effect received from the user terminal 200 is to be synthesized.

In one embodiment, the synthesis unit 105 may designate an area for combining special effects received from the user terminal 200 as a special effect input area in the original video.

Figure 11 is a flowchart showing the specific process of step S310 of Figure 10.

In one embodiment, when a special effect input area for the first frame is received from the user terminal 200 and a special effect input area is designated for the first frame, the compositing unit 105 enters the object division unit 101. You can request dynamic object tracking for the remaining frames.

The object segmentation unit 101 uses an image segmentation model to create a mask that matches the object included in the special effect input area of the first frame (S311).

The object division unit 101 inputs the special effect input area into the image segmentation model in the first frame, the image segmentation model identifies objects included in the special effect input area, and provides the identified object to the user terminal 200. do.

The object division unit 101 may receive a target object that will be the standard for dynamic object tracking among objects provided from the user terminal 200. The object division unit 101 may provide a user interface for selecting one of the objects provided to the user terminal 200. Once the target object is determined, the object dividing unit 101 separates the target object area and the background area and generates a mask matching the target object area.

The object segmentation unit 101 detects objects included in the remaining frames of the original video using the mask generated in the first frame and the video object segmentation model, and separates the remaining frames into a target object area and a background area (S312).

The specific process of separating the target object area and the background area in consecutive frames has been described in detail with reference to FIGS. 5 to 7, and this will be replaced.

When the target object area and the background area are separated for each of the remaining frames, the synthesis unit 105 creates a special effect input area for each of the remaining frames using the target object area of each of the remaining frames (S313).

In one embodiment, the synthesis unit 105 may generate a special effect input area for each of the remaining frames using the positional relationship between the target object area and the special effect input area of the first frame. For example, when the special effect input area is formed in a square, the composite unit 105 uses the left, right, upper, and lower shortest distances between the perimeter of the target object area of the first frame and the perimeter of the special effect input area. You can create a special effect input area for each remaining frame.

Referring again to FIG. 10, the video generator 104 generates a special effect video using the video generation model (S320).

FIG. 12 is a flowchart showing the specific process of step S320 of FIG. 10. Figure 13 is a diagram conceptually showing the process of creating a special effects video.

Referring to FIG. 12, the image generator 103 generates a special effect image using an image generation model (S321). For example, the image generator 103 may input “flame on a chromakey green background” as an input prompt to the image generation model and obtain an image of a flame burning on a green chromakey background from the image generation model. there is.

The video generator 104 generates a special effects video using the generated special effects image and video generation model (S322).

In one embodiment, when the video generation model is learned with images and video sequences related to the images as training data, the video generation unit 104 inputs the generated special effect image to the video generation model and retrieves the image from the video generation model. Acquire video sequences linked to special effect images.

In one embodiment, when the video generation model is learned with text and images and video sequences related to the text and images as training data, the video generation unit 104 inputs the special effect image and input prompt generated into the video generation model. And obtain video sequences corresponding to special effect images and input prompts from the video generation model. For example, a special effect image of a burning flame on a green chroma key background and an input prompt such as “flame is burning” can be input to the video generation model.

In an embodiment not shown, the video generator 104 may generate a special effect video using a special effect image pre-stored in a database, rather than a special effect image generated using an image generation model.

In an embodiment not shown, when the video generation model is learned with text and video sequences related to the text as training data, the video generation unit 104 does not generate a special effect image using the image generation model, but inputs the special effect image. Only the prompt can be input into the video generation model, and a video sequence corresponding to the input prompt can be obtained from the video generation model.

When a special effect video (video sequence) is created, the object segmentation unit 101 separates the special effect object area and the background area from the special effect video using an image segmentation model (S323).

In one embodiment, the object segmentation unit 101 may identify special effect objects included in a special effect video using an image segmentation model. For example, the object segmentation unit 101 may input each of a plurality of frames included in a special effect video into an image segmentation model and obtain a special effect object area included in each of the plurality of frames from the image segmentation model. .

In one embodiment, the object segmentation unit 101 inputs the first frame of the special effect video into an image segmentation model, and the image segmentation model creates a mask corresponding to the special effect object included in the special effect input area.

The object segmentation unit 101 detects objects included in the remaining frames of the special effect video using the mask generated in the first frame and the video object segmentation model, and separates the remaining frames into a special effect object area and a background area.

The specific process of separating the object area and the background area in consecutive frames has been described in detail with reference to FIGS. 5 to 7, and this will be replaced.

The video generator 104 removes the background area from each of the plurality of frames included in the special effect video (S324).

Referring again to FIG. 10, the synthesis unit 105 synthesizes a special effect video into the special effect input area of the original video (S330).

The synthesis unit 105 matches a plurality of frames of the original video with a plurality of frames of the special effects video 1:1, and inserts each of the plurality of frames of the special effect video into the special effect input area of each of the plurality of frames of the original video. synthesize.

FIG. 14 is a flowchart showing a process in which the service providing server 100 according to FIG. 1 generates a 3D model.

Referring to FIG. 14, the video generator 104 generates a plurality of modeling images captured from multiple angles using a video generation model (S410).

In one embodiment, the video generator 104 may input an input prompt to the video generation model and obtain a plurality of modeling images from the video generation model. For example, the video creation unit 104 may say, “A video sequence showcasing a chair from various angles, suitable for generation with a video creation model. The video should feature the chair in different positions: frontal, side, rear, and diagonal views. By entering ", you can obtain multiple modeling images of the chair taken at various angles from the video generation model.

In one embodiment, the image generator 103 may input an input prompt to an image generation model and obtain a plurality of modeling images from the image generation model. For example, the image generator 103 may say, "A collection of images showcasing a chair from various angles, suitable for generation with Stable Diffusion. The images should feature the chair in different positions: frontal, side, rear, and diagonal views. By entering ", you can obtain multiple modeling images of the chair taken at various angles from the image generation model.

FIG. 15 is a flowchart illustrating an example of step S410 of FIG. 14.

Referring to FIG. 15, the video generator 104 generates a plurality of modeling images using a video generation model (S411).

In one embodiment, the image generator 103 may generate a plurality of modeling images using an image generation model.

The video generator 104 designates a marker creation area in one of the plurality of modeling images (S412).

In one embodiment, the video generator 104 may designate a preset number of marker creation areas in object areas corresponding to objects included in the modeling image.

In one embodiment, the video generator 104 designates a preset number of internal marker creation areas in the object area corresponding to the object included in the modeling image, and creates a preset number of external markers in the background area other than the object area. You can specify the creation area.

In one embodiment, the video generator 104 may provide the user terminal 200 with a user interface that can designate a marker creation area in the modeling image. The video generator 104 may designate the marker creation area received from the user terminal 200 as the marker creation area of the modeling image.

The object division unit 101 separates the remaining modeling images among the plurality of modeling images into a marker creation area and a background area using the marker creation area and the video object segmentation model (S413).

The object segmentation unit 101 detects the marker creation area included in the remaining modeling image among the plurality of modeling images using a video object segmentation model and a mask corresponding to the marker creation area specified in one of the modeling images. Then, separate the remaining frames into a marker creation area and a background area.

The specific process of separating a specific object area and a background area in consecutive frames has been described in detail with reference to FIGS. 5 to 7, and this will be replaced.

The inpainting unit 102 creates a marker in the marker creation area of each of the plurality of modeling images using the inpainting model (S414).

FIG. 18 is a diagram illustrating a plurality of modeling images in which markers are created in a marker creation area using an inpainting model. Referring to FIG. 18, a marker (M) is created in the marker creation area of each insect modeling image generated at various angles.

In one embodiment, different markers may be inpainted into a marker creation area formed inside the object and a marker creation area created outside the object.

3D modeling performance can be improved as markers are created in modeling images.

FIG. 16 is a diagram illustrating an example of step S410 of FIG. 14.

Referring to FIG. 16, the image generator 103 generates a modeling image using the image generation model (S415).

The image generator 103 designates at least one marker creation area in the modeling image (S416).

In one embodiment, the image generator 103 may designate a preset number of marker creation areas in object areas corresponding to objects included in the modeling image.

In one embodiment, the image generator 103 designates a preset number of internal marker creation areas in the object area corresponding to the object included in the modeling image, and creates a preset number of external markers in the background area other than the object area. You can specify the creation area.

The inpainting unit 102 creates a marker in the marker creation area of the modeling image using the inpainting model (S417).

In one embodiment, different markers may be inpainted into a marker creation area formed inside the object and a marker creation area created outside the object.

The video generator 104 generates a plurality of modeling images photographed from various angles using the modeling image in which the marker is created and the video generation model (S418).

In one embodiment, the video generator 104 may input a modeling image with a marker and an input prompt to the video generation model and obtain a plurality of modeling images from the video generation model. For example, the video creation unit 104 may say, “A video sequence showcasing the input image from various angles, suitable for generation with a video creation model. The video should feature the input image in different positions: frontal, side, rear, and “diagonal views.” can be entered into the video generation model along with the modeling image from which the marker was created as an input prompt.

In one embodiment, the image generator 103 may generate a plurality of modeling images using an image generation model. For example, the image generator 103 says, "A collection of images showcasing the input image from various angles, suitable for generation with Stable Diffusion. The images should feature the input image in different positions: frontal, side, rear, and You can input “diagonal views” into the image generation model along with the modeling image from which the marker was created as an input prompt.

Referring again to FIG. 14, the 3D generation unit 106 generates a 3D model that matches a plurality of modeling images photographed from various angles using the 3D generation model (S420).

Figure 17 is a flowchart showing the specific process of step S420 of Figure 14.

The 3D generation unit 106 inputs a plurality of modeling images into a 3D generation model, feature points are determined in each of the plurality of modeling images by the 3D generation model (S421), and the 3D generation model creates a plurality of modeling images using the feature points. Create a 3D model for (S422).

In one embodiment, the 3D generation model determines feature points in each of a plurality of modeling images, calculates the arrangement of the feature points in 3D space based on the relationship between the determined feature points, and generates a 3D model based on the calculated arrangement. do.

The UV Map is mapped to the 3D model by the 3D generated model (S423).

Figure 19 is a diagram illustrating a UV map as an example.

In 3D modeling, UV Mapping refers to the process of applying 2D texture to the surface of a 3D model. UV refers to the axes of 2D texture space, which are used to avoid confusion with the X, Y, and Z axes of 3D modeling space, and U and V refer to the horizontal and vertical axes of 2D texture, respectively. UV maps define where each point in a 3D model will be located on a 2D texture, accurately mapping the texture to complex 3D objects.

In one embodiment, the object division unit 101 separates the UV map into a designated marker removal area and a background area. The object division unit 101 may provide a UV map to the user terminal 200 and provide a user interface capable of specifying a marker removal area in the provided UV map. The object division unit 101 separates the marker removal area input from the user terminal and the background area. The inpainting unit 102 may remove the marker removal area from the UV map and then create a new removed area.

If you create a 3D model by displaying a marker, 3D model creation performance improves, but a problem occurs where the marker is displayed on the surface of the 3D model because the marker is included in the UV map. If you delete the marked area from the UV map and then inpaint it to map it to the 3D model, you can remove the marker from the surface of the 3D model.

FIG. 20 is a diagram illustrating an exemplary hardware configuration of the service providing server 100 according to FIG. 1.

Referring to FIG. 20, the service providing server 100 stores at least one processor 110 and instructions instructing the at least one processor 110 to perform at least one operation. may include memory.

The at least one operation may include the components 101 to 106 of the service providing server 100 described above or other functions or operation methods.

Here, the at least one processor 110 may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to embodiments of the present invention are performed. there is. Each of the memory 120 and the storage device 160 may be comprised of at least one of a volatile storage medium and a non-volatile storage medium.

For example, the memory 120 may be one of read only memory (ROM) and random access memory (RAM), and the storage device 160 may be flash memory. , a hard disk drive (HDD), a solid state drive (SSD), or various memory cards (eg, micro SD card).

Additionally, the server 100 may include a transceiver 130 that performs communication through a wireless network. Additionally, the server 100 may further include an input interface device 140, an output interface device 150, a storage device 160, etc. Each component included in the server 100 is connected by a bus 170 and can communicate with each other.

Examples of the server 100 include a desktop computer, a laptop computer, a laptop, a smart phone, a tablet PC, and a mobile phone that can communicate. , smart watch, smart glass, e-book reader, PMP (portable multimedia player), portable game console, navigation device, digital camera, DMB (digital multimedia broadcasting) player. , a digital audio recorder, a digital audio player, a digital video recorder, a digital video player, a PDA (Personal Digital Assistant), etc.

Methods according to the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. Computer-readable media may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on a computer-readable medium may be those specifically designed and configured for the present invention, or may be known and usable by those skilled in the art of computer software.

Examples of computer-readable media may include hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions may include machine language code such as that created by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The above-described hardware device may be configured to operate with at least one software module to perform the operations of the present invention, and vice versa.

Additionally, the above-described method or device may be implemented by combining all or part of its components or functions, or may be implemented separately.

Although the present invention has been described above with reference to preferred embodiments, those skilled in the art may make various modifications and changes to the present invention without departing from the spirit and scope of the present invention as set forth in the following patent claims. You will understand that you can do it.

Claims (5)

As a server that provides special effects generation and 3D model generation services using generative AI models,
an object segmentation unit that determines objects included in the input image using an image segmentation model;
A video generator that generates a special effects video using a video generation model; and
It includes a synthesis unit that specifies a special effect input area in the original video using a user-specified area received from the user terminal and synthesizes a special effect video in the special effect input area of the original video;
The object division unit,
Using the image segmentation model and video object segmentation model, each of the multiple frames included in the special effect video is separated into a special effect object area and a background area,
Remove the background area from each of the multiple frames included in the special effects video,
The synthetic part,
Designate the user-specified area as the special effect input area of the first frame of the original video,
The object division unit,
Identify at least one object included in the special effect input area of the first frame of the original video using an image segmentation model,
Determines one of at least one object as the target object and creates a mask matching the target object,
Each remaining frame of the original video is separated into a target object area and a background area using a mask and video object segmentation model that matches the target object.
The synthetic part,
The target object area of each remaining frame of the original video is used to create a special effect input area of each remaining frame of the original video.
The server is,
Using each of the plurality of frames included in the special effects video and an image generation model, a plurality of frames are matched to each of the plurality of frames included in the special effects video and the special effect objects included in each of the plurality of frames are shown from various angles. An image generator that generates a modeling image; and
It further includes a 3D generation unit that generates a 3D model matching a plurality of modeling images using the 3D generation model,
The video generator,
Designate a marker creation area on one of the plurality of modeling images,
The object division unit,
Using the marker creation area and video object segmentation model, the remaining modeling images among the plurality of modeling images are separated into a marker creation area and a background area.
The server is,
An inpainting unit that generates a marker in the marker creation area of each of the plurality of modeling images using an inpainting model,
server. delete According to paragraph 1,
The synthetic part,
1:1 matching of each of the multiple frames of the original video and each of the multiple frames of the special effects video,
Composing each of a plurality of frames of a special effect video into the special effect input area of each of a plurality of frames of the original video,
server. delete delete