CN117893419A - Video generation method, device, electronic equipment and readable storage medium - Google Patents

Abstract

The present application discloses a video generation method, device, electronic device and readable storage medium, including: obtaining a video to be fused and an initial image, and performing frame decomposition on the video to be fused to obtain multiple frame images to be fused; obtaining a mask image of the target object in the initial image; fusing the initial image with each frame image to be fused based on a specified algorithm to obtain multiple images to be processed; determining a first color parameter corresponding to each image to be processed based on the mask image, and determining a second color parameter based on the initial image; relighting each image to be processed based on the second color parameter, the mask image, the initial image and the first color parameter corresponding to each image to be processed to obtain a target image; generating a target video based on the target image. The video to be fused and the initial image are fused by a specified algorithm, and the obtained image can be relighted by color parameters, thereby obtaining a target video with more realistic lighting and shadows.

Description

Video generation method, device, electronic device and readable storage medium

Technical Field

The present application relates to the field of image processing technology, and more specifically, to a video generation method, device, electronic device and readable storage medium.

Background technique

At present, with the development of electronic information technology, some elements in dynamic videos can be added to images to generate dynamic effect videos. However, the current methods of generating dynamic effect videos consume a lot of manpower, and the lighting and shadows of the generated dynamic effect videos are not realistic enough.

Summary of the invention

The present application proposes a video generation method, device, electronic device and readable storage medium.

In a first aspect, an embodiment of the present application provides a video generation method, comprising: obtaining a video to be fused and an initial image, and performing frame decomposition on the video to be fused to obtain a plurality of frame images to be fused; obtaining a mask image of a target object in the initial image; fusing the initial image with each of the frame images to be fused based on a specified algorithm to obtain a plurality of images to be processed; determining a first color parameter corresponding to each of the images to be processed based on the mask image, and determining a second color parameter based on the initial image; re-lighting each of the images to be processed based on the second color parameter, the mask image, the initial image, and the first color parameter corresponding to each of the images to be processed to obtain a target image; and generating a target video based on the target image.

In the second aspect, the embodiment of the present application also provides a video generation device, including: a first acquisition unit, a second acquisition unit, a fusion unit, a color parameter determination unit, a relighting unit and a video generation unit. Among them, the first acquisition unit is used to acquire the video to be fused and the initial image, and perform frame decomposition on the video to be fused to obtain multiple frame images to be fused; the second acquisition unit is used to acquire the mask image of the target object in the initial image; the fusion unit is used to fuse the initial image with each of the frame images to be fused based on a specified algorithm to obtain multiple images to be processed; the color parameter determination unit is used to determine the first color parameter corresponding to each of the images to be processed based on the mask image, and determine the second color parameter based on the initial image; the relighting unit is used to relight each of the images to be processed based on the second color parameter, the mask image, the initial image and the first color parameter corresponding to each of the images to be processed to obtain a target image; the video generation unit is used to generate a target video based on the target image.

In a third aspect, an embodiment of the present application further provides an electronic device comprising: one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method described in the first aspect.

In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the method described in the first aspect above.

The video generation method, device, electronic device and readable storage medium provided by the embodiment of the present application first obtain the video to be fused and the initial image, and perform frame decomposition on the video to be fused to obtain multiple frame images to be fused; then obtain the mask image of the target object in the initial image; fuse the initial image with each of the frame images to be fused based on a specified algorithm to obtain multiple images to be processed; then determine the first color parameter corresponding to each of the images to be processed based on the mask image, and determine the second color parameter based on the initial image; relight each of the images to be processed based on the second color parameter, the mask image, the initial image and the first color parameter corresponding to each of the images to be processed to obtain the target image; finally generate the target video based on the target image. First, the initial image is fused with each of the frame images to be fused by a specified algorithm to obtain multiple images to be processed, which avoids manual editing, saves labor costs, and improves the efficiency of obtaining the images to be processed. In addition, by relighting the obtained image through color parameters, a target video with more realistic lighting and shadows can be obtained.

Other features and advantages of the embodiments of the present application will be described in the subsequent description, and partly become apparent from the description, or can be understood by practicing the embodiments of the present application. The purposes and other advantages of the embodiments of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

FIG1 shows an application scenario diagram of a video generation method provided by an embodiment of the present application;

FIG2 shows a method flow chart of a video generation method provided by an embodiment of the present application;

FIG3 is a schematic diagram of an initial image provided by an embodiment of the present application;

FIG4 is a schematic diagram of a frame image to be fused provided by an embodiment of the present application;

FIG5 is a schematic diagram of a mask image provided by an embodiment of the present application;

FIG6 is a schematic diagram showing a target image provided in an embodiment of the present application;

FIG7 shows a method flow chart of a video generation method provided by another embodiment of the present application;

FIG8 is a schematic diagram showing a line drawing to be fused provided in an embodiment of the present application;

FIG9 shows a schematic diagram of an initial line drawing provided by an embodiment of the present application;

FIG10 is a schematic diagram showing a line drawing image to be processed provided by an embodiment of the present application;

FIG11 is a flowchart of a video generation method provided by another embodiment of the present application;

FIG12 is a flowchart of a video generation method provided by yet another embodiment of the present application;

FIG13 shows a method flow chart of a video generation method provided by another embodiment of the present application;

FIG14 shows a structural block diagram of a video generation device provided in an embodiment of the present application;

FIG15 shows a structural block diagram of an electronic device provided in an embodiment of the present application;

FIG. 16 shows a block diagram of the structure of a computer-readable storage medium provided in an embodiment of the present application.

Detailed ways

In order to make those skilled in the art better understand the present application scheme, the technical scheme in the present application embodiment will be clearly and completely described below in conjunction with the drawings in the present application embodiment. Obviously, the described embodiment is only a part of the present application embodiment, rather than all the embodiments. The components of the present application embodiment usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents the selected embodiment of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

At present, with the development of electronic information technology, some elements in dynamic videos can be added to images to generate dynamic effect videos. However, the current methods of generating dynamic effect videos consume a lot of manpower, and the lighting and shadows of the generated dynamic effect videos are not realistic enough. How to reduce the manpower consumed in generating dynamic effect videos and improve the realism of the lighting and shadows of the generated dynamic effect videos is an urgent problem to be solved.

The motion effect video may be a video generated by editing the elements of the motion effect on a static image, and the elements of the motion effect may be motion effect objects extracted from the video, such as motion effect objects extracted from the motion effect video.

In the prior art, dynamic elements can be added to a static image manually, for example, by using image editing software or animation software to add dynamic elements to a static image manually.

However, the inventors found in their research that manually adding dynamic elements to static images takes a long time and requires a lot of manpower. In addition, the lighting and shadows of the generated dynamic effect videos are not realistic enough.

Therefore, in order to solve or partially solve the above problems, the present application provides a video generation method, device, electronic device and readable storage medium.

Please refer to Figure 1, which shows an application scenario diagram of the video generation method provided in an embodiment of the present application, namely, a video generation scenario 100. The video generation scenario 100 may include an electronic device 110 and a server 120, wherein the electronic device 110 is connected to the server 120.

The electronic device 110 can access the Internet and thereby establish a connection with the server 120 which is also connected to the Internet. The electronic device 110 can access the Internet wirelessly, such as through wireless communication technologies such as Wi-Fi and Bluetooth, etc.; the electronic device 110 can also access the Internet in a wired manner, such as through an Rj45 network cable or optical fiber.

The user can control the electronic device 110 so that the electronic device executes the video generation method. For example, the user can directly operate the electronic device 110 to control the electronic device to execute the video generation method. The electronic device 110 can be locally deployed with a specified algorithm, for example, the specified algorithm can be pre-stored, so that the specified algorithm can be called to achieve video generation. For a detailed description, please refer to the subsequent implementation. Optionally, the electronic device 110 can also call a specified algorithm deployed in the server 120 to execute video generation.

The server 120 may be a cloud server or a local server.

For some embodiments, the video generation method can be applied to entertainment and games, for example, it can be used to generate corresponding video content for entertainment industries such as movies, TV series, cartoons, games, etc. It can also be applied to advertising and marketing, for example, it can be used to generate attractive advertising and marketing video content. Specific video content may include product demonstrations, dynamic advertising banners, animated logos, etc. It can also be applied to virtual reality (VR) and augmented reality (AR), for example, video content that can be used for virtual reality and augmented reality can be generated. Specific video content may include virtual scenes, dynamic elements, and interactive effect displays, etc. It can also be applied to the design and creative fields, for example, video content that can be used to assist the design and creative process can be generated. Specific video content may include dynamic design prototypes, artistic effects, visual effect previews, etc. In addition, it can also be applied to social media and emoticons, for example, video content that can be used for social media platforms and chat applications can be generated. Specific video content may include interesting dynamic emoticons and animated images that express rich emotions. Furthermore, it can also be applied to data visualization, for example, video content that can be used in the field of data visualization can be generated. Specifically, the video content may include dynamic charts, interactive graphics, and dynamic data displays.

It should be noted that the application scenarios of the video generation method provided by the present application shown above are only some examples and do not constitute a limitation on the embodiments of the present application.

Please refer to Figure 2, which shows a method flow chart of a video generation method provided by an embodiment of the present application. The video generation method can be applied to the electronic device in the video generation scene shown in Figure 1, and specifically, the processor of the electronic device can be used as the execution subject of the video generation method. The video generation method may include steps S110 to S160.

Step S110: obtaining the video to be fused and the initial image, and performing frame decomposition on the video to be fused to obtain a plurality of frame images to be fused.

Some motion effects in the video can be fused into the image, and then a video with motion effects can be generated. Therefore, the initial image and the video to be fused can be obtained first. The initial image can be the image to which the motion effects need to be fused, or it can be understood that the motion effects can be fused on the basis of the initial image; and the video to be fused is the video containing the motion effect object.

In some embodiments, the electronic device may pre-store some image files and video files, so that an image file may be selected from the pre-stored image files as the initial image; and a video file may be selected from the pre-stored video files as the video to be fused. Exemplarily, the electronic device may run a photo and video application, so that an image file may be selected from the photo and video application as the initial image; and a video file may be selected as the video to be fused.

In other embodiments, the electronic device can obtain an image file that is not stored locally as an initial image and obtain a video file as a video to be fused through an application. For example, the electronic device can run a web browsing application, and the web browsing application can correspond to an image file and a video file. Thus, the electronic device can obtain an image file in the web browsing application as an initial image and obtain a video file in the web browsing application as a video to be fused.

Exemplarily, the video to be fused may be a file in mp4 format, mkv format, mov format, etc., or may be a file in gif format, which is not specifically limited in the embodiments of the present application.

For example, please refer to Figure 3, which shows a schematic diagram of an initial image provided by an embodiment of the present application. The initial image 300 shown in Figure 3 includes a target object 301 and a background 302. The background 302 may be an area other than the target area corresponding to the target object 301.

It can be understood that the video to be fused can be composed of multiple frame images. For example, each second of the video to be fused can include a specified number of frame images. Specifically, the specified number can be 24, 30, 50, 60, 120, etc. Among them, the specified number of frame images included in each second can also be called the frame rate of the video to be fused. For example, if the specified number is 24, the frame rate of the video to be fused is 24; if the specified number is 60, the frame rate of the video to be fused is 60. Therefore, in order to fuse the content in the video to be fused into the initial image, the video to be fused can be frame-decomposed to obtain multiple frame images to be fused. Among them, multiple frame images to be fused are multiple frame images in the video to be fused.

For example, if the video to be fused includes N frames of images, each frame of image to be fused can be represented in sequence by a_n, where n=1, 2, 3...N.

For example, please refer to Fig. 4, which shows a schematic diagram of a frame image to be fused provided by an embodiment of the present application. The frame image 400 to be fused shown in Fig. 4 includes an object 401 to be fused.

Step S120: Acquire a mask image of the target object in the initial image.

In order to facilitate the subsequent acquisition of a target video with more realistic lighting and shadows, a mask image of the target object in the initial image may be first acquired, so that the mask image may be directly called later.

Among them, through the mask image, some specified areas in the image can be easily operated without affecting the areas in the image other than the specified areas. In some embodiments, the mask image can be a binary or Boolean image of the same size as the initial image, in which the selected area is marked as 1 (or True), and the remaining areas are marked as 0 (or False). Among them, the selected area is the target area corresponding to the target object in the initial image; and the remaining areas are the areas in the initial image other than the target area corresponding to the target object. Therefore, visually, the target area corresponding to the target object in the mask image can be one color, and the area other than the target area corresponding to the target object can be another color.

The target object may be a main character in the initial image, such as the target object 301 in the aforementioned FIG. 3 .

For some implementations, a mask image of the target object in the initial image may be obtained by a subject cutout model, wherein the subject cutout algorithm may identify the subject person in the input image, thereby obtaining the mask image of the target object in the initial image.

For some implementations, the subject cutout model may be obtained by adjusting or training a pre-trained model in advance. The subject cutout model may be implemented based on a u2net algorithm. The u2net algorithm is an image segmentation network algorithm based on a U-Net structure.

For example, please refer to Fig. 5, which shows a schematic diagram of a mask image provided by an embodiment of the present application, wherein the mask image 500 includes a target area 501 corresponding to the target object and an area 502 other than the target area corresponding to the target object.

Step S130: Based on a specified algorithm, the initial image is fused with each of the frame images to be fused to obtain a plurality of images to be processed.

It is understandable that, in order to generate a video later, multiple frame images need to be merged to obtain the video. In other words, the initial image can be fused with each frame image to be fused respectively, so that multiple images to be processed can be obtained. Subsequently, the multiple images to be processed can be processed to generate a video.

For some implementations, in order to reduce the need for manual work in generating videos and reduce the time required to generate videos, the initial image may be fused with each of the frame images to be fused by a specified algorithm to obtain multiple images to be processed.

The specified algorithm may be a graph-to-graph algorithm. For example, the specified algorithm may be executed by stable diffusion running in an electronic device. By intelligently optimizing the generation capability of stable diffusion, the process of generating a target video may be made more efficient and accurate.

Optionally, the inpainting algorithm can be called in combination with the image generation algorithm to obtain a better image to be processed. For detailed description, please refer to the subsequent embodiments. Among them, the inpainting algorithm is an algorithm technology based on image restoration, which is mainly used to repair missing or damaged parts in the image. The inpainting algorithm analyzes the information of the surrounding images and uses the undamaged information in the image to perform image restoration or repair. It can be used to restore damaged photos and remove or reduce noise in the image.

It can be understood that each image to be processed includes a target object and a motion effect object, and the motion effect object is the object to be fused in the image frame to be fused. In other words, the target object in each image to be processed is the same, that is, the target object in the initial image. However, the motion effect object in each image to be processed is different, that is, the motion effect object in each image to be processed corresponds to the object to be fused in the image frame to be fused that generates the image to be processed.

Therefore, in the embodiment of the present application, by directly calling the specified algorithm to fuse the initial image with each of the frame images to be fused, a plurality of images to be processed can be obtained, which can avoid manually processing the initial image and each frame image to be fused in sequence, reducing the demand for manual work and improving the efficiency of obtaining a plurality of images to be processed, thereby improving the efficiency of generating videos as a whole. At the same time, it can reduce the errors caused by manual work and further improve the stability of the video acquisition method.

Step S140: determining a first color parameter corresponding to each of the to-be-processed images based on the mask image, and determining a second color parameter based on the initial image.

After acquiring multiple images to be processed, since the area other than the target area of the target object in each image to be processed has changed compared to the initial image, the target object can be re-lit based on the color parameters of the area other than the target area of the target object in the image to be processed, so that the light and shadow of the target object are more realistic.

The first color parameter corresponding to each of the images to be processed may be obtained, wherein the first color parameter may be a color parameter corresponding to an area other than the target area of the target object in each of the images to be processed. Specifically, the first color parameter corresponding to each of the images to be processed may be determined based on the mask image. In some embodiments, the area other than the target area corresponding to the target object in the image to be processed may be determined first by the mask image, and then the first color parameter may be determined. For a detailed description, please refer to the subsequent embodiments.

Furthermore, a second color parameter may be determined based on the initial image, wherein the second color parameter is the color parameter corresponding to the initial image.

The color parameter can be determined by the parameters of the red, green and blue (RGB) color channels, and the color parameter is the aforementioned first color parameter and second color parameter. The specific determination method can be referred to in the subsequent embodiments.

Step S150: relighting each of the images to be processed based on the second color parameter, the mask image, the initial image and the first color parameter corresponding to each of the images to be processed to obtain a target image.

Furthermore, after obtaining the first color parameter corresponding to each of the images to be processed and the second color parameter of the initial image, the image to be processed can be re-lit to obtain an image with more realistic lighting and shadows. The re-lighting adjustment can eliminate the problem of uncoordinated light and shadow, making the generated target image more consistent with the light and shadow of the initial image, enhancing the audience's immersion and sense of reality, and making the target image more coordinated and natural.

Specifically, the target area of the target object in the initial image can be screened out by using a mask image, and then the target area of the target object in the screened initial image can be re-lit. The re-lighting can be performed based on a specified color parameter, and the specified color parameter can be determined based on the second color parameter and the first color parameter corresponding to each image to be processed. For a detailed description, please refer to the subsequent embodiments.

However, for the area in the initial image other than the target area corresponding to the target object in the image to be processed, no re-lighting may be performed.

Furthermore, the target area of the target object in the re-illuminated initial image can be fused with the area in the initial image that has not been re-illuminated except the target area corresponding to the target object in the image to be processed to obtain the target image. For a detailed introduction, please refer to the subsequent embodiments.

It is understandable that each frame image to be fused may generate a corresponding image to be processed, and then determine a target image. Therefore, if there are N frame images to be fused, there may be N images to be processed and N target images.

For example, please refer to FIG6 , which shows a schematic diagram of a target image provided in an embodiment of the present application. The target image 600 shown in FIG6 includes a target object 601 and a background 602, wherein the area corresponding to the background 602 is the area other than the area where the target object 601 is located. The background 602 also includes a motion effect object 603.

Step S160: Generate a target video based on the target image.

After acquiring the target image, a target video can be generated based on the target image. Among them, multiple target images can be regarded as frame images, and then the frame images are merged to generate a target video. In some embodiments, image video editing software can be called to generate a target video based on the target image. The generated target video may include a target object and a motion effect object. Moreover, in the target video, the motion effect object changes and moves, while the content outside the motion effect object remains static.

The video generation method provided by the embodiment of the present application first obtains the video to be fused and the initial image, and performs frame decomposition on the video to be fused to obtain multiple frame images to be fused; then obtains the mask image of the target object in the initial image; based on the specified algorithm, the initial image is respectively fused with each of the frame images to be fused to obtain multiple images to be processed; then the first color parameter corresponding to each of the images to be processed is determined based on the mask image, and the second color parameter is determined based on the initial image; each of the images to be processed is re-lit based on the second color parameter, the mask image, the initial image, and the first color parameter corresponding to each of the images to be processed to obtain the target image; finally, the target video is generated based on the target image. First, the initial image is respectively fused with each of the frame images to be fused by a specified algorithm to obtain multiple images to be processed, which avoids manual editing, saves labor costs, and improves the efficiency of obtaining images to be processed. In addition, by re-lighting the obtained image through color parameters, a target video with more realistic lighting and shadows can be obtained.

Please refer to Figure 7, which shows a method flow chart of a video generation method provided by an embodiment of the present application. The video generation method can be applied to the electronic device in the video generation scene shown in Figure 1, and specifically, the processor of the electronic device can be used as the execution subject of the video generation method. The video generation method may include steps S210 to S2120.

Step S210: obtaining the video to be fused and the initial image, and performing frame decomposition on the video to be fused to obtain a plurality of frame images to be fused.

Among them, step S210 has been introduced in detail in the above embodiment and will not be repeated here.

Step S220: determining an initial mask image of the target object from the initial image based on the pre-acquired subject matting model.

Step S230: adjusting the resolution of the initial mask image to a specified resolution to obtain the mask image, where the specified resolution is the resolution of the latent space corresponding to the initial image.

First, an initial mask image of the target object can be determined from the initial image. Specifically, the initial mask image of the target object can be determined from the initial image by using a pre-acquired subject cutout model. The subject cutout model can be obtained by adjusting or training a pre-trained model in advance. The body cutout algorithm can identify the subject in the input image, thereby obtaining a mask image of the target object in the initial image.

Exemplarily, the output initial image may be input into the subject cutout model, thereby obtaining an image output by the subject cutout model as a mask image.

In some embodiments, the initial image may be encoded to obtain a latent feature image of the initial image in a latent space. That is, the initial image may be converted to the latent space for processing, and processing the initial image in the latent space may save computing resources.

Therefore, optionally, the image output by the subject cutout model can be used as the initial mask image, and then the resolution of the initial mask image can be adjusted so that the resolution of the initial mask image after the resolution adjustment is the same as the resolution of the latent space. Then, the latent feature image or the mask image can be processed in the latent space, saving computing resources.

Specifically, the resolution of the initial mask image may be adjusted to a specified resolution to obtain the mask image, where the specified resolution is the resolution of the latent space corresponding to the initial image. For example, the specified resolution may be 128*128; or, for another example, the specified resolution may be 64*64.

Step S240: Obtain the line drawing to be fused corresponding to each of the frame images to be fused.

In some implementations, the line drawing to be fused corresponding to each frame image to be fused may be first obtained, wherein the line drawing to be fused may be used to represent information such as lines and shapes of the frame image to be fused.

Exemplarily, a line drawing to be fused corresponding to each frame image to be fused can be generated by a line drawing acquisition algorithm. Specifically, the line drawing acquisition algorithm can be a well-trained linear detector (LineartDetector) in the control network (Controlnet), where the linear detector can be used to identify and locate the target object in the image, thereby realizing functions such as target tracking and recognition in the control task. The linear detector can use a linear classifier (such as SVM, logistic regression, etc.) to classify and locate objects in the image, classify the input image through a trained model, and output the category and position information of the object, so as to determine the line drawing to be fused corresponding to the frame image to be fused. Among them, the control network is a neural network architecture that can control the model and allow the model to support more input conditions. The original model can accept the input of prompt words and original images. The control network provides a variety of input conditions including canny edges, semantic segmentation maps, key points, graffiti, etc., which greatly improves the controllability of artificial intelligence generated content (AIGC).

From the above introduction, it can be known that after decomposing the video to be fused, multiple frame images to be fused can be obtained. Therefore, the line drawing image to be fused corresponding to each frame image to be fused can be obtained through the line drawing algorithm.

Please refer to FIG8 , which shows a schematic diagram of a line drawing to be fused provided in an embodiment of the present application. The line drawing 800 to be fused shown in FIG8 includes an object to be fused 801. Please refer to FIG4 and FIG8 together, and it can be seen that the line drawing 800 to be fused shown in FIG8 is the line drawing corresponding to the frame image 400 to be fused shown in FIG4 .

For some implementations, the line draft images to be fused of N frame images to be fused can be obtained in sequence. For example, for the frame image a_n to be fused, the line draft images to be fused of N frame images to be fused can be obtained in sequence from 1 to n; or the line draft images to be fused of N frame images to be fused can be obtained in sequence from n to 1. The embodiments of the present application do not make specific limitations.

Step S250: Acquire a depth image of the initial image and an initial line drawing of the initial image.

A depth image of the initial image may also be obtained. Exemplarily, the depth image of the initial image may be extracted by a deep learning model. For example, the deep learning model may be an algorithm for extracting a depth map from a monocular image, specifically an algorithm provided in diffusers. The algorithm for extracting a depth map from a monocular image may be used to predict the depth of the image based on an input image.

Furthermore, an initial line drawing of the initial image can also be obtained. The method for obtaining the initial line drawing of the initial image is similar to the method for obtaining the line drawing corresponding to the frame image to be fused in the aforementioned step, and the initial line drawing of the initial image can also be obtained by a line drawing obtaining algorithm. For a detailed introduction, please refer to the aforementioned steps, which will not be repeated here.

Please refer to Figure 9, which shows a schematic diagram of an initial line drawing provided by an embodiment of the present application. The initial line drawing 900 in Figure 9 includes a target object 901 and a background 902. The background 902 may be an area other than the target area corresponding to the target object 901.

Step S260: Fusing the initial line drawing with each line drawing to be fused respectively to obtain a plurality of line drawing images to be processed.

Furthermore, after obtaining the initial line draft image and a plurality of line draft images to be fused, the initial line draft image may be fused with each line draft image to be fused respectively, so as to obtain a plurality of line draft images to be processed.

In some implementations, a target position may be first determined in the initial line drawing, and the target position is used to insert the line drawing to be fused, so as to achieve fusion of the initial line drawing with each line drawing to be fused.

Among them, a plane coordinate system can be established with the initial line drawing, for example, a vertex of the initial line drawing is used as the coordinate origin, and the two sides of the initial line drawing connected to the origin are used as the x-axis and the y-axis to establish a plane coordinate system. Thus, the target position can be represented by the coordinates (x, y). Then, the designated position of the line drawing to be fused can be aligned with the target position, so as to achieve the fusion of the initial line drawing and the line drawing to be fused. For example, the designated position can be set as the vertex of the upper left corner of the line drawing to be fused, so that the upper left corner vertex of the line drawing to be fused can be set at the target position.

Exemplarily, the target position can be manually selected in the initial line drawing. For example, a certain point in the initial line drawing can be manually clicked, and the clicked point is used as the target position. In another exemplary embodiment, any position in the initial line drawing except the target area of the target object can be set as the target position.

In an exemplary embodiment, if the object to be fused in the line drawing to be fused is a fish, an area corresponding to water can be found in the area other than the target area of the target object in the initial line drawing, thereby determining a point in the water area as the target position. In another exemplary embodiment, if the object to be fused in the line drawing to be fused is a bird or an animal, an area corresponding to the sky can be found in the area other than the target area of the target object in the initial line drawing, thereby determining a point in the sky area as the target position.

Furthermore, the initial line drawing is fused with each line drawing to be fused, which can be done by inserting the line drawing to be fused at the target position and then adding the line drawing to be fused and the initial line drawing pixel by pixel, thereby obtaining a plurality of line drawings to be processed. It should be noted that each line drawing to be fused is fused with the initial line drawing at the same target position.

For example, please refer to Fig. 10, which shows a schematic diagram of a line drawing image to be processed provided by an embodiment of the present application. The line drawing image 1000 to be processed shown in Fig. 10 includes a target object 1010, a motion effect object 1020 and a background 1030.

Step S270: Encode the initial image based on the autoencoder to obtain a latent feature image of the initial image in a latent space.

From the above introduction, it can be seen that the initial image can be converted into the latent space to obtain the latent feature image, so that subsequent processing can be performed in the latent space to save computing resources.

For some embodiments, the initial image can be encoded by a variational autoencoder (VAE) to obtain a potential feature image of the initial image in the latent space. The autoencoder includes an encoder and a decoder. Therefore, the initial image can be encoded by the encoder in the autoencoder to obtain a potential feature image of the initial image in the latent space. The autoencoder is a model that learns to generate high-dimensional data represented by latent variables by combining an autoencoder and variational inference.

Step S280: adding Gaussian distribution noise to the potential feature image to obtain the noise image.

Further, noise may be added to the latent feature image to obtain the noise image. For some embodiments, Gaussian distribution noise may be added to the latent feature image to obtain the noise image. For example, Gaussian distribution noise may be added to the feature image multiple times. Specifically, T times of Gaussian distribution noise may be added to the noise image, wherein the forward step may be a Markov chain, so that each step of adding Gaussian distribution noise is only related to the previous step, thereby converting a picture into pure Gaussian noise, i.e., obtaining a noise image.

Step S290: Obtaining an image to be processed corresponding to each of the line draft images to be processed through the depth image, the line draft image to be processed and the noise image based on a specified algorithm, wherein the noise image is an image obtained by adding noise to the initial image.

Then, the depth image, the line image to be processed and the noise image can be combined and processed by a specified algorithm to obtain the image to be processed corresponding to the line image to be processed. The noise image is the image obtained by adding noise to the initial image in the above step. Specifically, the initial image can be first converted into a potential feature image in the latent space, and then Gaussian distribution noise is added to the potential feature image to obtain the noise image.

It is understandable that, since the noise image is an image in the latent space, the image obtained by the specified algorithm is also an image in the latent space. Therefore, it is necessary to further perform decoding processing on the image obtained by the specified algorithm.

Specifically, step S290 may also include step S291 and step S292.

Step S291: obtaining an image to be decoded corresponding to each of the line draft images to be processed through the depth image, the line draft image to be processed and the noise image based on a specified algorithm.

Step S292: Decode each of the to-be-decoded images based on the autoencoder to obtain an image to be processed corresponding to each of the to-be-processed line draft images.

Therefore, the image to be decoded corresponding to each of the line draft images to be processed can be first obtained based on the specified algorithm through the depth image, the line draft image to be processed and the noise image. However, the image to be decoded corresponding to each of the line draft images to be processed is an image located in the latent space, so each image to be decoded can also be decoded by the decoder of the autoencoder to obtain the image to be processed corresponding to each of the line draft images to be processed.

Specifically, each of the to-be-decoded images may be decoded by a decoder in the autoencoder.

Among them, the multiple images to be processed determined by combining the depth image, the line draft image to be processed and the noise image in the specified algorithm can be combined with the semantics and contextual understanding of the initial image, so that the generated image to be processed is more consistent with the theme or semantics of the initial image, thereby improving the overall quality and attractiveness of the subsequent target video, so that the target video forms a consistent visual style and theme with the initial image.

A detailed description of the image to be processed corresponding to each of the line draft images to be processed is obtained through the depth image, the line draft image to be processed and the noise image based on a specified algorithm, and reference may be made to subsequent embodiments.

Step S2100: determining a first color parameter corresponding to each of the to-be-processed images based on the mask image, and determining a second color parameter based on the initial image.

Step S2110: relighting each of the images to be processed based on the second color parameter, the mask image, the initial image and the first color parameter corresponding to each of the images to be processed to obtain a target image.

Step S2120: Generate a target video based on the target image.

Among them, the detailed description of steps S2100 to S2120 can be found in the aforementioned embodiment and will not be repeated here.

The video generation method provided in the embodiment of the present application can be converted to the latent space to process the initial image, and processing the initial image in the latent space can save computing resources. Moreover, the multiple images to be processed determined by combining the depth image, the line draft image to be processed, and the noise image in the specified algorithm can be combined with the semantics and contextual understanding of the initial image, so that the generated image to be processed is more consistent with the theme or semantics of the initial image, thereby improving the overall quality and attractiveness of the subsequent target video, so that the target video and the initial image form a consistent visual style and theme. In other words, in the embodiment of the present application, obtaining the target video through AIGC can better understand and express the intention of the initial image, thereby improving the visual effect and overall quality of the target video.

Please refer to Figure 11, which shows a method flow chart of a video generation method provided by an embodiment of the present application. The video generation method can be applied to the electronic device in the video generation scene shown in Figure 1, and specifically, the processor of the electronic device can be used as the execution subject of the video generation method. The video generation method may include steps S310 to S3120.

Step S310: obtaining a video to be fused and an initial image, and performing frame decomposition on the video to be fused to obtain a plurality of frame images to be fused.

Step S320: Acquire a mask image of the target object in the initial image.

Step S330: Obtain the line drawing to be fused corresponding to each of the frame images to be fused.

Step S340: Acquire a depth image of the initial image and an initial line drawing of the initial image.

Step S350: fusing the initial line drawing with each line drawing to be fused respectively to obtain a plurality of line drawing images to be processed.

Among them, step S310 to step S350 have been introduced in detail in the above embodiments and will not be repeated here.

Step S360: extracting depth feature information of the depth image based on the depth model.

Step S370: extracting line drawing feature information of the line drawing image to be processed based on the line drawing model.

In some embodiments, when obtaining the to-be-processed image corresponding to each of the to-be-processed line draft images through the depth image, the to-be-processed line draft image, and the noise image based on the specified algorithm, the depth feature information of the depth image and the line draft feature information of the to-be-processed line draft image may be first obtained. In other words, the input to the specified algorithm may include the depth feature information extracted from the depth image and the to-be-processed image corresponding to the to-be-processed line draft image. Thus, through the specified algorithm, the depth feature information and the line draft feature information may be considered, and then the features, semantics, and context information of the initial image may be considered, so that the target video generated subsequently can form a more coordinated and unified visual effect.

Specifically, the depth feature information of the depth image can be extracted by the depth model (Controlnet-depth); and the line feature information of the line image to be processed can be extracted by the line model (Controlnet-lineart). The depth image can be input into the depth model as a condition (Condition), thereby obtaining the depth feature information output by the depth model; and the line image to be processed can be input into the line model as a condition, thereby obtaining the line feature information output by the line model.

The line drawing feature information may be used to characterize the contour and edge features of the input line drawing to be processed.

Step S380: Add Gaussian noise to the target object in the noise image to obtain an intermediate image.

For some implementations, noise may be further added to the target object in the noise image, such as adding Gaussian noise, to obtain an intermediate image. The noise image may be an image obtained by adding noise to the initial image. For a detailed introduction to obtaining the noise image, please refer to the introduction of the aforementioned embodiment, which will not be repeated here.

Specifically, step S380 may include steps S381 to S384.

Step S381: determining a target area corresponding to a target object in the noise image based on the mask image as a first image area.

Step S382: determining, based on the mask image, an area in the noise image other than the first image area as a second image area.

Step S383: adding Gaussian noise to the first image area to obtain a noisy image area.

Step S384: Fusing the noise image area with the second image area to obtain the intermediate image.

It is understandable that in order to add Gaussian noise to the target object in the noise image, the region corresponding to the target object in the noise image can be determined first. Specifically, the target region corresponding to the target object in the noise image can be determined by a mask image as the first image region.

Furthermore, the region other than the first image region in the noise image may also be determined. Similarly, the region other than the first image region in the noise image may also be determined by the mask image as the second image region.

In order to add Gaussian noise to the target object in the noise image, Gaussian noise may be added to the first image area, but not to the second area. In some embodiments, Gaussian noise may be added to the first image area to obtain a noise image area. The noise image area is then fused with the second image area to obtain the intermediate image. That is, the second image area included in the intermediate image is an area to which Gaussian noise is not added again.

Exemplarily, steps S341 to S344 can be represented by the following formula (1):

E＝Di*C+E*(1-C) (1)

Among them, C is used to characterize the mask image, for example, it can be an image of 128*128 pixels; Di is used to characterize the noise image; Di*C is equivalent to adding Gaussian noise to the first area, that is, adding Gaussian noise to the target object in the noise image, thereby obtaining the noise image area; (1-C) corresponds to the background mask, that is, the area in the mask image except the target area of the target object; E*(1-C) is equivalent to not adding Gaussian noise to the second area, that is, not adding Gaussian noise to the area outside the target area of the target object in the noise image, which is equivalent to directly multiplexing the second area. That is, the fusion of the noise image area and the second image area is realized to obtain the intermediate image.

It should be noted that the output obtained by the above formula (1) can be a noise image after the noise is added again, that is, E, and E can be used as the input quantity for the next execution of formula (1), so as to implement the iteration of the noise image. The above formula (1) can be used to implement the iteration of E multiple times, so as to add Gaussian noise to the first area multiple times. For example, in some embodiments, the multiple times can be i times, that is, by iterating E i times, adding i times of Gaussian noise to the first area, so that the output E after i iterations is the intermediate image.

Step S390: Based on a specified algorithm, the intermediate image is subjected to noise reduction processing using the depth feature information and the line draft feature information to obtain an image to be processed corresponding to each of the line draft images to be processed.

Furthermore, after obtaining the depth feature information and the line draft feature information through the aforementioned steps, the intermediate image may be subjected to noise reduction processing by a specified algorithm to obtain an image to be processed corresponding to each of the line draft images to be processed.

It should be noted that first adding noise to an image and then performing noise reduction on the noisy image actually utilizes the inpainting algorithm. Since the inpainting algorithm is mainly used to repair missing or damaged parts in an image. This method analyzes the information of surrounding images and uses the undamaged information in the image to perform image restoration or repair. It is usually used to restore damaged photos and remove or reduce noise in images. Therefore, by first adding noise to an image and then calling the inpainting algorithm to perform noise reduction on the noisy image, the information of the image before the noise is added can be restored.

It should also be noted that if an image is directly denoised without the guidance of other conditions, and then the denoised image is denoised, even if the image can be restored, the restored image details will be lost in the process of encoding and decoding through the autoencoder, for example, in the process of ups and downsampling. Therefore, in the embodiment of the present application, in the process of denoising the intermediate image, the specified algorithm is guided by the depth feature information and the line feature information, which can improve the effect of the image to be processed corresponding to each of the line draft images to be processed. Specifically, it can avoid the appearance of unnatural artifacts at the junction of the target area corresponding to the target object and the area other than the target area corresponding to the target object in the obtained image to be processed.

Optionally, in some implementations, step S390 may also include step S391 and step S392.

Step S391: Obtain effect prompt words, where the effect prompt words are used to adjust the light and shadow effects of the intermediate image.

Optionally, in some implementations, in addition to using depth feature information and line drawing feature information to guide a specified algorithm to operate on the intermediate image, the intermediate image may also be further adjusted in combination with effect prompt words.

The effect prompt words can be used to adjust the light and shadow effects of the intermediate image. For example, some preset effect prompt words can be pre-stored, so that the user can directly select the desired effect prompt word from the preset effect prompt words. For another example, the user can also directly input the desired effect prompt word.

Exemplarily, the effect prompt words may be “gold light”, “dark night”, etc.

Step S392: Based on a specified algorithm, a target operation is performed on the intermediate image through the depth feature information, the line drawing feature information and the effect prompt word to obtain a to-be-processed image corresponding to each to-be-processed line drawing image, wherein the target operation includes noise reduction processing and light and shadow effect adjustment.

After obtaining the effect prompt word, the target operation can be performed on the intermediate image based on the specified algorithm through the depth feature information, the line draft feature information and the effect prompt word to obtain the image to be processed corresponding to each of the line draft images to be processed. The target operation includes noise reduction processing and light and shadow effect adjustment. Among them, the relevant introduction of guiding the specified algorithm to perform noise reduction processing on the intermediate image through the depth feature information and the line draft feature information can refer to the aforementioned relevant introduction of obtaining the image to be processed corresponding to each of the line draft images to be processed through the depth image, the line draft image to be processed and the noise image based on the specified algorithm, which will not be repeated here.

Furthermore, the effect prompt words may be used to guide a designated algorithm to adjust the light and shadow effects of the intermediate image, thereby obtaining more realistic light and shadow effects.

Step S3100: determining a first color parameter corresponding to each of the to-be-processed images based on the mask image, and determining a second color parameter based on the initial image.

Step S3110: relighting each of the images to be processed based on the second color parameter, the mask image, the initial image and the first color parameter corresponding to each of the images to be processed to obtain a target image.

Step S3120: Generate a target video based on the target image.

Among them, step S3100 has been introduced in detail in the above embodiment and will not be repeated here.

The video generation method provided in the embodiment of the present application can consider the depth feature information and the line feature information through the specified algorithm, and then realize the consideration of the features, semantics and context information of the initial image, so that the target video generated subsequently can form a more coordinated and unified visual effect. In addition, it is possible to avoid unnatural defects at the junction of the target area corresponding to the target object and the area other than the target area corresponding to the target object in the obtained image to be processed. In addition, in the embodiment of the present application, the specified algorithm can be guided by the effect prompt word to adjust the light and shadow effects of the intermediate image, so as to obtain more realistic lighting and shadow effects.

It should be noted that although the motion effect object in the generated target video is specifically determined by the motion effect object in the video to be fused, for example, if the motion effect object in the video to be fused is a dragon, the motion effect object in the generated target video is also a dragon. However, the lighting, shadow, display transparency, etc. of the motion effect object in the target video can be flexibly adjusted in the video generation method provided in this application, so that the generated target video has a more coordinated and unified visual effect.

Please refer to Figure 12, which shows a method flow chart of a video generation method provided by an embodiment of the present application. The video generation method can be applied to the electronic device in the video generation scene shown in Figure 1, and specifically, the processor of the electronic device can be used as the execution subject of the video generation method. The video generation method may include steps S410 to S4110.

Step S410: obtaining a video to be fused and an initial image, and performing frame decomposition on the video to be fused to obtain a plurality of frame images to be fused.

Step S420: Acquire a mask image of the target object in the initial image.

Step S430: Based on a specified algorithm, the initial image is fused with each of the frame images to be fused to obtain a plurality of images to be processed.

Among them, step S410 to step S430 have been introduced in detail in the above embodiments and will not be repeated here.

Step S440: determining, based on the mask image, an area outside the target area corresponding to the target object in the image to be processed as a third image area.

From the introduction of the foregoing embodiments, it can be seen that since the area other than the target area of the target object in each image to be processed has changed compared to the initial image, the target object can be re-lit based on the color parameters of the area other than the target area of the target object in the image to be processed, so that the light and shadow of the target object are more realistic.

Therefore, the area outside the target area corresponding to the target object in the image to be processed can be determined first. Specifically, the area outside the target area corresponding to the target object in the image to be processed can be determined based on the mask image as the third image area, and then the third image area can be re-lit. Exemplarily, the first color parameter can be determined based on the third image area, and then the second color parameter can be determined in combination with the target area corresponding to the target object in the initial image, so that the third image area can be re-lit in combination with the first color parameter and the second color parameter. Please refer to the subsequent steps for a detailed introduction.

Step S450: Obtain a first red parameter corresponding to the red channel, a first green parameter corresponding to the green channel, and a first blue parameter corresponding to the blue channel in each third image region.

It is understandable that the region of the image may include a plurality of pixels, each pixel being determined by three color channels of red, green and blue (RGB), and each color channel may have a different value, so that the pixel may display a different color.

Therefore, in some embodiments, the method for determining the first color parameter corresponding to the third image area may specifically be to first obtain the first red parameter corresponding to the red channel, the first green parameter of the green channel, and the first blue parameter corresponding to the blue channel in the third image area. The first red parameter may represent the color average of the red channel in the third image area, that is, the average value of the red channel of each pixel in the third image area. Similarly, the first green parameter may represent the color average of the green channel in the third image area, that is, the average value of the green channel of each pixel in the third image area. The first blue parameter may represent the color average of the blue channel in the third image area, that is, the average value of the blue channel of each pixel in the third image area.

It should be noted that, since there are multiple images to be processed, the first red parameter, the first green parameter, and the first blue parameter determined based on each image to be processed may be slightly different. Specifically, color analysis may be performed on the image to be processed to obtain the first red parameter corresponding to the red channel, the first green parameter of the green channel, and the first blue parameter corresponding to the blue channel in each third image region.

Step S460: Determine a first color parameter corresponding to each of the to-be-processed images based on the first red parameter, the first green parameter, and the first blue parameter.

After obtaining the first red parameter, the first green parameter, and the first blue parameter, the first color parameter corresponding to each of the to-be-processed images may be determined based on the first red parameter, the first green parameter, and the first blue parameter. For some implementations, the average value of the first red parameter, the first green parameter, and the first blue parameter may be used as the first color parameter corresponding to each of the to-be-processed images. Exemplarily, the first color parameter may be characterized by F_mean.

Step S470: Obtain a second red parameter corresponding to the red channel, a second green parameter corresponding to the green channel, and a second blue parameter corresponding to the blue channel in the initial image.

The method for determining the second color parameter corresponding to the initial image may specifically be to first obtain the second red parameter corresponding to the red channel, the second green parameter of the green channel, and the second blue parameter corresponding to the blue channel in the initial image. The second red parameter may represent the color average of the red channel in the initial image, that is, the average value of the red channel of each pixel in the initial image. Similarly, the second green parameter may represent the color average of the green channel in the initial image, that is, the average value of the green channel of each pixel in the initial image. The second blue parameter may represent the color average of the blue channel in the initial image, that is, the average value of the blue channel of each pixel in the initial image.

It should be noted that determining the second red parameter corresponding to the red channel, the second green parameter of the green channel, and the second blue parameter corresponding to the blue channel in the initial image may be determining the second red parameter, the second green parameter, and the second blue parameter corresponding to the entire image area in the initial image.

Step S480: Determine the second color parameter based on a second red parameter, a second green parameter and a second blue parameter.

After obtaining the second red parameter, the second green parameter, and the second blue parameter, the second color parameter corresponding to the initial image can be determined based on the second red parameter, the second green parameter, and the second blue parameter. For some embodiments, the average value of the second red parameter, the second green parameter, and the second blue parameter can be used as the second color parameter corresponding to the initial image. Exemplarily, the second color parameter can be characterized by A_mean.

Step S490: determining a target area corresponding to the target object in the initial image based on the mask image as a fourth image area.

In order to relight the target object in the image to be processed, the target object in each image to be processed is the target object in the initial image, so that the target area corresponding to the target object in the initial image can be determined.

Specifically, a target area corresponding to the target object in the initial image may be determined based on the mask image as the fourth image area.

Step S4100: The fourth image area that has been re-lit based on specified color parameters is merged with the third image area to obtain a target image, wherein the specified color parameters are parameters determined based on the quotient of the first color parameter and the second color parameter, and the third image area is an area in the image to be processed determined based on the mask image excluding the target area corresponding to the target object.

After obtaining the first color parameter and the second color parameter, the fourth image area after relighting based on the specified color parameter can be fused with the third image area to obtain a target image. The specified color parameter is a parameter determined based on the quotient of the first color parameter and the second color parameter, and the third image area is an area other than the target area corresponding to the target object in the image to be processed determined based on the mask image.

Exemplarily, the specific method of obtaining the target image can be described by the following formula (2):

G＝A*C*F_mean/A_mean+F*(1-C) (2)

Among them, G is used to represent the target image; A is used to represent the initial image; C is used to represent the mask image; F is used to represent the image to be processed; F*(1-C) is equivalent to directly multiplexing the fourth area, that is, not relighting the fourth area; F_mean is used to represent the first color parameter, A_mean is used to represent the second color parameter; then F_mean/A_mean is used to represent the specified color parameter; A*C*F_mean/A_mean represents relighting the third area. And A*C*F_mean/A_mean+F*(1-C) represents the fusion of the fourth image area relighted based on the specified color parameter with the third image area to obtain the target image G.

Step S4110: Generate a target video based on the target image.

Among them, step S4110 has been introduced in detail in the above embodiment and will not be repeated here.

The video generation method provided in the embodiment of the present application realizes relighting of the target object in the fourth area based on the third area in the acquired target image. The target area where the target object is located and the area other than the target area where the target object is located have good consistency in lighting and shadow, and there will be no unnatural defects in the fusion edge of the two areas.

Please refer to Figure 13, which shows a method flow chart of a video generation method provided by an embodiment of the present application. The video generation method can be applied to the electronic device in the video generation scene shown in Figure 1, and specifically, the processor of the electronic device can be used as the execution subject of the video generation method. The video generation method can include steps S510 to S5250.

Step S510: Start.

Step S520: Obtain the video to be fused.

Step S530: decomposing the video to be fused into frames.

First, the video to be fused can be obtained, that is, the video containing the motion effect object. Then, the video to be fused is frame-decomposed. For example, if the video to be fused includes N frames of images, each frame of the image to be fused can be represented in turn by a_n, where n=1, 2, 3...N.

The specific method of obtaining the video to be fused and the method of performing frame decomposition can be found in the introduction of the aforementioned embodiment, which will not be repeated here.

Step S540: Obtain the line drawing to be fused.

Furthermore, the line drawing to be fused corresponding to each of the frame images to be fused may be obtained, thereby obtaining a plurality of line drawings.

Step S550: Check whether each line drawing to be merged has been traversed.

Each time, one line drawing to be fused may be processed until all the line drawings to be fused are traversed. If the traversal is not completed, the execution may jump to step S560; if the traversal is completed, the execution may jump to step S5240.

Step S560: Acquire an initial image.

Step S570: Acquire a mask image.

Step S580: Adjust to a specified resolution.

An initial image may be obtained, and then a mask image of the initial image may be obtained through a subject matting algorithm, wherein the mask image may be used to characterize a target object in the initial image. Then, the resolution of the mask image is adjusted so that the resolution of the mask image matches the resolution of the latent space.

Step S590: Autoencoder encoding.

Step S5100: Acquire potential feature image.

In some implementations, the initial image may be encoded by an autoencoder to obtain a latent feature image in a latent space.

Step S5110: Add Gaussian distribution noise.

Furthermore, Gaussian distribution noise can be added to the potential feature image to obtain a noise image. Specifically, T times of Gaussian distribution noise can be added to the noise image, wherein the forward step can be a Markov chain, so that each step of adding Gaussian distribution noise is only related to the previous step, so as to convert a picture into pure Gaussian noise, that is, to obtain a noise image.

Step S5120: Acquire a depth image.

A depth image of the initial image can be extracted through a deep learning model.

Step S5130: Obtain an initial line drawing.

The method for obtaining the initial line drawing of the initial image is similar to the method for obtaining the line drawing to be fused corresponding to the frame image to be fused in the aforementioned step. The initial line drawing of the initial image can also be obtained by a line drawing obtaining algorithm.

Step S5140: Obtain the line drawing to be processed.

In some implementations, the initial line drawing may be fused with each line drawing to be fused to obtain a plurality of line drawing images to be processed. The specific fusion method may refer to the description of the above embodiment, which will not be described in detail here.

Step S5150: Line drawing model.

Step S5160: Depth model.

The depth feature information of the depth image may be extracted through a depth model; and the line draft feature information of the line draft image to be processed may be extracted through a line draft model.

Step S5170: Obtain effect prompt words.

In addition to using depth feature information and line feature information to guide the specified algorithm to operate the intermediate image, the intermediate image can also be further adjusted in combination with effect prompt words. Among them, the effect prompt words can be used to adjust the light and shadow effects of the intermediate image. The specific acquisition method can refer to the description in the above embodiment, which will not be repeated here.

Step S5180: noise reduction and light and shadow effect adjustment.

Call the specified algorithm to perform noise reduction and light and shadow effect adjustment on the noisy image. The specified algorithm can adjust the light and shadow effect under the guidance of the effect prompt word, and perform noise reduction under the guidance of the depth feature information and line feature information.

Step S5190: Whether i-step noise reduction is completed.

Step S5200: Iterate the noise image to obtain an intermediate image.

The noise image can be denoised by specifying an algorithm, and it can be iterated i steps. That is, the noise image can be iterated first to obtain an intermediate image. Then, denoising is performed to determine whether the i-step denoising has been completed. If not, the process jumps to continue executing S5200; if the i-step denoising has been completed, the process jumps to execute step S5210.

Optionally, the light and shadow effects may be adjusted after obtaining the intermediate image in each iteration.

Step S5210: Decode the image to be processed from the encoder.

The image obtained by the specified algorithm is decoded. Specifically, the image obtained by the specified algorithm can be decoded by an autoencoder to obtain an image to be processed.

Step S5220: Obtain a first color parameter and a second color parameter.

Step S5230: Re-light to obtain the target image.

The first color parameter is used to characterize the color parameter corresponding to the area other than the target area of the target object in the image to be processed. The second color parameter is used to characterize the color parameter corresponding to the initial image. The specified color parameter can be determined based on the first color parameter and the second color parameter, so that the target object in the image to be processed is re-lit by the specified color parameter. For a detailed description, please refer to the aforementioned embodiment, which will not be repeated here.

The target image corresponding to the line drawing to be fused in this processing process is obtained, and then the process returns to step S550 for determination.

Step S5240: Generate target video.

When it is determined that each line drawing to be fused has been traversed, a target video can be generated based on multiple target images. For the specific introduction of generating the target video, please refer to the introduction of the above embodiment, which will not be repeated here.

Step S5250: End.

Please refer to Figure 14, which shows a structural block diagram of a video generating device provided in an embodiment of the present application. The video generating device 1400 includes: a first acquisition unit 1410, a second acquisition unit 1420, a fusion unit 1430, a color parameter determination unit 1440, a re-lighting unit 1450 and a video generating unit 1460.

The first acquisition unit 1410 is used to acquire the video to be fused and the initial image, and perform frame decomposition on the video to be fused to obtain a plurality of frame images to be fused.

The second acquisition unit 1420 is configured to acquire a mask image of the target object in the initial image.

Optionally, the second acquisition unit 1420 can also be used to determine an initial mask image of the target object from the initial image based on a pre-acquired subject cutout model; adjust the resolution of the initial mask image to a specified resolution to obtain the mask image, and the specified resolution is the resolution of the latent space corresponding to the initial image.

The fusion unit 1430 is used to fuse the initial image with each of the frame images to be fused based on a specified algorithm to obtain a plurality of images to be processed.

Optionally, the fusion unit 1430 can also be used to obtain a line draft image to be fused corresponding to each of the frame images to be fused; obtain a depth image of the initial image and an initial line draft image of the initial image; fuse the initial line draft image with each of the line draft images to be fused respectively to obtain a plurality of line draft images to be processed; based on a specified algorithm, obtain an image to be processed corresponding to each of the line draft images to be processed through the depth image, the line draft image to be processed and the noise image, wherein the noise image is an image obtained by adding noise to the initial image.

Optionally, the fusion unit 1430 can also be used to encode the initial image based on an autoencoder to obtain a potential feature image of the initial image in a latent space; and add Gaussian distribution noise to the potential feature image to obtain the noise image.

Optionally, the fusion unit 1430 can also be used to obtain, based on a specified algorithm, an image to be decoded corresponding to each of the line draft images to be processed through the depth image, the line draft image to be processed and the noise image; and decode each of the image to be decoded based on the autoencoder to obtain an image to be processed corresponding to each of the line draft images to be processed.

Optionally, the fusion unit 1430 can also be used to extract depth feature information of the depth image based on the depth model; extract line draft feature information of the line draft image to be processed based on the line draft model; add Gaussian noise to the target object in the noise image to obtain an intermediate image; perform denoising on the intermediate image based on the depth feature information and the line draft feature information based on a specified algorithm to obtain an image to be processed corresponding to each of the line draft images to be processed.

Optionally, the fusion unit 1430 can also be used to determine a target area corresponding to a target object in the noise image based on the mask image as a first image area; determine an area other than the first image area in the noise image based on the mask image as a second image area; add Gaussian noise to the first image area to obtain a noise image area; and fuse the noise image area with the second image area to obtain the intermediate image.

Optionally, the fusion unit 1430 can also be used to obtain effect prompt words, and the effect prompt words are used to adjust the light and shadow effects of the intermediate image; based on a specified algorithm, a target operation is performed on the intermediate image through the depth feature information, the line draft feature information and the effect prompt words to obtain a to-be-processed image corresponding to each of the to-be-processed line draft images, and the target operation includes noise reduction processing and light and shadow effect adjustment.

The color parameter determination unit 1440 is configured to determine a first color parameter corresponding to each of the to-be-processed images based on the mask image, and to determine a second color parameter based on the initial image.

Optionally, the color parameter determination unit 1440 can also be used to determine an area outside the target area corresponding to the target object in the image to be processed based on the mask image as a third image area; obtain a first red parameter corresponding to the red channel, a first green parameter of the green channel, and a first blue parameter corresponding to the blue channel in each third image area; determine the first color parameter corresponding to each of the images to be processed based on the first red parameter, the first green parameter, and the first blue parameter; obtain a second red parameter corresponding to the red channel, the second green parameter of the green channel, and the second blue parameter corresponding to the blue channel in the initial image; determine the second color parameter based on the second red parameter, the second green parameter, and the second blue parameter.

Optionally, the color parameter determination unit 1440 can also be used to use the average value of the first red parameter, the first green parameter and the first blue parameter as the first color parameter corresponding to each of the images to be processed; and use the average value of the second red parameter, the second green parameter and the second blue parameter as the second color parameter.

The relighting unit 1450 is used to relight each of the images to be processed based on the second color parameter, the mask image, the initial image and the first color parameter corresponding to each of the images to be processed to obtain a target image.

Optionally, the relighting unit 1450 can also be used to determine the target area corresponding to the target object in the initial image based on the mask image as a fourth image area; the fourth image area after relighting based on specified color parameters is merged with the third image area to obtain a target image, the specified color parameter is a parameter determined based on the quotient of the first color parameter and the second color parameter, and the third image area is an area in the image to be processed determined based on the mask image except the target area corresponding to the target object.

The video generating unit 1460 is configured to generate a target video based on the target image.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

In several embodiments provided in the present application, the coupling between the units may be electrical, mechanical or other forms of coupling. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

Please refer to FIG. 15, which shows a block diagram of an electronic device provided in an embodiment of the present application. The electronic device 110 may be a smart phone, a desktop computer, a vehicle-mounted computer, a server, or a tablet computer, etc. The electronic device 110 in the present application may include one or more of the following components: a processor 111, a memory 112, and one or more application programs, wherein the processor 111 is electrically connected to the memory 112, and the one or more program configurations are used to execute the methods described in the aforementioned embodiments.

The processor 111 may include one or more processing cores. The processor 111 uses various interfaces and lines to connect various parts of the entire electronic device 110, and executes various functions and processes data of the electronic device 110 by running or executing instructions, programs, code sets or instruction sets stored in the memory 112, and calling data stored in the memory 112. Optionally, the processor 111 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 111 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and computer programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 111, and it can be implemented separately through a communication chip. Specifically, the method described in the above embodiment can be executed by one or more processors 111.

For some embodiments, the memory 112 may include a random access memory (RAM) or a read-only memory (ROM). The memory 112 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 112 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the following various method embodiments, etc. The data storage area may also store data created by the electronic device 110 during use, etc.

Please refer to Figure 16, which shows a block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 1600 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

The computer readable storage medium 1600 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 1600 includes a non-transitory computer-readable storage medium. The computer readable storage medium 1600 has storage space for program code 1610 that performs any method step of the above method. These program codes can be read from or written to one or more computer program products. The program code 1610 can be compressed, for example, in an appropriate form.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims (14)

1. A video generation method, comprising: Acquire the video to be fused and the initial image, and perform frame decomposition on the video to be fused to obtain a plurality of frame images to be fused; Acquire a mask image of the target object in the initial image; Based on a specified algorithm, the initial image is respectively fused with each of the frame images to be fused to obtain a plurality of images to be processed; Determine a first color parameter corresponding to each of the to-be-processed images based on the mask image, and determine a second color parameter based on the initial image; Relighting each of the images to be processed based on the second color parameter, the mask image, the initial image, and the first color parameter corresponding to each of the images to be processed to obtain a target image; A target video is generated based on the target image. 2. The method according to claim 1, characterized in that the fusing of the initial image with each of the frame images to be fused based on a specified algorithm to obtain a plurality of images to be processed comprises: Obtaining the line drawing to be fused corresponding to each of the frame images to be fused; Acquire a depth image of the initial image and an initial line drawing of the initial image; The initial line drawing is merged with each line drawing to be merged respectively to obtain a plurality of line drawing images to be processed; Based on a specified algorithm, an image to be processed corresponding to each of the line draft images to be processed is obtained through the depth image, the line draft image to be processed and the noise image, wherein the noise image is an image obtained by adding noise to the initial image. 3. The method according to claim 2, characterized in that before obtaining the image to be processed corresponding to each of the line draft images to be processed through the depth image, the line draft image to be processed and the noise image based on the specified algorithm, it also includes: Encoding the initial image based on an autoencoder to obtain a latent feature image of the initial image in a latent space; Gaussian distribution noise is added to the potential feature image to obtain the noise image. 4. The method according to claim 3, characterized in that the step of obtaining the image to be processed corresponding to each of the line draft images to be processed by using the depth image, the line draft image to be processed and the noise image based on a specified algorithm comprises: Based on a specified algorithm, the depth image, the line draft image to be processed and the noise image are used to obtain an image to be decoded corresponding to each of the line draft images to be processed; Each of the to-be-decoded images is decoded based on the autoencoder to obtain an image to be processed corresponding to each of the to-be-processed line draft images. 5. The method according to claim 2, characterized in that the step of obtaining the image to be processed corresponding to each of the line draft images to be processed by using the depth image, the line draft image to be processed and the noise image based on a specified algorithm comprises: Extracting depth feature information of the depth image based on a depth model; Extracting line draft feature information of the line draft image to be processed based on the line draft model; Adding Gaussian noise to the target object in the noise image to obtain an intermediate image; Based on a specified algorithm, the intermediate image is subjected to noise reduction processing using the depth feature information and the line draft feature information to obtain an image to be processed corresponding to each of the line draft images to be processed. 6. The method according to claim 5, characterized in that the step of adding Gaussian noise to the target object in the noise image to obtain an intermediate image comprises: determining, based on the mask image, a target area corresponding to the target object in the noise image as a first image area; determining, based on the mask image, an area other than the first image area in the noise image as a second image area; Adding Gaussian noise to the first image region to obtain a noisy image region; The noise image region is fused with the second image region to obtain the intermediate image. 7. The method according to claim 5, characterized in that the step of performing noise reduction processing on the intermediate image based on the specified algorithm by using the depth feature information and the line feature information to obtain the image to be processed corresponding to each of the line images to be processed comprises: Acquire an effect prompt word, where the effect prompt word is used to adjust the light and shadow effect of the intermediate image; Based on a specified algorithm, a target operation is performed on the intermediate image through the depth feature information, the line draft feature information and the effect prompt word to obtain a to-be-processed image corresponding to each to-be-processed line draft image, wherein the target operation includes noise reduction processing and light and shadow effect adjustment. 8. The method according to claim 1, wherein obtaining a mask image of the target object in the initial image comprises: Determining an initial mask image of the target object from the initial image based on a pre-acquired subject matting model; The resolution of the initial mask image is adjusted to a specified resolution to obtain the mask image, where the specified resolution is a resolution of a latent space corresponding to the initial image. 9. The method according to claim 1, characterized in that the determining the first color parameter corresponding to each of the to-be-processed images based on the mask image and the determining the second color parameter based on the initial image comprises: Determine, based on the mask image, an area outside the target area corresponding to the target object in the image to be processed as a third image area; Obtaining a first red parameter corresponding to the red channel, a first green parameter corresponding to the green channel, and a first blue parameter corresponding to the blue channel in each third image region; Determine a first color parameter corresponding to each of the to-be-processed images based on a first red parameter, a first green parameter, and a first blue parameter; Obtain a second red parameter corresponding to the red channel, a second green parameter corresponding to the green channel, and a second blue parameter corresponding to the blue channel in the initial image; The second color parameter is determined based on a second red parameter, a second green parameter, and a second blue parameter. 10. The method according to claim 9, characterized in that the determining the first color parameter corresponding to each of the to-be-processed images based on the first red parameter, the first green parameter and the first blue parameter comprises: Taking an average value of the first red parameter, the first green parameter and the first blue parameter as the first color parameter corresponding to each of the to-be-processed images; The determining the second color parameter based on a second red parameter, a second green parameter, and a second blue parameter comprises: An average value of the second red parameter, the second green parameter and the second blue parameter is used as the second color parameter. 11. The method according to claim 1, characterized in that the step of relighting each of the images to be processed based on the second color parameter, the mask image, the initial image, and the first color parameter corresponding to each of the images to be processed to obtain a target image comprises: determining, based on the mask image, a target area corresponding to the target object in the initial image as a fourth image area; The fourth image area that has been re-lit based on specified color parameters is fused with the third image area to obtain a target image, wherein the specified color parameters are parameters determined based on the quotient of the first color parameter and the second color parameter, and the third image area is an area in the image to be processed determined based on the mask image excluding the target area corresponding to the target object. 12. A video generation device, comprising: A first acquisition unit is used to acquire the video to be fused and the initial image, and perform frame decomposition on the video to be fused to obtain a plurality of frame images to be fused; A second acquisition unit, used to acquire a mask image of the target object in the initial image; A fusion unit, used for fusing the initial image with each of the frame images to be fused based on a specified algorithm to obtain a plurality of images to be processed; A color parameter determination unit, configured to determine a first color parameter corresponding to each of the to-be-processed images based on the mask image, and to determine a second color parameter based on the initial image; a relighting unit, configured to relight each of the images to be processed based on the second color parameter, the mask image, the initial image, and the first color parameter corresponding to each of the images to be processed, so as to obtain a target image; A video generating unit is used to generate a target video based on the target image. 13. An electronic device, comprising: one or more processors; Memory; One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1-11. 14. A computer-readable storage medium, characterized in that program code is stored in the computer-readable storage medium, and the program code can be called by a processor to execute the method according to any one of claims 1 to 11.