CN115690637A - Dynamic video generation method, device, apparatus, medium, and program product - Google Patents

Abstract

The present disclosure provides a dynamic video generation method, device, equipment, medium, and program product, wherein the method includes: acquiring a target image; performing feature extraction on the target image to determine initial image features corresponding to the target image; , respectively determine that the initial image features correspond to the time series image features of each time slice in multiple consecutive time slices; based on the time series image features corresponding to each time slice, the dynamic video corresponding to the target image is generated; the dynamic object in the target image is in There is a change in position in the motion video. The embodiments of the present disclosure can efficiently generate a dynamic video corresponding to a target image.

Description

Dynamic video generation method, device, device, medium and program product

technical field

The present disclosure relates to the technical fields of image processing and video generation, and in particular, to a dynamic video generation method, device, device, medium and program product.

Background technique

In the prior art, most of the methods of generating dynamic videos based on the acquired static target pictures are to determine the objects that can be dynamically displayed in the target pictures, and then use specific image editing software, editing software, etc. to manually edit the selected objects. and performing special effect processing on the target image to obtain multiple images corresponding to the target picture, and then, based on the target image and the obtained multiple images, a dynamic video corresponding to the target picture can be generated.

However, the display effect of the dynamic video obtained by the above method is poor, and due to the complicated operation steps, the efficiency of the manual processing method is greatly low, which in turn affects the efficiency of generating the dynamic video.

Contents of the invention

Embodiments of the present disclosure at least provide a dynamic video generation method, device, device, medium and program product.

In the first aspect, the embodiment of the present disclosure provides a dynamic video generation method applied to a trained target neural network, including:

Get the target image;

performing feature extraction on the target image, and determining initial image features corresponding to the target image;

Based on the initial image features, respectively determine that the initial image features correspond to time-series image features of each time slice in a plurality of consecutive time slices;

Based on the time-series image features corresponding to each time slice, a dynamic video corresponding to the target image is generated; a position of a dynamic object in the target image in the dynamic video varies. .

In a possible implementation manner, the multiple consecutive time slices include multiple time slice groups, each of the time slice groups includes a first time slice, and adjacent to the first time slice, located in a second time slice after the first time slice;

The determining, based on the initial image features, respectively that the initial image features correspond to the temporal image features of each time slice in a plurality of consecutive time slices, includes:

For each time slice group in the plurality of time slice groups, using a preset time series feature extraction network, the time series image features corresponding to the first time slice in the time slice group and the initial image features are processed Feature fusion processing to obtain the time-series image features corresponding to the second time slice in the time slice group;

Wherein, when the first time slice is the earliest time slice among the plurality of consecutive time slices, the initial image features are processed using a preset temporal feature extraction network to obtain the first time slice corresponding to time-series image features.

In a possible implementation manner, the performing feature extraction on the target image and determining the initial image features corresponding to the target image include:

Use the image encoder in the target neural network to perform feature extraction on the target image, and determine the first image feature of the target image extracted under the target first convolutional layer as the initial image feature; wherein, the The image encoder includes a plurality of first convolutional layers, and the target first convolutional layer is the first convolutional layer with the smallest number of corresponding convolution kernels among the plurality of first convolutional layers.

In a possible implementation manner, the generating the dynamic video corresponding to the target image based on the time-series image features corresponding to each time slice includes:

Based on the time-series image features corresponding to each time slice, generate derivative images corresponding to each time slice respectively; wherein, the positions of dynamic objects in different derivative images are different;

A dynamic video corresponding to the target image is generated based on the generated derivative images and time slices respectively corresponding to the derivative images.

In a possible implementation manner, based on a time-series image feature corresponding to a time slice, generating a derivative image corresponding to the time slice includes:

Utilize the image decoder in the target neural network to decode the time-series image features corresponding to the time slices, and determine the second image features decoded under the target second convolutional layer as target image features; wherein, The image decoder includes a plurality of second convolutional layers, and the target second convolutional layer is the second convolutional layer with the largest number of corresponding convolution kernels among the plurality of second convolutional layers;

Based on the features of the target image, a derivative image corresponding to the time slice is generated.

In a possible implementation manner, the method also includes the step of training the target neural network:

Obtain a sample dynamic video; the sample video includes multiple frames of sample images;

Inputting one frame sample image in the multi-frame sample image into the target neural network to be trained, processing the frame sample image by using the target neural network to be trained, and determining the predicted dynamic video corresponding to the frame sample image ;

Determine the prediction loss corresponding to the target neural network to be trained based on each frame of the predicted image in the predicted dynamic video and the multi-frame sample images included in the sample video;

The target neural network to be trained is iteratively trained by using the prediction loss until the training cut-off condition is met, and the trained target neural network is obtained.

In the second aspect, the embodiment of the present disclosure also provides a dynamic video generation device, which is applied to a trained target neural network, including:

An acquisition module, configured to acquire a target image;

A first determining module, configured to extract features of the target image, and determine initial image features corresponding to the target image;

The second determination module is configured to determine respectively, based on the initial image features, the time-series image features corresponding to the initial image features corresponding to each time slice in a plurality of consecutive time slices;

The generating module is configured to generate a dynamic video corresponding to the target image based on the time-series image features corresponding to each time slice; the position of the dynamic object in the target image in the dynamic video changes.

In a possible implementation manner, the multiple consecutive time slices include multiple time slice groups, each of the time slice groups includes a first time slice, and adjacent to the first time slice, located in a second time slice after the first time slice;

The second determination module is configured to, for each time slice group in the plurality of time slice groups, use a preset timing feature extraction network to determine the time sequence corresponding to the first time slice in the time slice group performing feature fusion processing on the image features and the initial image features to obtain time-series image features corresponding to the second time slice in the time slice group;

Wherein, when the first time slice is the earliest time slice among the plurality of consecutive time slices, the initial image features are processed using a preset temporal feature extraction network to obtain the first time slice corresponding to time-series image features.

In a possible implementation manner, the first determining module is configured to use the image encoder in the target neural network to perform feature extraction on the target image, and determine the The first image feature of the target image is used as the initial image feature; wherein, the image encoder includes a plurality of first convolutional layers, and the target first convolutional layer is one of the plurality of first convolutional layers Corresponding to the first convolutional layer with the smallest number of convolutional kernels.

In a possible implementation manner, the generation module is configured to generate a derivative image corresponding to each time slice based on the time-series image features corresponding to each time slice; The location is different;

A dynamic video corresponding to the target image is generated based on the generated derivative images and time slices respectively corresponding to the derivative images.

In a possible implementation manner, the generation module is configured to use the image decoder in the target neural network to decode the time-series image features corresponding to the time slice, and determine the The second image feature obtained by decoding is used as the target image feature; wherein, the image decoder includes a plurality of second convolutional layers, and the target second convolutional layer is a corresponding one of the plurality of second convolutional layers The second convolution layer with the largest number of convolution kernels;

Based on the features of the target image, a derivative image corresponding to the time slice is generated.

In a possible implementation manner, the device also includes a training module:

The training module is used to train the target neural network according to the following steps:

Obtain a sample dynamic video; the sample video includes multiple frames of sample images;

Inputting one frame sample image in the multi-frame sample image into the target neural network to be trained, processing the frame sample image by using the target neural network to be trained, and determining the predicted dynamic video corresponding to the frame sample image ;

Determine the prediction loss corresponding to the target neural network to be trained based on each frame of the predicted image in the predicted dynamic video and the multi-frame sample images included in the sample video;

The target neural network to be trained is iteratively trained by using the prediction loss until the training cut-off condition is met, and the trained target neural network is obtained.

In a third aspect, an optional implementation manner of the present disclosure further provides a computer device, a processor, and a memory, the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the instructions stored in the memory. machine-readable instructions, when the machine-readable instructions are executed by the processor, when the machine-readable instructions are executed by the processor, the above-mentioned first aspect is executed, or any possible implementation of the first aspect steps in the method.

In a fourth aspect, an optional implementation manner of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed, the above-mentioned first aspect, or any one of the first aspects in the first aspect, may be executed. Steps in one possible implementation.

In the fifth aspect, an optional implementation manner of the present disclosure further provides a computer program product, including computer instructions, which execute the above-mentioned first aspect when the computer instructions are executed by a processor, or any of the possible implementation manners of the first aspect. step.

For the effect description of the above-mentioned dynamic video generation device, computer equipment, computer-readable storage medium and computer program product, please refer to the description of the above-mentioned dynamic video generation method, which will not be repeated here.

The dynamic video generation method, device, equipment, medium, and program product provided by the embodiments of the present disclosure use the trained target neural network to extract the features of the target image, and first can accurately extract the initial image that can reflect the image features of the target image feature, the image features corresponding to the dynamic objects in the target image in different time slices have temporal correlations, therefore, using the target neural network to perform feature processing on the determined initial image features, it is possible to accurately determine the initial image feature correspondence Based on the temporal image features of each time slice in multiple consecutive time slices, that is, the image features of dynamic objects corresponding to different time slices can be obtained. Furthermore, by using the time-series image features, images in which dynamic objects in the target image correspond to different time slices can be accurately obtained, thereby obtaining dynamic videos. By using the trained target neural network to process the target image, the dynamic video can be automatically generated. Compared with the manual processing method, the generation efficiency of the dynamic video is greatly improved, and the trained target neural network has undergone multiple rounds The training has a reliable output accuracy, which improves the accuracy of the obtained dynamic video and further improves the display effect of the dynamic video.

In order to make the above-mentioned objects, features and advantages of the present disclosure more comprehensible, preferred embodiments will be described in detail below together with the accompanying drawings.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present disclosure more clearly, the following will briefly introduce the accompanying drawings used in the embodiments. The accompanying drawings here are incorporated into the specification and constitute a part of the specification. The drawings show the embodiments consistent with the present disclosure, and are used together with the description to explain the technical solutions of the present disclosure. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be regarded as limiting the scope. For those skilled in the art, they can also make From these drawings other related drawings are obtained.

FIG. 1 shows a flowchart of a dynamic video generation method applied to a trained target neural network provided by an embodiment of the present disclosure;

FIG. 2 shows a schematic diagram of the architecture of an LSTM network provided by an embodiment of the present disclosure;

FIG. 3 shows a schematic diagram of generating a dynamic video corresponding to a target image provided by an embodiment of the present disclosure;

FIG. 4 shows a schematic diagram of using a target neural network to generate a dynamic video corresponding to a target image provided by an embodiment of the present disclosure;

Fig. 5 shows a schematic diagram of a dynamic video generation device provided by an embodiment of the present disclosure;

Fig. 6 shows a schematic structural diagram of a computer device provided by an embodiment of the present disclosure.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only It is a part of the embodiments of the present disclosure, but not all of them. The components of the disclosed embodiments generally described and illustrated herein may be arranged and designed in a variety of different configurations. Accordingly, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort shall fall within the protection scope of the present disclosure.

In addition, the terms "first", "second" and the like in the description and claims in the embodiments of the present disclosure and the above drawings are used to distinguish similar objects, and not necessarily used to describe a specific order or sequence. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments described herein can be practiced in sequences other than those illustrated or described herein.

"Plural or several" mentioned herein means two or more. "And/or" describes the association relationship of associated objects, indicating that there may be three types of relationships, for example, A and/or B may indicate: A exists alone, A and B exist simultaneously, and B exists independently. The character "/" generally indicates that the contextual objects are an "or" relationship.

After research, it is found that in the prior art, most of the methods of generating dynamic videos based on the acquired static target pictures are to determine the objects that can be dynamically displayed in the target pictures, and then use specific picture editing software, editing software, etc. to manually edit the video. The selected object and the target image are processed with special effects to obtain multiple images corresponding to the target picture, and then, based on the target image and the obtained multiple images, a dynamic video corresponding to the target picture can be generated.

However, the display effect of the dynamic video obtained by the above method is poor, and due to the complicated operation steps, the efficiency of the manual processing method is greatly low, which in turn affects the efficiency of generating the dynamic video.

Based on the above research, the present disclosure provides a dynamic video generation method, device, equipment, medium and program product, which uses the trained target neural network to extract features of the target image, and firstly can accurately extract an image that can reflect the target image The initial image features of the features, and the image features corresponding to the dynamic objects in different time slices in the target image have temporal correlations. Therefore, the target neural network is used to perform feature processing on the determined initial image features to accurately determine The original image feature corresponds to the sequential image feature of each time slice in multiple consecutive time slices, that is, the image features of the dynamic object corresponding to different time slices can be obtained. Furthermore, by using the time-series image features, images in which dynamic objects in the target image correspond to different time slices can be accurately obtained, thereby obtaining dynamic videos. By using the trained target neural network to process the target image, the dynamic video can be automatically generated. Compared with the manual processing method, the generation efficiency of the dynamic video is greatly improved, and the trained target neural network has undergone multiple rounds The training has a reliable output accuracy, which improves the accuracy of the obtained dynamic video and further improves the display effect of the dynamic video.

The defects in the above solutions are all the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure below for the above problems should be the result of the inventor Contributions made to this disclosure during the course of this disclosure.

It should be noted that like numerals and letters denote similar items in the following figures, therefore, once an item is defined in one figure, it does not require further definition and explanation in subsequent figures.

It should be noted that the specific nouns mentioned in the embodiments of the present disclosure include:

LSTM, Long Short Term Memory, long short-term memory network, is a neural network with the ability to memorize long-term and short-term information.

Autoencoding neural network: an autoencoding machine learning model with powerful feature extraction and representation capabilities, including data compression algorithms, where the data compression and decompression functions are data-dependent, lossy, and automatically learned from samples of.

In order to facilitate the understanding of this embodiment, a dynamic video generation method disclosed in the embodiment of the present disclosure is first introduced in detail. The execution subject of the dynamic video generation method provided in the embodiment of the present disclosure is generally a computer device with certain computing capabilities. In some possible implementation manners, the method for generating a dynamic video may be implemented by a processor invoking computer-readable instructions stored in a memory.

The dynamic video generation method provided by the embodiments of the present disclosure will be described below by taking the trained target neural network as an example.

As shown in FIG. 1, a flow chart of a dynamic video generation method applied to a trained target neural network provided by an embodiment of the present disclosure may include the following steps:

S101: Acquire a target image.

Here, the target image is a still image that has been shot, and may contain at least one image of a dynamic object that can be dynamically processed, wherein the dynamic object that can be dynamically processed may be a static object in the target image, but in the real world Objects that can be moved in , such as sea water, hair, windmills, etc.

Exemplarily, the target image can be a target person image, a target animal image, a landscape image, etc., and the dynamic object that can be dynamically processed can be the hair of a person in the target person image, sea water in the image background, etc., and the animal in the target animal image Hair, etc., tree branches, clouds, etc. in landscape images.

During specific implementation, when the user needs to generate a dynamic video corresponding to the target image, the target image can be directly input into the pre-trained target neural network, so that the target neural network can obtain the target image to be processed.

In a specific application, the pre-trained target neural network can be deployed in various image processing APPs (Application, application programs), for example, beauty APP, or can also be deployed in various computer program products, for example, deployment In the online image processing platform. The embodiment of the present disclosure does not limit the specific existence form of the target neural network.

Taking the deployment of the target neural network in the Beauty APP as an example, when the user needs to generate dynamic videos, he can directly input the target image into the Beauty APP, so that the target neural network in the Beauty APP can obtain the data to be processed. target image.

S102: Perform feature extraction on the target image, and determine initial image features corresponding to the target image.

Here, the initial image features are used to characterize the image features corresponding to the target image. Specifically, the initial image features may include target image color features, texture features, shape features and spatial relationship features of the image, and the color features may be pixels in the target image The color value of the point.

In specific implementation, after the target neural network acquires the target image, it can use the convolutional layer and pooling layer in the network to perform multiple consecutive convolutions on the target image, and each convolution can obtain a feature corresponding to the target image Each convolution can extract feature maps with different sizes and data volumes. Furthermore, the feature map obtained with the smallest size and data volume can be used as the initial image feature corresponding to the target image, or the feature map obtained by the last convolution in consecutive multiple convolutions can be used as the target image Corresponding initial image features. Wherein, the initial image feature may be a feature vector, for example, may be a 1×1024 feature vector.

S103: Based on the initial image features, respectively determine that the initial image features correspond to time-series image features of each time slice in a plurality of consecutive time slices.

Here, the time-series image feature is an image feature including time-series information, that is, an image feature corresponding to an initial image feature under different time slices. Multiple consecutive time slices can be specific time slices preset in the target neural network. The time interval between the times corresponding to each two time slices can be the same or different, and can be set according to development needs. Time slices The number of can also be set according to development needs, which are not limited here. For any acquired initial image feature, according to the set multiple time slices, it may be determined that the initial image feature corresponds to time-series image features of each time slice.

The first time slice in the multiple consecutive time slices may be the time slice corresponding to the initial image feature. Or, the time slice corresponding to the initial image feature can be any one of a plurality of consecutive time slices, so that after the initial image feature is acquired, the temporal image feature of the time slice before the time slice corresponding to the initial image feature can be determined , and the temporal image features of the time slice after the time slice corresponding to the initial image feature.

During specific implementation, for each time slice, after the target neural network determines the initial image features, it can first analyze the initial image features to determine that each dynamic object in the target image corresponds to the object image features in the initial image features, and then The time slice corresponding to the initial image feature can be determined, and based on the time slice, each object image feature and/or initial image feature in the initial image feature corresponding to the time slice is performed in feature replication, feature fusion, and feature splicing. A feature processing operation to obtain the time-series image features corresponding to the dynamic object under the time slice, that is, to obtain the time-series image features corresponding to the initial image features under the time slice.

Furthermore, based on the above steps, it is possible to obtain time-series image features whose initial image features correspond to each time slice in a plurality of consecutive time slices.

In addition, the types of dynamic objects that can be recognized by the target neural network are determined by the training samples. For example, the types of dynamic objects that can be recognized by the target neural network include type1, type2, and type3, and the types of dynamic objects included in the target image are type1, type3, and type4, the final target neural network can recognize the types of dynamic objects in the target image as type1 and type3, and the objects that can change positions in the final output dynamic video are the dynamic objects corresponding to type1 and the dynamic objects corresponding to type3.

S104: Based on the time-series image features corresponding to each time slice, generate a dynamic video corresponding to the target image, where the position of the dynamic object in the target image changes in the dynamic video.

Here, the dynamic video includes multiple derivative images related to the target image and the target image. The objects included in each image are the same, but the positions of the dynamic objects included in each image are different. Therefore, the dynamic objects in the target image There will be variations in position in the resulting motion video.

Taking the target image as the full body image of character H, and the dynamic object as the long hair of character H as an example, the position of the long hair of character H in the generated dynamic video will change, and when the dynamic video is displayed, the long hair will appear swing effect.

During specific implementation, the target neural network can use the convolutional layer to deconvolute the time-series image features corresponding to each time slice to obtain the derivative image corresponding to each time-series image feature, and then determine the generated derivative image and the image sequence corresponding to the target image, and generate a dynamic video corresponding to the target image according to the image sequence.

In this way, using the trained target neural network to perform feature extraction on the target image, first, the initial image features that can reflect the image features of the target image can be accurately extracted, and the image features corresponding to the dynamic objects in the target image in different time slices There is a time-series correlation. Therefore, using the target neural network to perform feature processing on the determined initial image features, it can be accurately determined that the initial image features correspond to the time-series image features of each time slice in multiple consecutive time slices. That is, image features of dynamic objects corresponding to different time slices can be obtained. Furthermore, by using the time-series image features, images in which dynamic objects in the target image correspond to different time slices can be accurately obtained, thereby obtaining dynamic videos. By using the trained target neural network to process the target image, the dynamic video can be automatically generated. Compared with the manual processing method, the generation efficiency of the dynamic video is greatly improved, and the trained target neural network has undergone multiple rounds The training has a reliable output accuracy, which improves the accuracy of the obtained dynamic video and further improves the display effect of the dynamic video.

In one embodiment, multiple consecutive time slices include multiple time slice groups, each time slice group includes a first time slice, and adjacent to the first time slice, located after the first time slice the second time slice of .

Here, one time slice can correspond to multiple time slice groups, and each time slice in a plurality of consecutive time slices can be the first time slice in one time slice group, and can be the second time slice in another time slice group. time slice. Whether each time slice is the first time slice or the second time slice can be determined according to the relative relationship between another time slice in the time slice group where it is located and the time slice.

For example, time slice C is connected to time slice B and located after time slice B, time slice B is connected to time slice A and located after time slice A, and the time slice group includes the previous time slice group 1 and the next time slice group 2. Time slice group 1 includes A time slice and B time slice, time slice group 2 includes B time slice and C time slice, then B time slice is the second time slice in time slice group 1, and in time slice group 2 for the first time slice.

The multiple time slice groups may be multiple consecutive time slice groups, wherein the second time slice included in the previous time slice group may be the first time slice among the two time slices included in the subsequent time slice group. During specific implementation, the number of time slice groups can be determined by the number of time slices. For example, if the number of time slices is 5, the number of time slice groups can be 4; if the number of time slices is 15, the number of time slice groups Quantity can be 14.

For S103, it can be implemented according to the following steps:

For each time slice group in multiple time slice groups, use the preset time series feature extraction network to perform feature fusion processing on the time series image features and initial image features corresponding to the first time slice in the time slice group to obtain the time Time series image features corresponding to the second time slice in the slice group;

Wherein, when the first time slice is the earliest time slice among the plurality of consecutive time slices, the initial image features are processed using a preset time-series feature extraction network to obtain the time-series image features corresponding to the first time slice.

Here, the preset time series feature extraction network may be preset in the target neural network, specifically, the time series feature extraction network may be an LSTM network. As shown in FIG. 2 , it is a schematic structural diagram of an LSTM network provided by an embodiment of the present disclosure.

For each time slice group in multiple time slice groups, the temporal image features and initial image features corresponding to the first time slice in the time slice group can be directly input into the LSTM network, and then the LSTM network can be used to The time series image features and the initial image features are fused, and then the LSTM network can output the time series image features corresponding to the second time slice in the time slice group. In FIG. 2, h _t-1 can represent the time-series image feature corresponding to the first time slice in any time slice group, X _t represents the initial image feature, and h _t represents the time-series image feature corresponding to the second time slice.

Furthermore, the temporal image features corresponding to the second time slice in the next time slice group adjacent to the above time slice group can be determined, specifically, the first time sequence in the next time slice group adjacent to the above time slice group The time-series image feature corresponding to the slice can be the time-series image feature corresponding to the second time slice just determined in the above-mentioned time-slice group, and then, the LSTM network can use the time-series image feature and the initial image feature to determine the time-series image feature adjacent to the above-mentioned time-slice group Time-series image features corresponding to the second time slice in the next time slice group. In Fig. 2, h _t can represent the time-series image feature corresponding to the first time slice input when calculating the time-series image feature corresponding to the second time slice in the next time-slice group adjacent to the above-mentioned time-slice group, X _{t +1} represents the initial image feature, and h _t+1 represents the time-series image feature corresponding to the newly determined second time slice.

In this way, based on the architecture of the LSTM network shown in FIG. 2 , the temporal image features corresponding to the first time slice and the second time slice in each time slice group can be determined.

Among them, for the earliest time slice among multiple consecutive time slices, the initial image features can be directly input into the LSTM network, and then the LSTM network can directly process the initial image features to determine the temporal image features corresponding to the earliest time slice , and the first time slice in each time slice group must be an earlier time slice, so the earliest time slice must be the first time slice in its corresponding time slice group. Furthermore, based on the first time slice, the time-series image features corresponding to the second time slice in the corresponding time slice group may be determined.

In Figure 2, X _t-1 represents the initial image feature, and h _t-1 can represent the time-series image feature corresponding to the earliest time slice output when calculating the time-series image feature corresponding to the earliest time slice.

In a possible implementation manner, for S102, the image encoder in the target neural network can be used to perform feature extraction on the target image, and the first image feature of the target image extracted under the target first convolutional layer can be determined as the initial image Features; wherein the image encoder includes a plurality of first convolutional layers, and the target first convolutional layer is the first convolutional layer with the smallest number of corresponding convolution kernels among the plurality of first convolutional layers.

Here, the target neural network may also include an image encoder, which may be composed of an automatic encoding neural network. The image encoder includes a plurality of preset first convolutional layers, and each first convolutional layer The number of convolution kernels included is different, the size of the convolution kernel is also different, and the size of the convolution kernel in each first convolution layer is positively correlated with the number of convolution kernels in the first convolution layer, a The larger the number of convolution kernels in the first convolution layer, the larger the convolution kernels in the first convolution layer; otherwise, the smaller the number of convolution kernels, the smaller the convolution kernels.

In an implementation manner, one first convolutional layer may correspond to one convolutional neural network, and the image encoder includes multiple convolutional neural networks.

The target first convolutional layer is the first convolutional layer with the smallest number of corresponding convolution kernels and the smallest convolution kernel among the plurality of first convolutional layers.

Each first convolutional layer has a certain convolution order, wherein, after the image encoder acquires the target image, the first convolutional layer with the largest number of convolution kernels will perform a convolution operation on the image data corresponding to the target image, Extract the first image feature corresponding to the target image under the first convolutional layer, specifically, the first image feature can be a feature map, and then, the first convolutional layer with the second largest number of convolution kernels will Convolve the image data of the first image feature corresponding to the first convolution layer to obtain the first image feature corresponding to the first convolution layer with the second largest number of convolution kernels. After that, the number of convolution kernels is the third The first convolutional layer with the largest number of convolutional kernels will perform a convolution operation on the first image feature corresponding to the first convolutional layer with the second largest number of convolutional kernels to obtain the first convolutional layer with the third largest number of convolutional kernels The corresponding first image feature, and so on, until the first image feature corresponding to the first convolutional layer with the least number of convolution kernels is determined, and the first image feature is used as the target extracted under the target first convolutional layer The initial image features of the image. Here, the first image feature corresponding to the first convolution layer with the least number of convolution kernels may be a feature vector.

Or, in another embodiment, after obtaining the first image feature of the target image extracted under the target first convolutional layer, reshape for adjusting the data dimension can be performed on the image data corresponding to the first image feature The operation is to convert the image data corresponding to the first image feature into a feature vector, and then use the feature vector as an initial image feature.

In a possible embodiment, for S104, the following steps may be followed:

Step 1: Generate derivative images corresponding to each time slice based on the time-series image features corresponding to each time slice.

Wherein, the positions of the dynamic objects in different derived images are different.

Here, each time slice corresponds to a derived image, and the objects included in the derived image are the same as the target image, but the positions of the dynamic objects included in each derived image are different. In this way, based on multiple derived images, it is possible to generate Dynamic video of dynamic objects.

During specific implementation, after determining the time-series image features corresponding to each time slice, the target nerve can perform multiple deconvolution operations on the time-series image features for each time slice, and when the number of deconvolutions reaches the target number of times , and use the generated feature map as the derivative image corresponding to the time slice.

Furthermore, a derived image corresponding to each time slice can be obtained.

In another embodiment, for step 1, after determining the time-series image features corresponding to each time slice, the time-series image features corresponding to the time slices in the same time slice group can be further fused, using fusion The final time-series image features are used to generate derivative images corresponding to each time slice group.

Alternatively, the time-series image features corresponding to multiple time slices with adjacent relationships can be further fused to obtain the fused time-series image features, and the derived images corresponding to the time-series image features can be generated by using the fused time-series image features .

In this way, the number of generated derivative images may not match the number of time slices, and the specific number of generated derivative images may be set by the developer according to requirements, which is not limited here.

Step 2: Based on the generated derived images and the time slices corresponding to the derived images, a dynamic video corresponding to the target image is generated.

In this step, after each derivative image is determined, the order of time corresponding to each image can be determined based on the time sequence corresponding to the time slice corresponding to each derivative image and the time corresponding to the target image. Sequentially, a dynamic video corresponding to the target image is generated. Wherein, the time sequence of each image in the dynamic video is consistent with the determined sequence.

As shown in FIG. 3 , it is a schematic diagram of generating a dynamic video corresponding to a target image provided by an embodiment of the present disclosure.

In one embodiment, for the time-series image features corresponding to any time slice, the process of generating the derivative image corresponding to the time slice may include the following two steps:

Step S1: Use the image decoder in the target neural network to decode the time-series image features corresponding to the time slices, and determine the second image features decoded under the target second convolutional layer as the target image features.

Wherein, the image decoder includes multiple second convolutional layers, and the target second convolutional layer is the second convolutional layer with the largest number of corresponding convolution kernels among the multiple second convolutional layers.

Here, the target neural network may also include an image decoder, which may also be composed of an auto-encoding neural network. The image decoder includes a plurality of preset second convolutional layers, each second convolutional layer The number of convolution kernels included in is different, the size of the convolution kernel is also different, and the size of the convolution kernel in each second convolution layer is positively correlated with the number of convolution kernels in the second convolution layer, The larger the number of convolution kernels in a second convolution layer, the larger the convolution kernel in the second convolution layer; otherwise, the smaller the number of convolution kernels, the smaller the convolution kernel.

In addition, a second convolutional layer may also correspond to a convolutional neural network, which is a deconvolutional neural network.

The target second convolutional layer is the first convolutional layer with the largest number of corresponding convolution kernels and the largest number of convolution kernels among the plurality of second convolutional layers.

Each second convolutional layer has a certain convolution order. During specific implementation, after the LSTM network outputs the time-series image features corresponding to the time slice, the image decoder can obtain the time-series image features, and then the second convolutional layer with the least number of convolution kernels will perform the time-series image features Decoding processing, that is, performing a deconvolution operation to obtain a second image feature of the time-series image feature under the second convolution layer, specifically, the second image feature may also be a feature map.

Here, in another embodiment, in the case where the time-series image feature corresponding to the time slice output by the LSTM network is a time-series feature vector, the reshape operation for adjusting the data dimension can be performed on the time-series image feature first, and the time-series The feature vector is adjusted to the initial feature map, and then the image decoder can use the second convolution layer with the least number of convolution kernels to decode the initial feature map to obtain the second image feature.

Alternatively, the image decoder may also directly decode the time-series feature vector to obtain the second image feature, which will not be repeated here.

Then, the second convolution layer with the second least number of convolution kernels will perform deconvolution processing on the second image features output by the second convolution layer with the smallest number of convolution kernels to obtain Then, the second convolutional layer with the third least number of convolution kernels will deconvolute the second image features output by the second convolutional layer with the second smallest number of convolution kernels , to obtain the second image feature under the second convolution layer with the third least number of convolution kernels, and so on, until the second image feature corresponding to the second convolution layer with the largest number of convolution kernels is determined, and The second image feature is used as a target image feature decoded under the target second convolutional layer.

In this way, by using the image decoder, the target image feature corresponding to each time slice can be determined.

Step S2, based on the characteristics of the target image, generate a derivative image corresponding to the time slice.

Here, when the feature of the target image is a feature map, the feature of the target image can be directly used as the derived image corresponding to the time slice. Furthermore, a derived image corresponding to each time slice can be obtained.

As shown in FIG. 4 , it is a schematic diagram of using a target neural network to generate a dynamic video corresponding to a target image provided by an embodiment of the present disclosure. Among them, L1, L2, L3, L4, and LN respectively represent different time slices, and the time slice corresponding to L1 is earlier than the time slice corresponding to L2, the time slice corresponding to L2 is earlier than the time slice corresponding to L3, and the time slice corresponding to L3 is earlier than the time slice corresponding to L3. It is earlier than the time slice corresponding to L4, and the time slice corresponding to L4 is earlier than the time slice corresponding to LN.

In addition, since the dynamic video generation provided by the disclosed example is executed by using the trained target neural network, the embodiment of the present disclosure also provides a method for training the target neural network, which may include the following steps:

T1: Obtain sample dynamic video;

Wherein, the sample video includes multiple frames of sample images.

In this way, the sample dynamic video can be a short video including dynamic objects, the sample video includes multiple frames of sample images, and each frame of sample images includes the same dynamic objects as static objects, and the positions of the static objects included in each frame of sample images are the same, and the dynamic Objects are positioned differently.

T2: Input one frame of sample images among the multiple frames of sample images into the target neural network to be trained, process the frame of sample images by using the target neural network to be trained, and determine the predicted dynamic video corresponding to the frame of sample images.

Here, any frame of sample images in the multi-frame sample images can be selected as the input sample image, so that by selecting sample images with different time sequences to train the target neural network, the trained target neural network can be The image before the time slice corresponding to the input sample image can be predicted, and the image after the time slice corresponding to the input sample image can also be predicted, which improves the adaptability and flexibility of the trained target neural network.

During specific implementation, any selected frame sample image can be input into the target neural network to be trained, and the target neural network to be trained will process it and output = predicted dynamic video corresponding to the frame sample image.

T3: Determine the prediction loss corresponding to the target neural network to be trained based on each frame of the predicted image in the predicted dynamic video and the multi-frame sample images included in the sample video.

In this step, a sample image matching each frame of the predicted image in the predicted dynamic video can be determined from the multi-frame sample images, and then the sub-prediction loss between each frame of the predicted image and its matched sample image can be determined, And based on the sub-prediction loss corresponding to each frame of prediction image, the prediction loss corresponding to the target neural network to be trained is determined.

Specifically, based on each determined sub-prediction loss, the mean square error corresponding to the sub-prediction loss may be determined, and the determined mean square error may be used as the prediction loss.

T4: Use the prediction loss to perform iterative training on the target neural network to be trained until the training cut-off condition is met, and the trained target neural network is obtained.

Here, the training cut-off condition may be that the number of iteration training reaches the target number, or at least one of the prediction accuracy of the target neural network obtained from training meets the preset accuracy.

In specific implementation, the target neural network to be trained can be directly used for iterative training by using the prediction loss, and the trained target neural network can be used as the trained target neural network when it is determined that the training cut-off condition is met.

Those skilled in the art can understand that in the above method of specific implementation, the writing order of each step does not mean a strict execution order and constitutes any limitation on the implementation process. The specific execution order of each step should be based on its function and possible The inner logic is OK.

Based on the same inventive concept, the embodiments of the present disclosure also provide a dynamic video generation device corresponding to the dynamic video generation method. Since the problem-solving principle of the device in the embodiments of the present disclosure is similar to that of the above-mentioned dynamic video generation method in the embodiments of the present disclosure, therefore For the implementation of the device, reference may be made to the implementation of the method, and repeated descriptions will not be repeated.

As shown in Figure 5, it is a schematic diagram of a dynamic video generation device provided by an embodiment of the present disclosure, including:

An acquisition module 501, configured to acquire a target image;

The first determining module 502 is configured to perform feature extraction on the target image, and determine initial image features corresponding to the target image;

The second determination module 503 is configured to respectively determine, based on the initial image features, that the initial image features correspond to time-series image features of each time slice in a plurality of consecutive time slices;

The generation module 504 is configured to generate a dynamic video corresponding to the target image based on the time-series image features corresponding to each time slice; the position of the dynamic object in the target image in the dynamic video varies.

In a possible implementation manner, the multiple consecutive time slices include multiple time slice groups, each of the time slice groups includes a first time slice, and adjacent to the first time slice, located in a second time slice after the first time slice;

The second determination module 503 is configured to, for each time slice group in the plurality of time slice groups, use a preset timing feature extraction network to perform performing feature fusion processing on the time-series image features and the initial image features to obtain the time-series image features corresponding to the second time slice in the time-slice group;

Wherein, when the first time slice is the earliest time slice among the plurality of consecutive time slices, the initial image features are processed using a preset temporal feature extraction network to obtain the first time slice corresponding to time-series image features.

In a possible implementation manner, the first determination module 502 is configured to use the image encoder in the target neural network to perform feature extraction on the target image, and determine the feature extracted under the target first convolutional layer. The first image feature of the target image is used as the initial image feature; wherein, the image encoder includes a plurality of first convolutional layers, and the target first convolutional layer is the plurality of first convolutional layers The first convolutional layer corresponding to the smallest number of convolutional kernels.

In a possible implementation manner, the generation module 504 is configured to generate a derivative image corresponding to each time slice based on the time-series image features corresponding to each time slice; wherein, the dynamic objects in the different derivative images different locations;

A dynamic video corresponding to the target image is generated based on the generated derivative images and time slices respectively corresponding to the derivative images.

In a possible implementation manner, the generation module 504 is configured to use the image decoder in the target neural network to decode the time-series image features corresponding to the time slice, and determine The second image feature obtained by decoding under the layer is used as the target image feature; wherein, the image decoder includes a plurality of second convolutional layers, and the target second convolutional layer is corresponding to the plurality of second convolutional layers. The second convolutional layer with the largest number of convolution kernels;

Based on the features of the target image, a derivative image corresponding to the time slice is generated.

In a possible implementation manner, the device further includes a training module 505:

The training module 505 is used to train the target neural network according to the following steps:

Obtain a sample dynamic video; the sample video includes multiple frames of sample images;

Inputting one frame sample image in the multi-frame sample image into the target neural network to be trained, processing the frame sample image by using the target neural network to be trained, and determining the predicted dynamic video corresponding to the frame sample image ;

Determine the prediction loss corresponding to the target neural network to be trained based on each frame of the predicted image in the predicted dynamic video and the multi-frame sample images included in the sample video;

The target neural network to be trained is iteratively trained by using the prediction loss until the training cut-off condition is met, and the trained target neural network is obtained.

For the description of the processing flow of each module in the device and the interaction flow between the modules, reference may be made to the relevant description in the above method embodiment, and details will not be described here.

An embodiment of the present disclosure also provides a computer device, as shown in FIG. 6 , which is a schematic structural diagram of a computer device provided by an embodiment of the present disclosure, including:

Processor 61 and memory 62; the memory 62 stores machine-readable instructions executable by the processor 61, the processor 61 is used to execute the machine-readable instructions stored in the memory 62, and the machine-readable instructions are executed by the processor 61 During execution, the processor 61 performs the following steps: S101: acquire the target image; S102: perform feature extraction on the target image, and determine the initial image features corresponding to the target image; S103: determine the initial image features corresponding to multiple Time-series image features of each time slice in consecutive time slices and S104: Generate a dynamic video corresponding to the target image based on the time-series image features corresponding to each time slice; the position of the dynamic object in the target image changes in the dynamic video .

Above-mentioned memory 62 comprises memory 621 and external memory 622; Memory 621 here is also called internal memory, is used for temporarily storing the operation data in processor 61, and the data exchanged with external memory 622 such as hard disk, processor 61 communicates with memory 621 through memory 621. The external memory 622 performs data exchange.

For the specific execution process of the above instructions, reference may be made to the steps of the dynamic video generation method described in the embodiments of the present disclosure, which will not be repeated here.

Embodiments of the present disclosure also provide a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, the steps of the dynamic video generation method described in the foregoing method embodiments are executed. Wherein, the storage medium may be a volatile or non-volatile computer-readable storage medium.

The computer program product of the dynamic video generation method provided by the embodiments of the present disclosure includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the dynamic video generation method described in the above method embodiments For details, please refer to the above-mentioned method embodiments, and details are not repeated here.

The computer program product can be specifically realized by means of hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium. In another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), etc. wait.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the device described above can refer to the corresponding process in the foregoing method embodiment, and details are not repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods may be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined. Or some features can be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some communication interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.

If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium executable by a processor. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk, and other media that can store program codes.

Finally, it should be noted that: the above-mentioned embodiments are only specific implementations of the present disclosure, and are used to illustrate the technical solutions of the present disclosure, rather than limit them, and the protection scope of the present disclosure is not limited thereto, although referring to the aforementioned The embodiments have described the present disclosure in detail, and those skilled in the art should understand that any person familiar with the technical field can still modify the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present disclosure Changes can be easily imagined, or equivalent replacements can be made to some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in this disclosure. within the scope of protection. Therefore, the protection scope of the present disclosure should be defined by the protection scope of the claims.

Claims (10)

1. A dynamic video generation method is applied to a trained target neural network, and comprises the following steps:

acquiring a target image;

performing feature extraction on the target image, and determining an initial image feature corresponding to the target image;

respectively determining the initial image features corresponding to the time sequence image features of each time slice in a plurality of continuous time slices based on the initial image features;

generating a dynamic video corresponding to the target image based on the time sequence image characteristics corresponding to each time slice; the position of the dynamic object in the target image in the dynamic video changes.

2. The method of claim 1, wherein the plurality of consecutive time slices comprises a plurality of time slice groups, each time slice group comprising a first time slice and a second time slice adjacent to the first time slice and subsequent to the first time slice;

the determining, based on the initial image features, that the initial image features correspond to time-series image features of each of a plurality of consecutive time slices respectively includes:

for each time slice group in the plurality of time slice groups, performing feature fusion processing on the time sequence image features corresponding to a first time slice in the time slice group and the initial image features by using a preset time sequence feature extraction network to obtain time sequence image features corresponding to a second time slice in the time slice group;

and when the first time slice is the earliest time slice in the plurality of continuous time slices, processing the initial image characteristics by using a preset time sequence characteristic extraction network to obtain the time sequence image characteristics corresponding to the first time slice.

3. The method according to claim 1, wherein the performing feature extraction on the target image and determining an initial image feature corresponding to the target image comprises:

performing feature extraction on the target image by using an image encoder in the target neural network, and determining a first image feature of the target image extracted under a target first convolution layer as the initial image feature; the image encoder comprises a plurality of first convolution layers, and the target first convolution layer is a first convolution layer with the minimum number of corresponding convolution kernels in the plurality of first convolution layers.

4. The method according to claim 1, wherein the generating a dynamic video corresponding to the target image based on the time-series image feature corresponding to each time slice comprises:

respectively generating a derivative image corresponding to each time slice based on the time sequence image characteristics corresponding to each time slice; wherein the position of the dynamic object in different derived images is different;

and generating a dynamic video corresponding to the target image based on the generated derivative images and the time slices corresponding to the derivative images respectively.

5. The method of claim 4, wherein generating the derivative image corresponding to a time slice based on the time-series image feature corresponding to the time slice comprises:

decoding the time sequence image characteristics corresponding to the time slice by using an image decoder in the target neural network, and determining second image characteristics obtained by decoding under a target second convolution layer as target image characteristics; wherein the image decoder includes a plurality of second convolutional layers, and the target second convolutional layer is a second convolutional layer having a largest number of corresponding convolutional kernels among the plurality of second convolutional layers;

and generating a derivative image corresponding to the time slice based on the target image characteristic.

6. The method of any one of claims 1 to 5, further comprising the step of training the target neural network:

acquiring a sample dynamic video; the sample video comprises a plurality of frames of sample images;

inputting one frame of sample image in the multi-frame sample images into the target neural network to be trained, processing the frame of sample image by using the target neural network to be trained, and determining a predicted dynamic video corresponding to the frame of sample image;

determining a prediction loss corresponding to the target neural network to be trained based on each frame of predicted image in the predicted dynamic video and a plurality of frames of sample images included in the sample video;

and performing iterative training on the target neural network to be trained by utilizing the predicted loss until a training cut-off condition is met, so as to obtain the trained target neural network.

7. A dynamic video generation device is applied to a trained target neural network, and comprises the following components:

the acquisition module is used for acquiring a target image;

the first determining module is used for extracting the features of the target image and determining the initial image features corresponding to the target image;

a second determining module, configured to determine, based on the initial image features, time-series image features of each of a plurality of consecutive time slices corresponding to the initial image features, respectively;

the generating module is used for generating a dynamic video corresponding to the target image based on the time sequence image characteristics corresponding to each time slice; the position of the dynamic object in the target image in the dynamic video changes.

8. A computer device, comprising: a processor, a memory storing machine readable instructions executable by the processor, the processor for executing the machine readable instructions stored in the memory, the processor performing the steps of the dynamic video generation method as claimed in any one of claims 1 to 6 when the machine readable instructions are executed by the processor.

9. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when executed by a computer device, performs the steps of the dynamic video generation method according to any one of claims 1 to 6.

10. A computer program product comprising computer instructions, characterized in that the computer instructions, when executed by a processor, implement the steps of the dynamic video generation method of any of claims 1 to 6.

Images