CN118101856A - Image processing method and electronic device - Google Patents

Abstract

The embodiment of the present application is applied to the field of image processing, and provides an image processing method and electronic device. The method includes: obtaining an image to be processed; generating N target images in a target video and N target audios in a target video according to the image to be processed, wherein the N target images correspond to the N target audios one by one, and N is an integer greater than 1, and each target audio is used to be played when the target image corresponding to the target audio is displayed. Based on the technical method of the present application, the content of the played target audio can be more coordinated with the content of the displayed target image, thereby improving the user experience.

Description

Image processing method and electronic device

Technical Field

The present application relates to the field of image processing, and more specifically, to an image processing method and an electronic device.

Background technique

Nature is always in motion. Motion is one of the most eye-catching visual signals, and humans are very sensitive to it. Generating animated images based on images can improve user experience.

At the same time, there are various sounds in nature. By matching audio to the animated image, the audio can be played while the animated image is displayed, thereby enhancing emotional expression and making the displayed content more interactive and attractive. However, the sound matched to the animated image may not be consistent with the content of the image in the displayed animated image, affecting the user experience.

Summary of the invention

The present application provides an image processing method and an electronic device, which can improve the coordination between the played sound and the displayed image content and improve the user experience.

In a first aspect, an image processing method is provided, comprising: acquiring an image to be processed; generating N target images in a target video and N target audios in the target video based on the image to be processed, wherein the N target images correspond one-to-one to the N target audios, N is an integer greater than 1, and each target audio is used to be played when the target image corresponding to the target audio is displayed.

The method provided in the present application generates multiple target images according to the image to be processed, and generates audio corresponding to each target image, so that the played target audio is more coordinated with the content of the displayed target image, thereby improving the user experience.

In some possible implementations, generating N target images in a target video and N target audios in the target video according to the image to be processed includes: using an image processing system to process multiple first images in sequence to obtain at least one second image corresponding to each first image and the target audio corresponding to each second image, the multiple target images include the at least one second image corresponding to each first image, the at least one second image corresponding to each first image is an image after the first image in the target video, the first first image processed by the image processing system is the image to be processed, the i-th first image processed by the image processing system is an image in the at least one second image corresponding to the i-1-th first image processed by the image processing system, i is a positive integer greater than 1, and the image processing system includes a trained neural network model.

By using the second image obtained in the previous processing as the first image to be processed next time by the image processing system, when the target video is long, the generated image can be prevented from being offset or divergent, so that the target image in the target video has good image quality, thereby improving the user experience.

In some possible implementations, the method also includes: obtaining a target subject indicated by a user in the image to be processed; when the first image processed by the image processing system is the image to be processed, the image processing system is used to process the first image and target subject area information to obtain at least one second image corresponding to the first image, and the target audio corresponding to each second image, the target subject area information represents a target subject area, the target subject is recorded in the target subject area in the first image, and the contents recorded in other areas outside the target subject area in each second image are the same as the contents recorded in the other areas in the first image.

Compared with the first image, the part outside the target subject area where the target subject is located does not change in the second image. Therefore, the movement recorded in the target video is performed according to the user's instructions, which improves the user's sense of participation in video generation and improves user experience.

In some possible implementations, when the first image processed by the image processing system is the image to be processed, the image processing system is used to process the first image and the target subject area information to obtain at least one second image corresponding to the first image, and the target audio corresponding to each second image.

When the first image is the first image, the image processing system processes the first image and the target subject area information; when the first image processed by the image processing system is not the first image, the image processing system can process the first image, and the target subject area information can no longer be used as an input to the image processing system. Thus, the target video can reflect the movement of the target subject and the movement of other subjects affected by the target subject, making the movement recorded in the target video more reasonable and improving the user experience.

In some possible implementations, the method also includes: obtaining the target motion trend of the target subject indicated by the user; when the first image processed by the image processing system is the image to be processed, the image processing system is used to process the first image, the target subject area information and the motion information to obtain the target image corresponding to each second moment in the at least one second moment, and the target audio corresponding to each second moment, and the target subject in at least one of the target images corresponding to the at least one second moment moves according to the target motion trend represented by the motion information.

The image processing system generates a target video according to the target subject indicated by the user and the target motion trend of the target subject. The motion of the target subject in the target video is performed according to the target motion trend, thereby increasing the user's sense of participation in video generation and improving the user experience.

In some possible implementations, the image processing system includes a feature prediction model, an image generation model, and an audio generation model; the feature prediction model is used to process the first image to obtain at least one predicted feature, different predicted features correspond to different second moments, and each second moment is a moment after the first moment corresponding to the first image in the target video; the image generation model is used to process the at least one predicted feature respectively to obtain a second image corresponding to each second moment; the audio generation model is used to process the at least one predicted feature respectively to obtain a target audio corresponding to each second moment.

The image processing system processes the first image to obtain the predicted features corresponding to the second moment, and generates the target audio corresponding to the second moment and the second image corresponding to the second moment based on the predicted features corresponding to the second moment, so that the contents of the target audio and the target image corresponding to the same moment are more coordinated, thereby improving the user experience.

In some possible implementations, the feature prediction model includes a motion displacement field prediction model, a motion feature extraction model, an image feature extraction model and an adjustment model; the motion displacement field prediction model is used to process the first image to obtain a motion displacement field corresponding to each second moment, and the motion displacement field corresponding to each second moment represents the displacement of multiple pixels in the first image at the second moment relative to the first moment; the motion feature extraction model is used to perform feature extraction on the at least one motion displacement field respectively to obtain the motion feature corresponding to each second moment; the image feature extraction model is used to perform feature extraction on the first image to obtain image features; the adjustment model is used to adjust the image features according to the motion features corresponding to each second moment to obtain the predicted features corresponding to the second moment.

By predicting a motion displacement field for representing the displacement of a plurality of pixels in a first image between a first moment and a second moment, and adjusting the image features of the first image according to the motion features of the motion displacement field, the predicted features are made more accurate.

In some possible implementations, the method further includes: obtaining a target subject indicated by a user in the image to be processed; the motion displacement field prediction model is used to process the first image and the target subject area information to obtain the motion displacement field corresponding to each second moment, the target subject area information represents a target subject area, and the target subject is recorded in the target subject area in the image to be processed; the displacement of pixels outside the motion displacement field corresponding to each second moment is 0, and the pixels outside the area are located outside the target subject area in the first image.

In some possible implementations, the method further includes: obtaining a target motion trend of the target subject indicated by a user; when the first image processed by the image processing system is the image to be processed, the motion displacement field prediction model is used to process the first image, the target subject area information and the motion information to obtain the motion displacement field corresponding to each second moment, the target displacement of the target subject pixel represented by the motion displacement field corresponding to each second moment at the second moment relative to the first moment conforms to the target motion trend represented by the motion information, and the target subject pixel is a pixel located on the target subject in the first image.

In a second aspect, a training method for an image processing system is provided, the method comprising: obtaining training samples and label images and label audios, the training samples comprising sample images in a training video, the label images being images subsequent to the sample images in the training video, and the label audio being audio in the training video when the label images in the training video are displayed; processing the training samples using an initial image processing system to obtain training images and training audios; adjusting parameters of the initial image processing system according to a first difference between the training image and the label image, and a second difference between the training audio and the label audio, the adjusted initial image processing system being the trained image processing system.

In some possible implementations, the training sample may also include training subject area information, where the training subject area information represents the training subject area, where the training subject area in the sample image records the training target subject, and the content recorded in other areas outside the training subject area in the label image is the same as the content recorded in the other areas in the sample image.

In some possible implementations, when the sample image is the first frame image in a training video, the training sample includes training subject region information.

In some possible implementations, the training sample includes training motion information. Compared with the sample image, the training target subject in the label image moves according to the training motion trend represented by the training motion information.

Exemplarily, when the sample image is the first frame image in a training video, the training sample may include training motion information.

In some possible implementations, the initial image processing system includes an initial feature prediction model, an initial image generation model, and an initial audio generation model. The initial feature prediction model is used to process training samples to obtain training prediction features. The initial image generation model is used to process the training prediction features to obtain training images. The initial audio generation model is used to process the prediction features to obtain training audio. The initial feature prediction model after parameter adjustment is the feature prediction model in the image processing system, the initial image generation model after parameter adjustment is the image generation model in the image processing system, and the initial audio generation model after parameter adjustment is the audio generation model in the image processing system.

In some possible implementations, the initial feature prediction model includes a motion displacement field prediction model, an initial motion feature extraction model, an initial image feature extraction model, and an adjustment model. The motion displacement field prediction model is used to process the training samples to obtain a training motion displacement field, and the training motion displacement field represents the displacement of multiple training pixels in the sample image at the second training moment corresponding to the label image relative to the first training moment corresponding to the sample image. The second training moment corresponding to the label image and the first training moment corresponding to the sample image can be understood as the moments of the label image and the sample image in the training video, respectively. The initial motion feature extraction model is used to extract features from the motion displacement field to obtain training motion features. The initial image feature extraction model is used to extract features from the sample image to obtain training image features. The adjustment model is used to adjust the training image features according to the training motion features to obtain training prediction features. The initial motion feature extraction model after parameter adjustment is the motion feature extraction model in the image processing system, and the initial image feature extraction model after parameter adjustment is the image feature extraction model in the image processing system.

In a third aspect, an image processing apparatus is provided, comprising a unit for executing the method of the first aspect and/or the second aspect. The apparatus may be a terminal device or a chip in the terminal device.

In a fourth aspect, an electronic device is provided, comprising one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions so that the electronic device executes the method of the first aspect and/or the second aspect.

In a fifth aspect, a chip system is provided, which is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method of the first aspect and/or the second aspect.

In a sixth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes instructions, and when the instructions are executed on an electronic device, the electronic device executes the method of the first aspect and/or the second aspect.

According to a seventh aspect, a computer program product is provided. When the computer program product is run on an electronic device, the electronic device executes the method of the first aspect and/or the second aspect.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a hardware system of an electronic device applicable to the present application;

FIG2 is a schematic diagram of a software system of an electronic device applicable to the present application;

FIG3 is a schematic flow chart of an image processing method provided in an embodiment of the present application;

FIG4 is a schematic structural diagram of an image processing system provided in an embodiment of the present application;

FIG5 is a schematic diagram of the principle of a generation model applicable to an embodiment of the present application;

FIG6 is a schematic structural diagram of a random motion displacement field prediction model provided in an embodiment of the present application;

FIG7 is a schematic diagram of the change of the position of pixels in a video over time;

FIG8 is a schematic diagram of an implicit diffusion model provided in an embodiment of the present application;

FIG9 is a schematic flow chart of a training method for an image processing system provided in an embodiment of the present application;

FIG10 is a schematic structural diagram of a system architecture provided in an embodiment of the present application;

11 to 14 are schematic diagrams of graphical user interfaces provided in embodiments of the present application;

FIG. 15 is a schematic structural diagram of an image processing device provided in the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings.

FIG. 1 shows a hardware system of an electronic device suitable for the present application.

The method provided in the embodiments of the present application can be applied to various electronic devices capable of image processing, such as mobile phones, tablet computers, wearable devices, laptop computers, netbooks, personal digital assistants (PDAs), etc. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

1 shows a schematic diagram of the structure of an electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and/or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

For the software structure of the electronic device 100, the layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, as shown in FIG2, from top to bottom, respectively, the application layer, the application framework layer, the system library of the Android runtime (Android runtime), and the kernel layer.

The application layer may include a series of application packages. The application layer may include camera, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, wallpaper, gallery, setting and other application (application, APP). Among them, one or more applications such as wallpaper, gallery, setting can be used to execute the image processing method provided in the embodiment of the present application. The gallery can also be called album or media library, etc.

The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

The application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, a data integration module, and the like.

The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data and make the data accessible to applications. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon can include a view for displaying text and a view for displaying pictures. The phone manager is used to provide communication functions for the electronic device 100. The resource manager provides various resources for applications. The notification manager enables applications to display notification information in the status bar, which can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction.

Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

The system library can include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library, 2D graphics engine, etc.

The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of multiple common audio and video formats, as well as static image files. The media library can support multiple audio and video encoding formats. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing. The 2D graphics engine is a drawing engine for 2D drawing.

The kernel layer is the layer between hardware and software. The kernel layer can include driver modules such as display driver, camera driver, audio driver and sensor driver.

As the functions of electronic devices become increasingly rich, people can use the camera function of the electronic devices to capture images, and can also use the communication function of the electronic devices to receive images.

Generally speaking, users store a large number of pictures in the gallery of their mobile phones. Usually, users want to select some exquisite pictures or manually edited pictures to use as screen wallpapers. These exquisite pictures can only be used as background pictures to become static wallpapers, lacking dynamic display effects.

The natural world is always in motion, and even seemingly still scenes contain subtle vibrations caused by wind, water flow, breathing, or other natural rhythms. Movement is one of the most striking visual cues, and humans are particularly sensitive to it. If the image is captured without movement (or the movement is not realistic), it will usually feel unnatural or unreal. Static wallpapers are no longer enough to meet people's needs.

Dynamic images can be used as wallpapers. The production of dynamic images is a multimodal problem that integrates images, voice, effect rendering and artificial intelligence, and the processing is relatively complex.

Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is also the study of the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

Machine learning is an important branch of artificial intelligence, and deep learning is an important branch of machine learning. Deep learning refers to the use of multi-layer neural network structures to learn from big data the representation of various things in the real world that can be directly used for computer calculations (for example, things in images, sounds in audio, etc.). In the field of image processing, deep learning has achieved excellent results in object detection, image generation, image segmentation and other problems.

Machine learning can be applied to the generation of dynamic images, which can be understood as videos that include multiple images but do not include sound.

There are various sounds in nature. If only dynamic images are displayed, the user experience is poor.

Matching audio to animated images can add dubbing to them. This can enhance emotional expression and make the displayed content more interactive and attractive. However, when the audio is matched to the animated image, the sound may not be consistent with the image content, affecting the user experience.

In order to solve the above problems, the present application provides an image processing method and an electronic device.

The image processing method provided in the embodiment of the present application is described in detail below in conjunction with Figure 3. The execution subject of the method provided in the present application can be an electronic device, or a software/hardware module capable of image processing in the electronic device. For the sake of convenience, the following embodiments are described by taking the electronic device as an example.

Fig. 3 is a schematic flow chart of an image processing method provided in an embodiment of the present application. The method may include steps S310 to S320, and these steps are described in detail below.

Step S310, obtaining an image to be processed.

The image to be processed may be obtained by reading the image to be processed from a memory, or receiving the image to be processed from other electronic devices, etc. The image to be processed may also be an image selected by the user in an album.

Step S320, based on the image to be processed, generate N target images in the target video and N target audios in the target video, the N target images correspond one to one with the N target audios, N is an integer greater than 1, and each target audio is used to play when the target image corresponding to the target audio is displayed.

That is, multiple target images and target audio corresponding to each target image can be generated according to the image to be processed, and the target video can include the multiple target images and the target audio corresponding to each target image. The target audio corresponding to each target image can also be understood as the audio matching the target image.

A plurality of target images are generated according to the image to be processed, and audio corresponding to each target image is generated, so that the content of the played target audio is more coordinated with the content of the displayed target image, thereby improving the user experience.

In step S320, the image to be processed may be processed by an image processing system to obtain the N target images and the N target audios. However, if too many target images are generated from one image, the target images may be offset or divergent, blurred, and have poor image quality, resulting in a poor user experience.

When the target video is long and the number of target images is large, in order to make the generated target image have high clarity and image quality, the image processing system can be used to process multiple first images in sequence, at least one second image corresponding to each first image and the audio corresponding to each second image. The multiple target images include at least one second image corresponding to each first image. The audio corresponding to each second image is the target audio corresponding to the second image. The at least one second image corresponding to each first image is the image after the first image in the target video.

The first first image processed by the image processing system may be an image to be processed. The i-th first image processed by the image processing system may be an image in at least one second image corresponding to the i-1-th first image processed by the image processing system, where i is a positive integer greater than 1. Exemplarily, the i-th first image may be the last image in the target video in at least one second image corresponding to the i-1-th first image.

The image processing system includes a trained neural network model.

Exemplarily, when the duration of the target video is greater than or equal to a preset duration threshold, the image processing system may be used to process the plurality of first images in sequence.

In the case where a first image corresponds to more than one second image, the processing of the first image by the image processing system can be understood as the processing of the first image and time information by the image processing system, and different time information indicates different time intervals with the first image. The image processing system processes the first image and certain time information to obtain a second image corresponding to the first image whose time interval with the first image is the time interval indicated by the time information.

The difference between the time intervals represented by the time information may be an integer multiple of a preset time length, or may be other values. The preset time length may be understood as the time interval between two adjacent target images. In other words, the time intervals between two adjacent target images may be the same or different.

Exemplarily, when the differences between the time intervals represented by the time information are all integer multiples of a preset duration, the duration of each target audio may be the preset duration, and within the preset duration, the sound in the target audio may or may not change.

When the difference between the time intervals represented by the time information is not determined according to a preset duration, the playback duration of each target audio may be determined according to the time interval represented by the time information, and the sound of the target audio may not change.

In the case where the number of at least one second image corresponding to each first image is one, the second image may be an image of a preset duration after the first image in the target video. In other words, the time intervals between two adjacent target images may be equal.

The image processing system may include a feature prediction model, an image generation model, and an audio generation model.

The feature prediction model is used to process the first image to obtain at least one prediction feature corresponding to at least one second moment, each of which is a moment after the first moment corresponding to the first image in the target video.

The image generation model is used to process the prediction features corresponding to each second moment respectively to generate a second image corresponding to the second moment.

The audio generation model is used to process the predicted features corresponding to each second moment respectively to generate the target audio corresponding to the second moment.

The target audio that is the same as the second moment corresponding to a second image may be understood as the target audio corresponding to the second image.

Image generation models and audio generation models may be generation models.

The first image is processed to obtain the predicted features corresponding to the second moment, and the target audio corresponding to the second moment and the second image corresponding to the second moment are generated according to the predicted features corresponding to the second moment, so that the target audio and the second image are more correlated, that is, the content is more coordinated, thereby improving the user experience.

Exemplarily, the feature prediction model may include a motion displacement field prediction model, a motion feature extraction model, an image feature extraction model, and an adjustment model.

The motion displacement field prediction model can also be called a random motion displacement field prediction model, which is used to process the first image to obtain the motion displacement field corresponding to each second moment. The motion displacement field corresponding to each second moment represents the displacement of multiple pixels in the first image at the second moment relative to the first moment.

The motion feature extraction model is used to extract features from at least one motion displacement field to obtain motion features corresponding to each second moment.

The motion feature extraction model may be a convolutional neural network. The motion feature may include the output of the last layer of the motion feature extraction model, and may also include the output of multiple hidden layers of the motion feature extraction model.

The image feature extraction model is used to extract features from the first image to obtain image features.

The image feature extraction model may be a convolutional neural network. The image feature may include the output of the last layer of the image feature extraction model, and may also include the output of multiple hidden layers of the motion feature extraction model.

The adjustment model is used to adjust the image features according to the motion features corresponding to each second moment to obtain the prediction features corresponding to the second moment.

The adjustment model may be an activation function, for example, an activation function softmax.

By predicting a motion displacement field for representing the displacement of a plurality of pixels in a first image between a first moment and a second moment, and adjusting the image features of the first image according to the motion features of the motion displacement field, the predicted features are made more accurate.

The motion displacement field prediction model may include a first transformation module, a motion texture prediction model and a second transformation module.

The first transform module is used to perform Fourier transform on the first image to obtain image frequency domain data. The image frequency domain data can be understood as the frequency domain representation of the first image.

The motion texture prediction model may also be called a random motion texture prediction model, which is used to process image frequency domain data to generate motion texture. The motion texture prediction model may be a generative model, such as a diffusion model or an implicit diffusion model (latent diffusion models, LDM).

The second transformation module is used to perform inverse Fourier transformation on the motion texture to obtain a motion displacement field.

In the case where the motion texture prediction model is LDM, the motion texture prediction model is LDM including a compression model and a motion texture generation model.

The compression model is used to compress the image frequency domain data to obtain image compressed data. The compression model can be an encoder, for example, an autoencoder in a VAE.

The motion texture generation model is used to process the image compression data to generate motion texture. The motion texture generation model may be a diffusion model.

Exemplarily, the motion texture generation model may include a diffusion model and a decompression model. The diffusion model may be used to perform multiple denoising processes on the compressed frequency domain noise data according to the image compression data to obtain the compressed motion texture. The decompression model may be used to decompress the compressed motion texture to obtain the motion texture. The denoising process may be understood as the removal of Gaussian noise.

The compressed frequency domain noise data may be obtained by compressing the frequency domain noise data with a compression model, or may be preset.

Before performing step S320, a target subject indicated by the user in the image to be processed may also be obtained.

According to the target subject area indicated by the user, target subject area information can be determined. The target subject area information indicates the target subject area. The target subject indicated by the user is recorded in the target subject area in the image to be processed.

The image processing system is used to process the first image and the target subject area information to obtain at least one second image corresponding to the image to be processed, and the target audio corresponding to each second image.

The content recorded in other areas outside the target subject area in each second image is the same as the content recorded in the other areas in the first image.

That is, in the other area, the pixel value of the second image is the same as that of the first image at the same pixel. Compared with the first image, the part outside the target subject area where the target subject is located does not change in the second image. Therefore, the movement recorded in the target video is performed according to the user's instructions, which improves the user's sense of participation in the video generation and improves the user experience.

The pixel value of an image can be a color value, and the pixel value can be a long integer representing a color. For example, the pixel value can be a color value represented by red, green, and blue (RGB). In each color component, the smaller the value, the lower the brightness, and the larger the value, the higher the brightness. For a grayscale image, the pixel value can be a grayscale value.

In the image processing system, the feature prediction model is used to process the first image and the target subject area information to obtain the prediction features corresponding to each second moment. Thus, the influence of the target subject indicated by the user is reflected in the prediction features. The image generation model and the audio generation model can perform subsequent processing based on the prediction features reflecting the influence of the target subject indicated by the user.

Exemplarily, the motion displacement field prediction model in the feature prediction model can be used to process the first image and the target subject area information to obtain the motion displacement field corresponding to each second moment. Thus, the motion displacement field reflects the influence of the target subject indicated by the user. The motion feature extraction model and the adjustment model can perform subsequent processing based on the motion displacement field reflecting the influence of the target subject indicated by the user.

During the processing of the first image and the target subject area information by the motion displacement field prediction model, the first transformation module can perform Fourier transformation on the first image to obtain image frequency domain data. The motion texture prediction model is used to process the image frequency domain data and the target subject area information to obtain the motion texture corresponding to each second moment. The second transformation module is used to perform inverse Fourier transformation on the motion texture to obtain the motion displacement field.

In the motion texture prediction model, the compression model can be used to compress the image frequency domain data to obtain image compression data. The motion texture generation model can be used to process the image compression data and the target subject area information to generate motion texture.

Alternatively, the compression model can also be used to compress the target subject area information to obtain compressed area information. The motion texture generation model can be used to process the compressed area information and the image compression data to generate motion texture.

After obtaining the target subject indicated by the user in the image to be processed, if the image processing system processes the first image and the target subject area information for each first image, the movement of the subject in the target video may be unreasonable. For example, in practice, the movement of the target subject may cause the movement of other subjects, and the other subjects may be located outside the target subject area.

In order to make the movement of the subject in the target video more reasonable, when the first image processed by the image processing system is the image to be processed, the image processing system can process the first image and the target subject area information; and when the first image processed by the image processing system is not the first image, the image processing system can process the first image, and the target subject area information can no longer be used as the input of the image processing system. Thus, the movement of the target subject and the movement of other subjects affected by the target subject can be reflected in the target video, making the movement of the subject in the target video more reasonable and improving the user experience.

In the case of acquiring the target subject indicated by the user in the image to be processed, before step S320 , the target motion trend of the target subject indicated by the user may also be acquired.

The target motion trend of the target subject may include one or more of the target subject's motion direction, motion amplitude, motion speed, and the like.

The image processing system can be used to process the first image, target subject area information and motion information to obtain the target image corresponding to each second moment in at least one second moment, and the target audio corresponding to each second moment. In at least one of the target images corresponding to the at least one second moment, the target subject moves according to the target motion trend represented by the motion information.

The image processing system generates a target video according to the target subject indicated by the user and the target motion trend of the target subject. The motion of the target subject in the target video is performed according to the target motion trend, thereby increasing the user's sense of participation in video generation and improving the user experience.

The target motion trend of the target subject indicated by the user is generally for the image to be processed in the image to be processed. Since the motion of the target subject may be affected by various factors, the motion trend of the target subject may not always be the same as the target motion trend in the target video.

When the first image processed by the image processing system is the image to be processed, the image processing system can process the first image, the target subject area information and the motion information; and when the first image processed by the image processing system is not the first image, the image processing system can process the first image, or the image processing system can process the first image and the target subject area information, and the motion information may no longer be used as an input to the image processing system.

Exemplarily, the motion displacement field prediction model in the feature prediction model can be used to process the first image, the target subject area information and the motion information to obtain the motion displacement field corresponding to each second moment. Thus, the motion displacement field reflects the target motion trend influence of the target subject indicated by the user. The motion feature extraction model and the adjustment model can perform subsequent processing based on the motion displacement field.

In the process of processing the first image, the target subject area information and the motion information by the motion displacement field prediction model, the first transformation module can perform Fourier transformation on the first image to obtain image frequency domain data. The motion texture prediction model is used to process the image frequency domain data, the target subject area information and the motion information to obtain the motion texture corresponding to each second moment. The second transformation module is used to perform inverse Fourier transformation on the motion texture to obtain the motion displacement field.

In the motion texture prediction model, the compression model can be used to compress the image frequency domain data to obtain image compression data. The motion texture generation model can be used to process the image compression data, target subject area information and motion information to generate motion texture.

Alternatively, the compression model can also be used to compress the target subject area information to obtain compressed area information. The motion texture generation model can be used to process the compressed area information, image compression data and motion information to generate motion texture.

Alternatively, the compression model can compress the image frequency domain data, target subject area information and motion information respectively, and the motion texture generation model can be used to process the compressed image frequency domain data, compressed target subject area information and compressed motion information to generate motion texture.

Thus, the target video can reflect the movement of the target subject and the influence of other subjects on the target subject, and the target subject starts to move according to the target movement trend, making the movement of the subject in the target video more reasonable and improving the user experience.

The image processing method provided in the embodiment of the present application generates multiple target images according to the image to be processed, and generates audio corresponding to each target image, so that the content of the played target audio is more coordinated with the content of the displayed target image, thereby improving the user experience.

It should be understood that the above examples are intended to help those skilled in the art understand the embodiments of the present application, rather than to limit the embodiments of the present application to the specific numerical values or specific scenarios illustrated. Those skilled in the art can obviously make various equivalent modifications or changes based on the above examples, and such modifications or changes also fall within the scope of the embodiments of the present application.

The image processing system used in the method shown in FIG. 3 is described below in conjunction with FIG. 4 .

FIG. 4 is a schematic structural diagram of an image processing system provided in an embodiment of the present application.

The image processing system 400 includes a random motion displacement field prediction model 410 , a motion feature extraction model 420 , an image feature extraction model 430 , an adjustment model 440 , an image generation model 450 , and an audio generation model 460 .

The image processing system 400 can be used to process multiple first images in sequence to obtain multiple second images and audio corresponding to each second image. The output of the image processing system 400 includes sound and image, and the output of the image processing system 400 can be understood as multimodal.

The image processing system 400 processes the multiple first images sequentially. Among the multiple first images, the first image used by the image processing system 400 for the first processing may be the image to be processed. The first image used by the image processing system 400 for the second and subsequent processing may be the second image obtained in the previous processing. A video can be obtained based on the multiple second images and the audio corresponding to each second image. The video may include multiple second images and the audio corresponding to each second image. Among the multiple second images, different second images correspond to different moments. The video can be understood as a video generated by the image processing system 400.

Exemplarily, the video may also include an image to be processed, which may be understood as the first frame of the video.

The number of second images obtained by the image processing system 400 each time processing a first image may be one or more.

When the number of second images obtained by the image processing system 400 from processing the first image is one, the second image is an image corresponding to a time after the time corresponding to the first image. The second image can be used as the first image to be processed by the image processing system 400 next time.

When the number of second images obtained by the image processing system 400 from processing the first image is multiple, the multiple second images are images corresponding to multiple moments after the moment corresponding to the first image. The image with the latest moment among the multiple second images can be used as the first image to be processed by the image processing system 400 next time. The image with the latest moment among the multiple second images can also be understood as the image with the largest time interval between the moment corresponding to the first image and the moment corresponding to the first image.

In the case where the first image is the image to be processed, the input of the image processing system 400 may further include target subject area information and/or motion information, etc.

The target subject region information is used to indicate the target subject region in the image to be processed. The target subject region may be determined according to the target subject in the image to be processed indicated by the user. In the image to be processed, the target subject is located in the target subject region. The shape of the target subject region may be preset or determined according to the shape of the target subject. The shape of the target subject region may be regular or irregular.

The target subject area may also include an area around the target subject in the image to be processed. For example, the target subject area may include all or part of an area in the image to be processed whose distance to the area where the target subject is located is less than or equal to a preset distance.

The target subject area information may be represented by an image or other means. For example, the target subject area information may be an area image. The area image has the same size as the image to be processed. The area image may indicate whether the pixel is located in the target subject area by the value of each pixel. Exemplarily, the value of the pixel located in the target subject area may be 1, and the value of the pixel located outside the target subject area may be 0; or, the value of the pixel located in the target subject area may be greater than or equal to a preset value, and the value of the pixel located outside the target subject area may be less than the preset value. Different pixels may be understood as different points, i.e., different positions.

When the shape of the target subject area is a preset shape, the target subject area information may represent the target subject area through information such as position and size.

For example, if the preset shape is a square, the target body area information may include the positions of two vertices located on the diagonal line of the target body area in the avatar to be processed, or the target body area information may include the position of the center of the target body area in the avatar to be processed, and information used to represent the length and width of the target body area, such as the length and width, or the length and aspect ratio, etc. If the preset shape is a circle, the target body area information may include the position of the center of the target body area in the avatar to be processed, and information used to represent the radius of the target body area.

On the image to be processed, a coordinate system can be established with the center or a vertex of the image to be processed as the origin. The position of a point in the image to be processed can be expressed in the form of coordinates.

The motion information may be used to indicate the target motion trend of the target subject indicated by the user, such as one or more of the motion amplitude, motion direction, motion speed, etc.

The first image may be represented as I _0. The random motion displacement field prediction model 410 is used to process the first image I ₀ to obtain at least one random motion displacement field of the first image I ₀ , and the at least one random motion displacement field of the first image I ₀ corresponds to at least one time after the time corresponding to the first image I _0. Different random motion displacement fields correspond to different times.

The displacement field refers to the displacement of multiple points in space. The displacement of each point includes magnitude and direction. The displacement of each point can be represented by a vector.

The random preset displacement field of the first image I ₀ at a certain moment represents the displacement of the positions of the multiple pixels in the first image I ₀ at the moment relative to the positions of the multiple pixels in the first image I _0. The multiple pixels in the first image I ₀ represented by the random preset displacement field may be all or part of the pixels in the first image I ₀ .

The motion feature extraction model 420 is used to extract features from the random motion displacement field D of the first image I ₀ , and obtain motion features. The motion features can be understood as features of the random motion displacement field D of the first image I ₀ .

The image feature extraction model 430 is used to extract features from the first image I ₀ to obtain a first feature. The first feature can be understood as a feature of the first image I ₀ , and can also be called an image feature.

The adjustment model 440 is used to adjust the first feature according to the motion feature to obtain the second feature. The second feature can also be called a prediction feature.

The image generation model 450 is used to process the second feature to generate a second image.

The audio generation model 460 is used to process the second feature to generate audio corresponding to the second image.

The random motion displacement field of the first image I ₀ may have the same size as the first image I ₀ .

The random motion displacement field prediction model 410 , the motion feature extraction model 420 , the image feature extraction model 430 , the image generation model 450 , and the audio generation model 460 are all deep neural networks (DNN).

A neural network may be composed of neural units, and a neural unit may refer to an operation unit with x _s and an intercept l as input, and the output of the operation unit may be:

Where s=1, 2, ...n, n is a natural number greater than 1, _Ws is the weight of _xs , and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

DNN can also be called a multi-layer neural network, which can be understood as a neural network with multiple hidden layers. According to the position of different layers, DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.

Although DNN looks complicated, the work of each layer is not complicated. The work of each layer in the deep neural network can be expressed mathematically as To describe: From a physical level, the work of each layer in a deep neural network can be understood as completing the transformation from input space to output space (i.e., from row space to column space of a matrix) through five operations on the input space (a set of input vectors). These five operations include: 1. Dimension increase/reduction; 2. Enlargement/reduction; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are represented by/> Completed, the operation of 4 is done by/> Completed, the operation of 5 is done by/> The reason why the word "space" is used here is because the object being classified is not a single thing, but a class of things, and space refers to the collection of all individuals of this class of things. Among them, /> is a weight vector, each value in which represents the weight value of a neuron in this layer of neural network. Determines the spatial transformation from the input space to the output space mentioned above, that is, the weight of each layer/> Controls how to transform space. The purpose of training a deep neural network is to eventually obtain the weight matrix of all layers of the trained neural network (the weight matrix formed by many layers of vectors W ). Therefore, the training process of a neural network is essentially learning how to control spatial transformation, more specifically learning the weight matrix.

Therefore, DNN is simply the following linear relationship expression: , where /> is the input vector, /> is the output vector, /> is the offset vector, W is the weight matrix (also called coefficient), /> is the activation function. Each layer is just an input vector/> After such a simple operation, the output vector is obtained/> . Due to the large number of DNN layers, the coefficient W and the offset vector/> The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as/> The superscript 3 represents the layer number of the coefficient W , while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

In summary, the coefficient from the kth neuron in the L−1th layer to the jth neuron in the Lth layer is defined as .

It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better depict complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector W ).

The random motion displacement field prediction model 410 may be a neural network model, for example, a convolutional neural network.

A convolutional neural network is a deep neural network with a convolutional structure. It is a deep learning architecture. A deep learning architecture refers to multiple levels of learning at different levels of abstraction through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network in which each neuron can respond to the data input into it.

Convolutional neural networks contain a feature extractor consisting of a convolution layer and a subsampling layer. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve an input data or convolution feature plane (feature map). The convolution layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolution layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layers. A convolution layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are the convolution kernels. Taking the input data as an image as an example, shared weights can be understood as the way to extract image information is independent of position. The implicit principle is that the statistical information of a part of the image is the same as that of other parts. This means that the image information learned in a certain part can also be used in another part. So for all positions on the image, we can use the same learned image information. In the same convolution layer, multiple convolution kernels can be used to extract different image information. Generally speaking, the more convolution kernels there are, the richer the image information reflected by the convolution operation.

The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

A convolutional neural network may include an input layer, a convolutional layer, and a neural network layer. A convolutional neural network may also include a pooling layer.

The convolution layer can include many convolution operators, also called kernels. Its role in natural language processing is equivalent to a filter that extracts specific information from the input speech or semantic information. The convolution operator can essentially be a weight matrix, which is usually predefined.

The weight values in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values obtained through training can extract information from the input data, thereby helping the convolutional neural network to make correct predictions.

Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer. It can be a convolution layer followed by a pooling layer, or multiple convolution layers can be followed by one or more pooling layers.

In the data processing process, the only purpose of the pooling layer is to reduce the spatial size of the data. The pooling layer may include an average pooling operator and/or a maximum pooling operator to sample the input data features to obtain data features of smaller size.

After being processed by the convolution layer/pooling layer, the convolutional neural network is not sufficient to output the required output information. Because as mentioned above, the convolution layer/pooling layer will only extract features and reduce the parameters brought by the input data. However, in order to generate the final output information (the required class information or other related information), the convolutional neural network needs to use the neural network layer to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer may include multiple hidden layers and an output layer, and the parameters contained in the multiple hidden layers can be pre-trained according to the relevant training data of the specific task type. For example, the task type may include speech or semantic recognition, classification or generation, etc.

After the multiple hidden layers in the neural network layer, the last layer of the entire convolutional neural network is the output layer. The output layer has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network is completed, the back propagation will begin to update the weight values and biases of the previously mentioned layers to reduce the loss of the convolutional neural network and the error between the result output by the convolutional neural network through the output layer and the ideal result.

It should be understood that the above introduction to the convolutional neural network is only an example of a convolutional neural network. In specific applications, the convolutional neural network can also exist in the form of other network models. For example, multiple convolutional layers/pooling layers are in parallel, and the features extracted from each layer are input to the full neural network layer for processing.

The random motion displacement field prediction model 410 may be a generative model.

The generative model is essentially a set of probability distributions. As shown in (a) of Figure 5, the training data in the training dataset can be understood as The random samples taken from the data set are independent and identically distributed with the data set. The right side is its generative model (i.e., generative model). Generative model determination and distribution/> The nearest distribution/> , then in the distribution/> By collecting new samples, we can obtain a steady stream of new data.

The structure of the generative model can be a generative adversarial network (GAN), a variational autoencoder (VAE), a flow-based model, or a diffusion model.

(b) in Figure 5 shows the principle of GAN. GAN consists of at least two modules: one module is the generative model (generator), and the other module is the discriminative model (discriminator). These two modules learn from each other through game, so as to produce better output. The basic principle of GAN is as follows: Taking the GAN that generates pictures as an example, suppose there are two networks, the generative model G and the discriminative model D, where G is a network that generates pictures. It receives a random noise z and generates pictures through this noise, which is recorded as G(z); D is a discriminative network used to determine whether a picture is "real". Under ideal conditions, G can generate pictures G(z) that are "real", while D has difficulty in determining whether the pictures generated by G are real, that is, D(G(z)) = 0.5. In this way, an excellent generative model G is obtained, which can be used to generate pictures or other types of data.

(c) in Figure 5 shows the principle of VAE. VAE is a deep generative model based on the autoencoder structure. Autoencoders are widely used in fields such as dimensionality reduction and feature extraction. VAE includes an autoencoder and a decoder. The data is mapped to latent variables in a low-dimensional space through the encoding process of the autoencoder. Then the decoder performs a decoding process to restore the latent variables to reconstructed data. In order to make the decoding process have generative capabilities rather than a unique mapping relationship, the latent variables are random variables that obey a normal distribution.

(d) in Figure 5 shows the principle of the flow-based generative model. The flow-based generative model can also be called a flow model. The flow model includes a series of reversible transformers. The data x is processed by the series of reversible transformers, which can be expressed as f(x) to obtain the transformed data z. The transformed data z is subjected to the inverse transformation f ^-1 (z) of f(x) to obtain the reconstructed data. The flow model enables the model to learn the data distribution more accurately, and its loss function is a negative log-likelihood function. The training process of the flow model can be understood as learning the reversible transformation f(x) required to transform from the complex distribution of data x to the simple distribution of data z. In the training process of the flow model, data z is obtained by random sampling, and the inverse transformation f ^-1 (z) of f(x) is used to convert the data z into the reconstructed data.

(e) in Figure 5 shows the principle of the diffusion model. The diffusion model is essentially a Markov chain architecture, but the training process uses the back-propagation algorithm of deep learning. The idea of the diffusion model comes from non-equilibrium thermodynamics, and the theoretical basis of the algorithm is to train parameterized Markov chains through variational inference. A key property of the Markov chain is stationarity, that is, if a probability changes over time, then under the action of the Markov chain, it will tend to a certain stationary distribution, and the longer the time, the more stable the distribution. This process requires the property of "no memory", that is, the probability distribution of the next state can only be determined by the current state, and the previous events in the time series are irrelevant.

The framework of the diffusion model can adopt the denoising diffusion probabilistic model (DDPM), score-based generative models (SGM) or stochastic differential equation (SDE).

The model structure of the diffusion model can be a convolutional neural network, for example, a U-net, a noise conditional score network (NCSN), an upgraded version of NCSN (NCSN++), etc.

The use of diffusion models involves forward diffusion process and reverse diffusion process.

The forward diffusion process can also be called the forward diffusion process, the forward process or the diffusion process, which corresponds to the diffusion process in molecular thermodynamics. The reverse diffusion process can also be called the reverse diffusion process, the abstract process, the reverse process or the denoising process. In the forward diffusion process, the Markov chain of samples is generated by slowly adding noise. In the reverse diffusion process, the Markov chain of samples is generated by slowly removing noise.

The Markov chain consists of a series of states and a series of change probabilities. The state here refers to data with different noise levels, and the change probability refers to the probability of changing from the current state to the next state, which is implemented using a change matrix.

As shown in (e) of Figure 5, for the data , in the forward diffusion process, in the tth step of the T-step forward diffusion process, to the data/> Add a small amount of Gaussian noise to get the data/> , data/> represents the data obtained after t steps. Where t=1,2…,T, T is a positive integer. That is to say, for data/> , after T steps, the data obtained in each step are respectively/> , , ..., /> . Data/> is pure noise.

The noise added under different step indexes t may be different, and the change of the noise added under different step conditions may be called a difference schedule, for example, a linear schedule, a square schedule, a variance schedule, etc. The difference schedule may be preset.

For the image data shown in (f) in Figure 5, the forward diffusion process is to continuously add noise to the image until the image becomes pure noise. To data/> It is a Markov chain, which represents the random process of transition from one state to another in the state space.

During the forward diffusion process, the step index t can represent the number of times noise is added.

data ,/> , ..., /> It can be images of the same size, or it can be audio, frequency domain data, or other types of data.

Data obtained through the forward diffusion process To data/> , the diffusion model can be trained. The initial diffusion model is used to process the training noise data corresponding to each step index t when the step index t is t=1,2…,T to obtain the training denoised data corresponding to each step index t, and the parameters of the initial diffusion model are adjusted according to the difference between the training denoised data corresponding to the step index t and the label denoised data corresponding to the step index t. The difference can be represented by the loss value. The diffusion model is the initial diffusion model after parameter adjustment.

The training noise data corresponding to the step index t=T can be pure noise data. When t is other values than T, the training noise data corresponding to the step index t can be understood as the data obtained by adding t times of noise to the noise-free data. The label denoised data corresponding to the step index t can be understood as the data obtained by adding t-1 times of noise to the noise-free data.

That is, the training noise data corresponding to the step index t=T can be understood as data , when t is other than T, the training noise data corresponding to the step index t can be understood as data/> , the label denoised data corresponding to the step index t can be understood as data/> .

In the forward diffusion process, the training noise data corresponding to the step index t=T can be pure noise data. When t is a value other than T, the training noise data corresponding to the step index t can be obtained by removing t-1 noises from the pure noise data in the initial diffusion model.

The output of the initial diffusion model may be training denoised data, that is, the training denoised data may be the output of the initial diffusion model. Alternatively, the output of the initial diffusion model may be noise, that is, the training denoised data may be obtained by removing the noise represented by the output of the initial diffusion model based on the training noise data.

In the case where the training denoised data corresponding to the step index t is obtained by removing the noise represented by the output of the initial diffusion model based on the training noise data corresponding to the step index t, the parameters of the initial diffusion model are adjusted according to the difference between the training denoised data corresponding to the step index t and the label denoised data. This can be understood as adjusting the parameters of the initial diffusion model by the difference between the output of the initial diffusion model and the noise added by the training noise data relative to the training denoised data. The difference can be represented by a loss value.

The back diffusion process can be understood as the inference process of the diffusion model. In the back diffusion process, t can be understood as the remaining number of iterations plus 1.

In the reverse diffusion process, from the noise data Start by gradually removing data /> The noise in the data Restore to original data/> . (e) in Figure 5 shows the original image data/> Take the image as an example. For the image data shown in (e) in Figure 5, the reverse diffusion process gradually denoises it from the pure noise image until a real image is obtained. The pure noise image can also be called a Gaussian noise image or a noise image.

As shown in (g) of Figure 5, the trained diffusion model is used to denoise the input data based on the step index t to obtain the output data. The input data can be understood as data , the output data can be understood as data/> .

Both the forward diffusion process and the reverse diffusion process can be represented by corresponding stochastic differential equations and ordinary differential equations. And the optimization goal of the diffusion model (predicting the noise added at each step) can actually be understood as learning a gradient direction that is optimal for the current input to the target data distribution. This is very consistent with intuitive understanding, that is, the noise we add to the input actually makes the input far away from its original data distribution, and learning an optimal gradient direction in the data space is actually equivalent to walking in the direction of removing noise. In the reverse diffusion process, the use of deterministic ordinary differential equations can obtain deterministic sampling results; in the forward diffusion process, the final noise-added result of the forward diffusion process can also be obtained by constructing the inverse process of the ordinary differential equation of the reverse diffusion process (in fact, if we have a deterministic path, then the forward diffusion process and the reverse diffusion process are nothing more than walking forward and backward). This conclusion makes diffusion generation highly controllable, and there is no need to worry about the diffusion model's effects on the data. Processed data /> With data/> The complete similarity makes a series of controls possible.

The training process of the diffusion model can be simply understood as the process of gradually removing Gaussian noise from pure noise data through neural network learning.

For a T-step diffusion model, the index of each step is t. In the forward diffusion process, we start from a real image At the beginning, some Gaussian noise is randomly generated at each step, and then the generated noise is gradually added to the input image. When T is large enough, the image after adding noise is close to a Gaussian noise image. For example, for the model DDPM, T=1000 can be set. Through a training process, the neural network learns to learn from the data/> To data/> The added noise is then used in a reverse diffusion process to transform the noisy data At the beginning (the result of adding noise to the real image during training and random noise during sampling), the final image to be generated is obtained by gradually removing the noise. This means that each processing of the diffusion model is used to generate data that is the same as the input./> Similar data/> Fundamentally, the diffusion model works by corrupting noisy data by continuously adding Gaussian noise./> , and then learn to recover the data by reversing this process of gradually increasing noise. The randomly sampled noise is denoised multiple times to remove Gaussian noise using the diffusion model, and data can be generated through the reverse diffusion process.

In the back diffusion process, the input of the diffusion model, in addition to the data , and may also include conditions. The conditions may include information for indicating the step index t, and may also include other information. The other information may be related to the data /> related, so that the data generated by the diffusion model are more consistent with this other information.

The diffusion model is actually a latent variable model that uses a Markov chain (MC) to map to a latent space. Through the Markov chain, noise is gradually added to the data at each step t. To obtain the posterior probability/> , where/> Indicates data when t is 1 to T respectively/> , represents the input data and is also the latent space. The latent space has the same dimension as the input data.

In some embodiments, the random motion displacement field of the first image I ₀ is predicted in the time domain.

Through autoregression, the random motion displacement field can be predicted in the time domain. Autoregression is to predict the value that should be output next based on the sequence value at certain previous moments.

The random motion displacement field prediction model 410 may process the first image to obtain the random motion displacement field of the first image I _0. The moment corresponding to the random motion displacement field of the first image I ₀ is a moment after the moment corresponding to the first image.

When the random motion displacement field prediction model 410 predicts the random motion displacement field of the first image _I0 in the time domain, the training data of the random motion displacement field prediction model 410 may include training samples and label random motion displacement field. The training samples in the training data of the random motion displacement field prediction model 410 include sample images.

The training sample is processed using the initial random motion displacement field prediction model to obtain a training random motion displacement field. According to the difference between the training random motion displacement field and the label random motion displacement field, the parameters of the initial random motion displacement field prediction model are adjusted to obtain a random motion displacement field prediction model 410. The difference can be represented by a loss value. The random motion displacement field prediction model 410 can be understood as the initial random motion displacement field prediction model after parameter adjustment.

The sample image may be a frame image in a training video, and the label random motion displacement field may be a random motion displacement field of the sample image at a training time determined according to the training video. The training time is a time after the time corresponding to the sample image in the training video. The label random motion displacement field, i.e., the random motion displacement field of the sample image at the training time, indicates the positions of multiple pixels in the sample image at the training time relative to the time corresponding to the sample image.

Exemplarily, the random motion displacement field prediction model 410 may also process the first image and the time information to obtain the random motion displacement field at the moment indicated by the time information. The moment indicated by the time information may be a moment after the moment corresponding to the first image. The moment corresponding to the random motion displacement field is the moment indicated by the time information.

In the case where the input of the random motion displacement field prediction model 410 does not include time information, the time intervals between the moments corresponding to the random motion displacement fields obtained by the random motion displacement field prediction model 410 for processing different first images and the moments corresponding to the first images processed by the random motion displacement field prediction model 410 may be equal. In other words, the random motion displacement field prediction model 410 processes the first images to obtain the random motion displacement fields corresponding to the moments corresponding to the first images that are a preset time interval apart.

When the input of the random motion displacement field prediction model 410 includes time information, the time information can be understood as time embedding. When the time information changes, the random motion displacement field prediction model 410 can output the random motion displacement field corresponding to the moment represented by different time information. That is, by introducing time information, the random motion displacement field at different moments after the moment of the first image can be predicted based on the first image.

In the case where the input of the random motion displacement field prediction model 410 includes time information, the training samples in the training data of the random motion displacement field prediction model 410 may also include training time information. The training time information is used to indicate the training moment.

Exemplarily, the first image input into the random motion displacement field prediction model 410 may also be the first image after adding noise. Adding noise to the first image may be understood as adding noise in the spatial domain. In the case where the first image input into the random motion displacement field prediction model 410 is the first image after adding noise, in the training data of the random motion displacement field prediction model 410, the sample image may be an image obtained by adding noise to a frame image in the training video.

Exemplarily, the time information input into the random motion displacement field prediction model 410 may also be the time information after adding noise. The time information after adding noise may represent the training moment after adding noise. Adding noise to the time information may be understood as adding noise in the time domain. In the case where the first image input into the random motion displacement field prediction model 410 is the first image after adding noise, in the training data of the random motion displacement field prediction model 410, the training time information may be the information obtained by adding noise to the information representing the training moment.

By adding noise in the time domain or space domain, it can be understood that random noise is added to the motion, which increases the randomness of the motion.

According to the random motion displacement field at a certain moment, the second image at that moment can be determined. The second image can also be used as the first image and processed again by the image processing system 400, so as to obtain second images at more moments.

In the case where the sample image is determined based on the first frame image in the training video, the training sample in the training data of the random motion displacement field prediction model 410 may also include training target subject area information and/or training motion information.

The training target subject region information may be used to indicate a region where the training target subject is located in the first frame image of the training video, and this region may also be referred to as a training target subject region.

For a period of time after the training video starts, the training target subject is in a state of motion. For example, in the training video, the training target subject may always be in a state of motion.

Moreover, for a period of time after the training video starts, all subjects except the training target subject in the training video are in a stationary state.

The training motion information may represent one or more of the motion amplitude of the training target subject in the training video, the motion direction of the training target subject at the beginning of the training video, and the like.

Exemplarily, the duration of the training video may be preset, randomly determined, determined according to a preset rule, or manually determined.

Relevant research in computer graphics shows that the movement in nature, especially the oscillatory movement, can be described as the superposition of a set of harmonic vibrations, which can be represented by different frequencies, amplitudes and phases.

Adding random noise in the time domain and/or space domain may cause the picture to be unrealistic or unstable. In the prediction of the random motion displacement field in the time domain, the video generated according to the random motion displacement field cannot be guaranteed to have good temporal consistency and spatial continuity over a long period of time. As the number of generated second images increases, the video including the multiple second images may drift or diverge.

Video drift and divergence can be understood as ghosting or blurring of objects recorded in images corresponding to later frames, or even making them unrecognizable.

In other embodiments, the random motion displacement field may be predicted in the frequency domain.

As shown in FIG. 6 , the random motion displacement field prediction model 410 may include a first transformation module 411 , a random motion texture prediction model 412 , and a second transformation module 413 .

The first transform module 411 is used to perform Fourier transform on the first image I ₀ , convert the first image I ₀ into the frequency domain, and obtain image frequency domain data. The Fourier transform can be implemented by fast Fourier transform (FFT).

The random motion texture prediction model 412 is used to process the image frequency domain data to obtain the random motion texture S of the first image I ₀ .

The random motion texture S may also be referred to as a random motion spectrum, and includes a frequency representation of the motion trajectory of all or part of the pixels of the first image I ₀ , that is, a motion trajectory in the frequency domain. The random motion texture S includes a frequency representation of the motion trajectory of each pixel in the plurality of pixels. The frequency representation of the motion trajectory of each pixel may include a component of each frequency point in the K frequencies f ₀ to f _K.

The frequency representation of the motion trajectory of the pixel at position p in the first image I ₀ can be expressed as The pixel at position p may be a pixel of the first image I _0. The pixel at position p may also be understood as pixel p.

The second transformation module 413 is used to perform inverse Fourier transform on the random motion texture S of the first image _I0 , so as to convert the random motion texture S into a random motion displacement field D of the first image _I0 corresponding to at least one time after the time corresponding to the first image _I0 .

Exemplarily, the second transformation module 413 is used for frequency representation of the motion trajectory of pixel p in the random motion texture S of the first image _I0. Convert to obtain the motion displacement of pixel p at at least one moment/> .

That is, the random motion displacement field D corresponding to the first image _I0 at a certain moment includes the motion displacement of each pixel p in the plurality of pixels represented by the random motion texture S at that moment. .

In the random motion displacement field D at each preset moment, the motion displacement of pixel p at a certain moment is It is used to indicate the change of the position of pixel p at this moment relative to the position in the first image I ₀ , that is, it indicates the displacement of pixel p at this moment relative to the moment corresponding to the first image. That is to say, at this moment, the position of pixel p can be expressed as / > Therefore, combined with the position of pixel p in the first image I ₀ , according to the motion displacement of pixel p/> , the position of pixel p in the image at that moment can be determined.

The training data of the random motion texture prediction model 412 may include training samples and label random motion textures. The training samples of the random motion texture prediction model 412 may include training image frequency domain data. The training samples are processed using the initial random motion texture prediction model to obtain the training random motion texture. According to the difference between the training random motion texture and the label random motion texture, the parameters of the initial random motion texture prediction model are adjusted to obtain the random motion texture prediction model 412. The random motion texture prediction model 412 is the initial random motion texture prediction model after parameter adjustment. The adjustment of the parameters of the initial random motion texture prediction model can minimize the difference between the training random motion texture and the label random motion texture, or make the difference gradually converge. The difference can be represented by a loss value.

The training image frequency domain data may be obtained by performing Fourier transform on a sample image in the training video. The sample image may be a frame of image in the training video.

The labeled random motion texture includes a frequency representation of a motion trajectory of each pixel in a plurality of pixels in the sample image. The labeled random motion texture may be obtained by analyzing a segment of video following the sample image in the training video.

By processing a video after the sample image in the training video through feature tracking, optical flow extraction or particle video, the motion trajectory of each pixel in the sample image can be obtained. The motion trajectory of each pixel point is Fourier transformed to obtain the frequency domain representation of the motion trajectory, thereby obtaining the label random motion texture.

Feature tracking is a key task in computer vision. Feature tracking can determine the position changes of feature points in a video by tracking feature points. By detecting image feature points with common properties in multiple images in a video and matching the same feature points in different images, image comparison and tracking can be achieved, the position of feature points in different images in the video can be determined, and the feature points can be tracked, thereby estimating the motion of the feature points.

According to the motion estimation result of the feature point, the motion trajectory of the feature point in the first frame image of the video can be determined. The motion trajectory of the feature point can also be understood as the motion trajectory of the pixel point.

Optical flow (or optic flow) is a concept in the detection of object motion in the field of view. It is used to describe the motion of an observed target, surface or edge caused by the motion relative to the observer. Optical flow is the instantaneous speed of the pixel motion of a moving object in space on the observation imaging plane. It is a method that uses the change of pixels in the time domain in an image sequence and the correlation between adjacent frames to find the corresponding relationship between the previous frame and the current frame, thereby calculating the motion information of objects between adjacent frames.

A particle video is a video and a corresponding set of particles. That is, a particle video uses a set of particles to represent video motion. Particle i has a time-varying position (xi(t); yi(t)) defined between the particle's start and end frames. A particle video can be constructed by sweeping the video forward and then backward. Each particle is an image point sample with a long-duration trajectory and other properties.

Particle videos improve frame-to-frame optical flow by enforcing long-range appearance consistency and motion consistency.

(a), (b) and (c) in Figure 7 respectively show the change of the position of pixels in the video determined by feature point tracking, optical flow estimation and particle point video over time. In Figure 7, the horizontal axis represents time and the vertical axis represents the horizontal coordinate of the pixel in the video. As shown in (b) in Figure 7, optical flow estimation can easily lead to excessive smoothing of the optical flow field but a short trajectory, and the resulting motion texture is unrealistic or spots appear. As shown in (a) in Figure 7, the motion trajectory extracted by feature point tracking is long but sparse. As shown in (c) in Figure 7, the motion trajectory extracted by particle video is dense and long. Therefore, the motion trajectory of pixels in the training video can be extracted by particle video.

The embodiment of the present application avoids drift or divergence of the generated video as time increases by representing the motion texture of each pixel in the image in the frequency domain, that is, converting the first image I ₀ into image frequency domain data and determining the random motion texture based on the image frequency domain data.

In specific processing, the 10th frame and the subsequent 149 frames of the acquired real video can be used as training videos to generate motion trajectories of multiple pixels. Trajectories with incorrect motion estimation, trajectories with excessive motion due to camera motion, and trajectories where all pixels move at all times are removed to obtain the motion trajectories of multiple pixels after filtering. Based on the motion trajectories of multiple pixels after filtering, a labeled random motion texture can be obtained. The labeled random motion texture can be understood as the ground truth in the training process.

Generally speaking, motion texture has a specific distribution characteristic in frequency. The amplitude of motion texture can range from 0 to 100, and decays roughly exponentially with increasing frequency. Since the output of the diffusion model is between 0 and 1, the model can be stably trained and denoised. Therefore, the labeled random motion texture extracted from the real video can be normalized.

If the amplitude of the label random motion texture is scaled to [0,1] according to the width and height of the image, almost all coefficients will be close to zero at higher frequencies. Such training data will train the model, resulting in inaccurate motion. When the amplitude of the normalized label random motion texture is very close to zero, even a small prediction error during inference may lead to a large relative error after denormalization. To solve this problem, a simple but effective frequency adaptive normalization technique can be used. Specifically, the Fourier system of each frequency processing is normalized independently based on the statistics calculated based on the unnormalized motion texture.

The random motion texture S output by the random motion texture prediction model 412 can be a 2D motion spectrum diagram including 4K channels, where K is the number of frequencies in the random motion texture S, which can also be understood as the number of Fourier coefficients. At each frequency in the random motion texture S, four scalars can be used to represent the complex Fourier transform coefficients of the x and y dimensions. The image frequency domain data can be represented in the form of an image, and x and y can respectively represent the coordinates in the image frequency domain data represented in the form of an image. In the image frequency domain data represented in the form of an image, the coordinates x and y can be represented by complex Fourier transform coefficients. The complex Fourier transform coefficient z can be expressed as z=x+iy, where i is an imaginary unit.

Most natural oscillations and other motions are mainly composed of low-frequency components, and the spectrum of the motion decreases exponentially as the frequency increases, which indicates that most natural vibrations and other motions can be well represented by low-frequency frequencies. In the case of K=16, the number of Fourier coefficients is sufficient to realistically reproduce the original natural motion in a series of videos and scenes. Therefore, K can be greater than or equal to 16 and less than a preset threshold.

The random motion texture prediction model 412 may be a neural network model, for example, a generation model such as a diffusion model, or an LDM model.

During the training process of the random motion texture prediction model 412, the parameters of the initial random motion texture prediction model are adjusted according to the difference between the training random motion texture and the label random motion texture. It can be understood that the parameters of the initial random motion texture prediction model are adjusted according to the difference between the training denoised data obtained by removing Gaussian noise each time and the label denoised data corresponding to the denoising. The label denoised data corresponding to each denoising is determined based on the label random motion texture. The label denoised data corresponding to the last denoising is the label random motion texture, and the label denoised data corresponding to other denoising is obtained by adding Gaussian noise to the label random motion texture. The degree of Gaussian noise added to the label denoised data corresponding to each denoising is different.

In the case where the sample image is the first frame image in the training video, the training sample in the training data of the random motion texture prediction model 412 may also include training target subject area information and/or training motion information.

In some embodiments, the random motion texture prediction model 412 may be a diffusion model. In the case where the random motion texture prediction model 412 is a diffusion model, the condition may include image frequency domain data. If the first image is an image to be processed, the condition may also include target subject area information and/or motion information.

During the training process of the random motion texture prediction model 412, the training samples are processed using the initial random motion texture prediction model. This can be understood as using the initial random motion texture prediction model to remove Gaussian noise from the noise frequency domain data multiple times based on the training samples, thereby obtaining the training random motion texture.

Noise frequency domain data can be understood as noise represented in the frequency domain, that is, the frequency domain representation of pure Gaussian noise data. Training samples can be understood as the conditions in the diffusion model training process.

The random motion texture S uses four channels to represent one frequency. A direct method to predict a random motion texture S with K frequencies is to use a diffusion model as the random motion texture prediction model 412. The output of the diffusion model is a tensor with 4K channels. Training a model with a large number of output channels often results in over-smoothing of the output and reduced accuracy. Another method is to inject additional frequency parameters into the diffusion model to independently predict the motion spectrum of each frequency, but this will result in unrelated predictions in the frequency domain, resulting in unrealistic motion. In order to avoid the above problems, the random motion texture prediction model 412 can adopt LDM.

In some other embodiments, the random motion texture prediction model 412 may be an LDM.

The random motion texture prediction model 412 may also be an LDM, which is more efficient than the diffusion model while maintaining the quality of the generated image.

As shown in (a) of Fig. 8 , the LDM 500 may include two parts, namely, an encoder 510, a diffusion model (DM) 520, and a decoder 530. The encoder 510 and the decoder 530 may be an autoencoder and a decoder in a variational auto-encoder (VAE), respectively.

The encoder 510 is used to compress the input data, that is, to map the input data to a latent space. The latent space can also be called a potential space. Based on the result of the encoder 510 compressing the data, the diffusion model 520 is used to perform multiple iterations, and each iteration removes Gaussian noise, so that the result of multiple iterations is denoised compressed data. The decoder 530 is used to decode the denoised compressed data to obtain denoised data.

When the random motion texture prediction model 412 adopts the LDM 500, as shown in (a) of FIG8 , the encoder 510 in the LDM 500 is used to compress the image frequency domain data to obtain image compressed data. The compression of the image frequency domain data by the encoder 510 can also be understood as mapping the image frequency domain data to a latent space. The image compressed data can also be referred to as frequency embedding. The amount of the image compressed data is less than the amount of the image frequency domain data.

The encoder 510 in the LDM 500 may also be used to compress the noise frequency domain data to obtain compressed noise frequency domain data.

The diffusion model 520 in the LDM 500 is used to perform multiple denoising processes on the compressed noise frequency domain data according to the conditions to obtain compressed data. The decoder 530 in the LDM 500 is used to decode the compressed data to obtain the random motion texture S of the first image I _0. The conditions input to the diffusion model 520 include the image compression data. The conditions input to the diffusion model 520 may also include the step index t.

That is to say, each denoising process of the diffusion model 520 is performed on the data obtained by the previous denoising process according to the conditions. The data obtained by the previous denoising process can be understood as data , the data obtained from this denoising process can be understood as/> . Data/> The noise frequency domain data may be obtained by performing Tt denoising processes on the compressed noise frequency domain data by the diffusion model 520 , where T is the total number of denoising processes performed by the diffusion model 520 .

The noise frequency domain data may be obtained by adding Gaussian noise in the frequency domain to the motion texture.

When the random motion texture prediction model 412 is an LDM, the encoder and decoder in the random motion texture prediction model 412 may be pre-trained. For the pre-training of the encoder and decoder, the training data may include label data. The training data used in the pre-training process of the encoder and decoder may also be referred to as pre-training data.

In the pre-training process, the label data can be compressed by the initial encoder to obtain the training compressed data, and the training compressed data can be decompressed by the initial decoder to obtain the training decompressed data. According to the difference between the label data and the training decompressed data, the parameters of the initial encoder and the initial decoder are adjusted, and the adjusted initial encoder can be used as the encoder 510 in the random motion texture prediction model 412, and the adjusted initial decoder can be used as the decoder 530 in the random motion texture prediction model 412. The difference can be represented by a loss value.

As shown in (b) of FIG8 , in order to obtain the training data required for the diffusion model training in LDM 500, the label data can be input into the encoder 510 to obtain the label latent space features without adding noise. Then, Gaussian noise is gradually added to the latent space features to generate label latent variable features with different noise levels. The training data includes the label latent space features without adding noise and the label latent variable features with different noise levels. Among them, the label latent variable features with the highest noise level can be understood as pure noise.

For the LDM 500 applied to the image processing system of the present application, the label data may be a label motion texture.

During the process of training the diffusion model in LDM 500, the label latent variable features with the highest noise level are subjected to multiple denoising processes using the initial diffusion model to obtain multiple training latent variable features in turn.

The number of times the diffusion model performs denoising is the same as the number of times noise is added to the latent space features. The multiple training latent variable features correspond one-to-one to the multiple label latent variable features except the label latent variable feature with the highest degree of noise. The label latent variable feature corresponding to the first obtained training latent variable feature is the label latent variable feature with the highest degree of noise among the multiple label latent variable features except the label latent variable feature with the highest degree of noise, and the label latent variable feature corresponding to the last obtained training latent variable feature is the label latent variable feature with the lowest degree of noise. The degree of noise in the label latent variable feature corresponding to each training latent variable feature decreases as the sequence number of the multiple training latent variable features increases.

According to the difference between each training latent variable feature and the label latent variable feature corresponding to the training latent variable feature, the parameters of the initial diffusion model are adjusted to obtain a diffusion model. The diffusion model is the initial diffusion model after parameter adjustment. The difference between each training latent variable feature and the label latent variable feature corresponding to the training latent variable feature can be represented by a loss value.

If the first image is an image to be processed, the condition may further include target subject area information and/or motion information.

When the random motion texture prediction model 412 is LDM, the process of generating motion texture can also be understood as a process of frequency domain coordinated denoising.

In yet other embodiments, the random motion texture prediction model 412 may include a compression model and a diffusion model.

The compression model can be used to compress the image frequency domain data to obtain image compression data. The diffusion model can perform multiple denoising processes on the noise frequency domain data according to the conditions to obtain the random motion texture S of the first image I _0. The conditions input to the diffusion model 520 can include the image compression data and can also include the step index t.

In multiple denoising processes, each denoising process of the diffusion model can be understood as removing Gaussian noise from the input data according to conditions, and the obtained data can be used as the input data for the next denoising process of the diffusion model. The input data of the first denoising process is the noise frequency domain data, and the input data of the diffusion model in other denoising processes after the first one is the output of the diffusion model of the previous denoising process. The data obtained by the last denoising process of the diffusion model is the random motion texture S of the first image I _0.

The number of times of the multiple denoising processes is T, and the noise frequency domain data can be understood as the noise represented in the frequency domain. In other words, the input of the first denoising process of the diffusion model, ie, can be understood as the frequency domain representation of pure Gaussian noise data.

By data Taking the noise frequency domain data as an example, the i-th denoising process of the diffusion model can be understood as the input data according to the conditions/> Gaussian noise is removed to obtain data/> , t can be expressed as T-i+1. The data obtained by the last denoising process of the diffusion model/> is the random motion texture S of the first image I ₀ .

In the case where the random motion texture prediction model 412 includes a compression model and a diffusion model, the compression model may be obtained by pre-training. The pre-training of the compression model may refer to the pre-training of the encoder 510. The compression model may be an encoder.

When the random motion texture prediction model 412 includes a compression model and a diffusion model, the training of the random motion texture prediction model 412 can be understood as the training of the diffusion model.

In the training process of the random motion texture prediction model 412, the training samples are processed using the initial random motion texture prediction model. It can be understood that the image frequency domain data in the training samples is compressed using the VAE to obtain training image compression data, and the noise frequency domain data is repeatedly Gaussian noise removed based on the training samples using the initial diffusion model to obtain the training random motion texture. The training samples include the training image compression data.

According to the difference between the training random motion texture and the label random motion texture, the parameters of the initial random motion texture prediction model are adjusted. It can be understood that the parameters of the initial random motion texture prediction model are adjusted according to the difference between the training denoised data obtained by removing Gaussian noise each time and the label denoised data corresponding to the denoising. The label denoised data corresponding to each denoising is determined based on the label random motion texture. The label denoised data corresponding to the last denoising is the label random motion texture, and the label denoised data corresponding to other denoising is obtained by adding Gaussian noise to the label random motion texture. The degree of Gaussian noise added to the label denoised data corresponding to each denoising is different.

The training denoising data obtained by each Gaussian noise removal can be used as the basis for the next Gaussian noise removal. In other words, the first Gaussian noise removal can be performed on the noise frequency domain data, and the other Gaussian noise removal can be performed on the training denoising data obtained by the previous Gaussian noise removal.

The second transformation module 413 processes the random motion texture S of the first image I _0. The random motion texture S of the first image I ₀ is subjected to inverse Fourier transformation to obtain a random motion displacement field at least one preset time after the first image I _0. In the random motion displacement field at each preset time, the motion displacement field of each pixel p in the first image I ₀ is obtained by inverse Fourier transformation of the frequency representation _Sp of the motion trajectory of the pixel.

The inverse Fourier transform can be an inverse fast Fourier transform. That is, the motion displacement field of each pixel p is , where /> represents the inverse fast Fourier transform. The random motion displacement field at a preset time can be expressed as , where P is the number of pixels in the first image I ₀ .

According to the motion displacement field of each pixel in the random motion displacement field at the preset moment, the position of the pixel at the preset moment can be determined.

Compared with the diffusion model processing the image frequency domain data, the LDM 500 compresses the image frequency domain data and processes the compressed image data, which can reduce the overall data processing amount and improve the processing efficiency.

Using a generative diffusion model to predict the motion displacement field at future moments can provide fine-grained control over the subject of motion. Processing in the frequency domain to obtain motion textures instead of directly processing the original RGB pixels not only improves computational efficiency, but also keeps the generated motion representation consistent for a longer period of time.

In the positions of each pixel at the next moment determined according to the random motion displacement field of the first image I ₀ , the positions of multiple pixels at the next moment may overlap. In the image represented by the positions of each pixel at the next moment determined according to the random motion displacement field of the first image I ₀ , there may also be a position outside the position of each pixel p at the next moment, that is, there may be a hole in the image determined by the positions of each pixel at the next moment determined according to the random motion displacement field D of the first image I ₀ .

In order to avoid the above situation, the features of the first image I ₀ can be adjusted according to the features of the random motion displacement field D of the first image I ₀ , and the second image at the next moment and the audio at the next moment are generated according to the adjusted features. The audio at the next moment can be understood as the audio corresponding to the second image at the next moment.

The image feature extraction model 430 may be a convolutional neural network including multiple convolutional layers. The first feature may only include the first sub-feature output by the last convolutional layer in the process of the image feature extraction model 430 performing feature extraction on the first image I _0. Alternatively, the first feature may include multiple first sub-features respectively output by multiple convolutional layers in the image feature extraction model 430. The multiple convolutional layers that output the multiple first sub-features include the last convolutional layer of the image feature extraction model 430.

When a convolutional neural network has multiple convolutional layers, the initial convolutional layer often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network increases, the features extracted by the subsequent convolutional layers become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

In the case where the first feature includes a plurality of first sub-features respectively output by a plurality of convolutional layers in the image feature extraction model 430, the first feature can be expressed as , where G is the number of the first sub-features, /> are respectively the multiple first sub-features, and J is a positive integer greater than 1.

Thus, the second image and the audio corresponding to the second image are generated according to the first feature, and the influence of more information is taken into consideration, so that the second image and the audio corresponding to the second image are more accurate.

The sizes of the multiple first sub-features output by the multiple convolutional layers in the image feature extraction model 430 are different. The multiple first sub-features can also be understood as multi-scale features of the first image I ₀ obtained by the image feature extraction model 430 performing multi-scale encoding on the first image I _0. The image feature extraction model 430 can be understood as an encoder.

Among the multiple first sub-features, the size of the first sub-feature output by the later convolutional layers of the image feature extraction model 430 is smaller. Therefore, the image feature extraction model 430 can also be called a feature pyramid extractor.

The motion feature extraction model 420 may be a convolutional neural network including a plurality of convolutional layers. The motion feature may only include the motion sub-feature output by the last convolutional layer in the process of extracting the random motion displacement field D of the first image _I0 by the motion feature extraction model 420. Alternatively, the motion feature may include a plurality of motion sub-features respectively output by a plurality of convolutional layers in the motion feature extraction model 420. The plurality of convolutional layers that output the plurality of motion sub-features include the last convolutional layer of the motion feature extraction model 420.

The sizes of the multiple motion sub-features output by the multiple convolution layers in the motion feature extraction model 420 are different. Among the multiple motion sub-features, the size of the first sub-feature output by the later convolution layers of the motion feature extraction model 420 is smaller. Therefore, the motion feature extraction model 420 can also be called a feature pyramid extractor.

In the case where the first feature includes a plurality of first sub-features, different first sub-features correspond to different motion sub-features. The size of each first sub-feature is the same as the size of the motion sub-feature corresponding to the first sub-feature.

The adjustment model 440 can adjust each first sub-feature according to the motion sub-feature corresponding to the first sub-feature to obtain a second sub-feature corresponding to the first sub-feature. The second feature includes the second sub-feature corresponding to each first sub-feature in the plurality of first sub-features.

Alternatively, the first feature may be obtained by splicing multiple first sub-features, and the motion sub-feature may be obtained by splicing multiple motion sub-features. In other words, the splicing result of splicing multiple motion sub-features and the splicing result of splicing multiple first sub-features may be input into the adjustment model 440, so that the adjustment model 440 may output the second feature.

According to the multiple motion sub-features, a weight corresponding to each pixel p in the first image I ₀ may be determined. The weight corresponding to each pixel p may be positively correlated with the amplitude of the pixel p in the multiple motion sub-features.

For example, the weight corresponding to pixel p can be expressed as , where G is the number of the multiple motion sub-features, /> represents the amplitude of pixel p in the g-th motion sub-feature, where g is a positive integer and g≤G. The amplitudes of motion sub-features can be different. The amplitude of pixel p in the g-th motion sub-feature can be obtained by scaling the g-th motion sub-feature to the same amplitude as the first image. Scaling of the motion sub-feature can be achieved by interpolation.

The adjustment model 440 can obtain the second sub-feature according to the multiple first sub-features, the multiple motion sub-features, and the weights corresponding to the multiple pixels in the first image I ₀ .

The adjustment model 440 can adjust the first feature using an activation function. The activation function can also be called an excitation function, which is a nonlinear distortion force in the entire structure of the model. The activation function is often a fixed nonlinear transformation without parameters, which determines the range of values output by a neuron.

The adjustment model 440 makes an adjustment to the first feature using the activation function, which can also be understood as the adjustment model 440 performing a warping calculation. Warping is used to describe the process of mapping data to another data according to a certain transformation rule. By performing a geometric transformation on the data, the transformed data is aligned or matched with the reference data. Common warping calculations include affine transformation, perspective transformation, and general nonlinear transformation.

Adjusting the model 440 may perform a direct mean warp or a baseline depth warp, among others.

Alternatively, the adjustment model 440 may use the activation function softmax to process the weights corresponding to the plurality of first sub-features, the plurality of motion sub-features, and the plurality of pixels p in the first image I ₀ , respectively, to obtain the second feature. In other words, the adjustment model 440 may use the activation function softmax to process the motion feature, the weights corresponding to the plurality of pixels p in the first image I ₀ , and the first feature, respectively, to obtain the second feature. The second feature may also be called a distortion feature.

By using a distortion strategy in the image generation process, holes are avoided in the generated image, and multiple original pixels are avoided from being mapped to the same position. The multi-scale technology is used, that is, the image is generated based on the output of multiple layers in the feature extraction model. The generated image is more detailed and accurate, which is conducive to accurately predicting the motion mode and obtaining a more realistic video.

The image generation model 450 and the audio generation model 460 may be generation models. The structures of the image generation model 450 and the audio generation model 460 may be the same or different.

The image generation model 450 and the audio generation model 460 process the same second feature respectively to obtain the second image and the audio corresponding to the second image. For the second image and audio generated according to the same second feature, the same identifier can be set, and different identifiers can be set for different second images. In other words, the second image and the audio corresponding to the second image can be set with the same identifier to indicate the correspondence between the image and the audio. Thus, during the playback of a target video including multiple second images and the audio corresponding to each second image, when any second image is displayed, the audio corresponding to the second image can be determined based on the identifier of the second image, thereby avoiding a large difference between the content of the displayed image and the played audio due to inconsistent playback speeds of the image and audio during the playback of the target video.

The identification of the second image and the audio setting corresponding to the second image may also be understood as time embedding.

In the image processing system 400, the motion feature extraction model 420, the image feature extraction model 430, the image generation model 450 and the audio generation model 460 are jointly trained.

The training of the encoder 510 may be to train the initial encoder and the initial decoder using the image training data. The encoder 510 may be the initial encoder after the parameters are adjusted obtained through the training.

For the joint training of the motion feature extraction model 420, the image feature extraction model 430, the image generation model 450 and the audio generation model 460, the initial motion feature extraction model, the initial image feature extraction model, the initial image generation model and the initial audio generation model may be trained using the joint training data. The motion feature extraction model 420 may be the initial motion feature extraction model after training and parameter adjustment, the image feature extraction model 430 may be the initial image feature extraction model after training and parameter adjustment, the image generation model 450 may be the initial image generation model after training and parameter adjustment, and the audio generation model 460 may be the initial audio generation model after training and parameter adjustment.

The joint training data may include training samples and label data, the training samples include sample images and training random motion displacement fields, and the label data include label images and label audio. The initial motion feature extraction model is used to extract features from the training random motion displacement field to obtain training motion features. The initial image feature extraction model is used to extract features from the training image to obtain a first training motion feature. The adjustment model 440 is used to adjust the first training feature according to the motion feature to obtain a second training feature. The initial image generation model is used to generate a training image according to the second training feature. The initial audio generation model is used to generate training audio according to the second training feature. According to the difference between the training image and the label image, and the difference between the label audio and the training audio, the parameters of the initial motion feature extraction model, the initial image feature extraction model, the initial image generation model, and the initial audio generation model are adjusted to obtain a motion feature extraction model 420, an image feature extraction model 430, an image generation model 450, and an audio generation model 460. The difference can be represented by a loss value.

The image processing system provided in the embodiment of the present application can generate multiple images and audio in a video based on the image. Based on a single image to be processed, the generation of a dynamic video with sound can be achieved. The video can be used as a wallpaper for an electronic device. In some cases, the generated video can be played in a seamless loop.

The image processing system can generate a video based on the target subject indicated by the user and the target motion trend of the target subject, so as to enable the user to edit the motion, improve the user's participation and user experience. The video generated based on the user's instructions can be understood as an interactive animation.

The training video used in the training process can be acquired by a camera. Therefore, the image processing system trained according to the training video can generate a video with high authenticity and can realize dynamic simulation. It should be understood that the target motion trend indicated by the user can be understood as an external force applied to the target body.

The following describes the training method of the image processing system used in the image processing method shown in FIG. 3 in conjunction with FIG. 9 .

Fig. 9 is a schematic flow chart of a training method for an image processing system provided in an embodiment of the present application. The method shown in Fig. 9 includes steps S910 to S930.

Step S910, obtaining training samples, labeled images, and labeled audio.

The training sample may include a sample image in a training video, the label image may be an image following the sample image in the training video, and the label audio may be the audio in the training video when the label image in the training video is displayed.

Step S920: Process the training samples using the initial image processing system to obtain training images and training audio.

Step S930, adjusting parameters of the initial image processing system according to the first difference between the training image and the label image, and the second difference between the training audio and the label audio, and processing the adjusted initial image.

The system is a trained image processing system.

The first difference and the second difference may both be represented by a loss value. For example, the first difference may be represented by a perceptual loss. The perceptual loss may be a visual geometry group (VGG) perceptual loss, or a learned perceptual image patch similarity (LPIPS), etc.

In the process of training the neural network model, because the output of the neural network model is as close as possible to the value that we really want to predict, we can compare the predicted value of the current network with the target value that we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the neural network model). For example, if the model's predicted value is high, adjust the weight vector to make it predict lower, and keep adjusting until the neural network model can predict the target value that we really want or a value that is very close to the target value that we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value of the loss function, that is, the loss value (loss), the greater the difference, so the training of the neural network model becomes a process of minimizing this loss as much as possible.

The back propagation (BP) algorithm can be used to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial neural network model are updated by backpropagating the error loss information, so that the error loss converges. The backpropagation algorithm is a backpropagation movement dominated by error loss, which aims to obtain the optimal parameters of the neural network model, such as the weight matrix.

Optionally, the training sample may further include training time information, where the training time information is used to indicate a training time interval, and the label image and the label audio may be images in the training video that follow the sample image and whose time interval with the sample image is the training time interval.

Optionally, the training sample may further include training subject area information, which represents the training subject area. The training subject area in the sample image records the training target subject, and the content recorded in other areas outside the training subject area in the label image is the same as the content recorded in the other areas in the sample image.

The training target subject may be a subject moving in the training video. For example, the training target subject may be a subject moving at the beginning of the training video.

Optionally, when the sample image is the first frame image in a training video, the training sample includes training subject region information.

It should be understood that the number of training samples of the image processing system is multiple. In step S910, each training sample, as well as the label image corresponding to the training sample and the label audio corresponding to the training sample can be obtained.

Optionally, the training sample may further include training motion information. When the training sample includes the training motion information, compared to the sample image, the training target subject in the label image moves according to the training motion trend represented by the training motion information. In other words, the motion of the training target subject conforms to the training motion trend represented by the training motion information.

Optionally, when the sample image is the first frame image in a training video, the training sample includes training motion information.

Optionally, the initial image processing system includes an initial feature prediction model, an initial image generation model and an initial audio generation model.

The initial feature prediction model is used to process the training samples to obtain training prediction features.

The initial image generation model is used to process the training prediction features to obtain the training image.

The initial audio generation model is used to process the predicted features to obtain training audio.

The initial feature prediction model after parameter adjustment is the feature prediction model in the image processing system, the initial image generation model after parameter adjustment is the image generation model in the image processing system, and the initial audio generation model after parameter adjustment is the audio generation model in the image processing system.

Optionally, the initial feature prediction model includes a motion displacement field prediction model, an initial motion feature extraction model, an initial image feature extraction model and an adjustment model.

The motion displacement field prediction model is used to process the training samples to obtain the training motion displacement field, which represents the displacement of multiple training pixels in the sample image at the second training moment corresponding to the label image relative to the first training moment corresponding to the sample image. The second training moment corresponding to the label image and the first training moment corresponding to the sample image can be understood as the moments of the label image and the sample image in the training video, respectively.

The initial motion feature extraction model is used to extract features from the motion displacement field to obtain training motion features.

The initial image feature extraction model is used to extract features from sample images to obtain training image features.

The adjustment model is used to adjust the training image features according to the training motion features to obtain the training prediction features.

The initial motion feature extraction model after parameter adjustment is the motion feature extraction model in the image processing system, and the initial image feature extraction model after parameter adjustment is the image feature extraction model in the image processing system.

The motion displacement field prediction model may be a pre-trained neural network model.

Alternatively, the initial feature prediction model may include an initial motion feature extraction model, an initial image feature extraction model, and an adjustment model.

The training motion displacement field can be determined by processing the training samples by feature tracking, optical flow extraction or particle video. The training motion displacement field obtained by feature tracking, optical flow extraction or particle video processing can also be called an optical flow field.

Optionally, the motion displacement field prediction model may include a first transformation module, a motion texture prediction model and a second transformation module.

The first transformation module is used to perform Fourier transformation on the sample image to obtain frequency domain data of the training image.

The motion texture prediction model is used to process the frequency domain data of the training image to generate the training motion texture.

The second transformation module is used to perform inverse Fourier transformation on the training motion texture to obtain a training motion displacement field.

Training motion texture can be understood as training the frequency domain representation of motion displacement field. Through inverse Fourier transform, frequency domain data can be converted into time domain data.

In the motion displacement field prediction model, the motion displacement field prediction model may be obtained through pre-training.

Optionally, the motion texture prediction model may include a compression model and a motion texture generation model. That is, the motion texture prediction model may be an LDM model.

The compression model is used to compress the training image frequency domain data to obtain training image compressed data.

The motion texture generation model is used to process the training image compression data to generate training motion texture.

In the case where the training sample includes training subject region information, the motion texture generation model is used to process the training image compression data and the training subject region information to generate training motion texture.

In the case where the training sample includes training subject region information and training motion information, the motion texture generation model is used to process the training image compression data, the training subject region information and the training motion information to generate training motion texture.

The pre-training of the motion displacement field prediction model may include a first pre-training of the compression model and a second pre-training of the motion texture generation model.

The first pre-training can be performed using the first pre-training data to obtain a compression model. During the first pre-training process, the first pre-training data can be compressed using the initial compression model to obtain pre-training compressed data, and the pre-training compressed data can be decompressed using the initial decompression model to obtain pre-training decompressed data. The amount of data of the pre-training compressed data is less than the amount of data of the first pre-training data. During the first pre-training process, the parameters of the initial compression model and the initial decompression model can be adjusted according to the difference between the pre-training decompressed data and the first pre-training data, and the initial compression model after the parameter adjustment can be the compression model in the motion texture prediction model.

The first pre-training data may be an image, or may be frequency domain data obtained by performing Fourier transform on the image.

It should be understood that in the first pre-training process, the number of the first pre-training data may be one or more.

The pre-trained data of the second pre-training may include pre-trained image frequency domain data and label motion texture. The pre-trained image frequency domain data may be obtained by Fourier transforming the pre-trained image in the pre-trained video. The label motion texture may be obtained by Fourier transforming the pre-trained displacement field, that is, the label motion texture may be understood as a frequency domain representation of the pre-trained displacement field. The pre-trained displacement field represents the displacement of multiple pixels in the pre-trained image in the pre-trained video in at least one image after the pre-trained image in the pre-trained video. The pre-trained displacement field may be obtained by processing the pre-trained video in the manner of a particle point video.

It should be understood that the number of pre-training data used in the second training process can be one or more.

In the second pre-training process, the pre-training image frequency domain data can be compressed using the compression model to obtain pre-training image compressed data, and the pre-training image compressed data can be processed using the initial motion texture generation model to obtain the pre-training motion texture. According to the difference between the pre-training motion texture and the label motion texture, the parameters of the initial motion texture generation model are adjusted, and the initial motion texture generation model after parameter adjustment can be the motion texture generation model in the motion texture prediction model.

The training data may or may not include training subject region information. In the second pre-training process of the motion texture generation model, the influence of the subject region, or the subject region and the motion trend is considered.

The pre-training data of the second pre-training may include pre-training subject area information. The pre-training subject area information indicates a pre-training subject area where the pre-training subject is located in the pre-training image. In the pre-training displacement field represented by the label motion texture, the displacement of pixels outside the pre-training subject area is 0.

The pre-trained data of the second pre-training may also include pre-trained subject area information and pre-trained motion information. The pre-trained motion information indicates the pre-trained motion trend of the pre-trained subject. In the label motion texture, the position of the pixel in the pre-trained subject area indicates that the motion of the pre-trained subject conforms to the pre-trained motion trend.

Optionally, the motion texture generation model may include a diffusion model and a decompression model. The decompression model may be an initial decompression model adjusted through the first pre-training.

The motion texture generation model is used to process the compressed data of the training image and perform multiple denoising processes on the compressed frequency domain noise data to obtain the training compressed motion texture.

The compressed frequency domain noise data may be obtained by compressing the frequency domain noise data with a compression model, or may be preset.

The decompression model can be used to decompress the training compressed motion texture to obtain the training motion texture.

The initial motion texture generation model may include a decompression model and an initial diffusion model. In the second pre-training process of the motion texture generation model, the frequency domain noise data may be compressed using the compression model to obtain compressed frequency domain noise data. The initial motion texture generation model may be used to perform multiple denoising processes on the compressed frequency domain noise data according to the compressed frequency domain noise data to obtain a pre-trained compressed motion texture. The decompression model may be used to decompress the pre-trained compressed motion texture to obtain the pre-trained motion texture.

The image processing system trained by the method provided in the embodiment of the present application can be applied to the image processing method shown in FIG. 3 .

The above describes in detail the image processing method of the embodiment of the present application in conjunction with Figures 3 to 9, and the device embodiment of the present application will be described in detail below in conjunction with Figures 10 and 11. It should be understood that the image processing device in the embodiment of the present application can perform the various image processing methods of the aforementioned embodiments of the present application, that is, the specific working processes of the following various products can refer to the corresponding processes in the aforementioned method embodiments.

FIG. 10 is a schematic structural diagram of a system architecture provided in an embodiment of the present application.

As shown in the system architecture 1100, the data acquisition device 1160 is used to collect training data. In the embodiment of the present application, the training data includes: training samples, label images, and label audio. The training samples may include sample images in the training video. The label image may be an image after the sample image in the training video, and the label audio may be the audio in the training video when the label image in the training video is displayed.

The data collection device 1160 may also be used to collect first pre-training data and second pre-training data.

In the embodiment of the present application, the first pre-training data is an image, and may also be frequency domain data obtained by performing Fourier transform on the image.

In the embodiment of the present application, the second pre-training data may include pre-training image frequency domain data and label motion texture. The second pre-training data may also include pre-training subject area information. The second pre-training data may also include pre-training motion information.

The training data is stored in the database 1130 , and the training device 1120 obtains the target model/rule 1101 through training based on the training data maintained in the database 1130 .

In the embodiment provided in the present application, the image processing system is obtained through training. A detailed description of how the training device 1120 obtains the target model/rule 1101 based on the training data can be found in the description of FIG9 . That is, the training device 1120 can be used to execute the training method of the image processing system shown in FIG9 .

The target model/rule 1101 may be the image processing system 400 shown in FIG4. The target model/rule 1101 may be used to implement the image processing method shown in FIG3 provided in the embodiment of the present application, that is, the image to be processed is input into the target model/rule 1101, and the target video may be obtained. The target model/rule 1101 in the embodiment of the present application may specifically be an image processing system.

It should be noted that, in actual applications, the training data maintained in the database 1130 may not all come from the data acquisition device 1160, but may also be received from other devices. It should also be noted that the training device 1120 may not train the target model/rule 1101 entirely based on the training data maintained by the database 1130, but may also obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

The target model/rule 1101 obtained by training with the training device 1120 can be applied to different systems or devices, such as the execution device 1110 shown in FIG. 10 , which can be a terminal such as a mobile phone terminal, a tablet computer, a laptop computer, AR/VR, a vehicle terminal, etc., or a server or a cloud, etc. In FIG. 10 , the execution device 1110 is configured with an I/O interface 1112 for data interaction with an external device, and the user can input data to the I/O interface 1112 through the client device 1140. The input data can include the image to be processed in the embodiment of the present application, and can also include the target subject indicated by the user in the image to be processed, the target motion trend of the target subject indicated by the user, etc.

The preprocessing module 1113 and the preprocessing module 1114 are used to perform preprocessing according to the input data (such as the image to be processed) received by the I/O interface 1112 .

In the embodiment of the present application, the preprocessing module 1113 and the preprocessing module 1114 may be omitted (or only one of the preprocessing modules may be provided), and the calculation module 1111 may be directly used to process the input data.

When the execution device 1110 preprocesses the input data, or when the computing module 1111 of the execution device 1110 performs calculations and other related processing, the execution device 1110 can call the data, code, etc. in the data storage system 1150 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing into the data storage system 1150.

Finally, the I/O interface 1112 returns the processing result, such as the deblurred original image file obtained above, to the client device 1140 for providing to the user.

It is worth noting that the training device 1120 can generate corresponding target models/rules 1101 based on different training data for different goals or different tasks. The corresponding target models/rules 1101 can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

In the case shown in FIG. 10 , the user can manually give input data, and the manual giving can be operated through the interface provided by the I/O interface 1112. In another case, the client device 1140 can automatically send input data to the I/O interface 1112. If the client device 1140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 1140. The user can view the results output by the execution device 1110 on the client device 1140, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 1140 can also be used as a data acquisition terminal to collect the input data of the input I/O interface 1112 and the output results of the output I/O interface 1112 as shown in the figure as new sample data, and store them in the database 1130. Of course, it is also possible not to collect through the client device 1140, but the I/O interface 1112 directly stores the input data of the input I/O interface 1112 and the output results of the output I/O interface 1112 as new sample data in the database 1130.

11 to 14 illustrate application scenarios of the image generation method provided in the embodiments of the present application.

FIG. 11 (a) shows a graphical user interface (GUI) of an electronic device, which may be a client device 1140, and the GUI is a desktop 1010 of the electronic device. When a user clicks on an album application (APP) icon 1011 on the desktop 1010 to start an album operation, the electronic device may start the album application and display another GUI as shown in FIG. 11 (b). The GUI shown in FIG. 11 (b) may be referred to as a first album interface 1020. The album interface 1020 may include a plurality of thumbnails.

When it is detected that the user clicks on any thumbnail on the album interface 1020, the electronic device may display the GUI shown in (a) of FIG. 12, which may be referred to as the second album interface 1030. The second album interface 1030 may include a video generation icon 1031 and an image to be processed 1032 corresponding to the thumbnail clicked by the user. When it is detected that the user clicks on the video generation icon 1031, a video editing interface 1040 as shown in (b) of FIG. 12 may be displayed. The video editing interface 1040 includes the image to be processed 1032, and includes the text message "Please select a target and indicate the movement of the target".

The user can touch the target subject with his finger and slide it on the screen. The subject where the user touches is the target subject.

The direction in which the user's finger slides may be the target subject's starting direction of motion. Alternatively, the position where the user's finger stops sliding may be the target subject's position at the maximum motion amplitude. Alternatively, the speed at which the user's finger slides may be the target subject's starting speed or maximum speed. The target motion trend of the target subject may include one or more of the target subject's starting direction of motion, the position at the maximum motion amplitude, the starting speed or the maximum speed, etc.

When the sliding of the user's finger is detected to end, the electronic device can execute the method shown in FIG3 to generate the target video. After the target video is generated, the electronic device can display the video interface 1050 shown in FIG13. The video interface 1050 includes a play icon 1051 of the target video. When the user clicks the play icon 1051, the electronic device can play the target video.

In other embodiments, the user may also input the target movement trend of the target subject by voice or writing.

Fig. 14 shows a graphical user interface (GUI) of an electronic device, where the GUI is a lock screen interface 1410 of the electronic device. The wallpaper image displayed on the lock screen interface 1410 can be understood as an image to be processed.

When it is detected that the user clicks on any subject in the image to be processed, the electronic device may take the subject clicked by the user as the target subject, execute the method shown in FIG. 3 , generate a target video, and display the target video.

Alternatively, when detecting that the user's sliding on the display screen ends, the electronic device can use the subject at the starting position of the sliding as the target subject, generate a target video according to the target motion trend represented by the sliding operation, and display the target video.

Therefore, through the image processing method provided by the present application, a personalized dynamic wallpaper application can be generated, and audio corresponding to each image in the dynamic wallpaper can be generated and played simultaneously.

The target subject moving in the dynamic wallpaper may be instructed by the user, and the movement of the target subject may be performed according to the instruction of the user.

In other words, users do not need too much professional knowledge. They only need to select a beautiful picture from the gallery as the wallpaper, click on the moving subject they are interested in in the picture, and drag the subject appropriately to see the desired dynamic sound wallpaper.

It is worth noting that FIG10 is only a schematic diagram of a system architecture provided by an embodiment of the present invention, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in FIG10, the data storage system 1150 is an external memory relative to the execution device 1110. In other cases, the data storage system 1150 can also be placed in the execution device 1110.

As shown in Figure 10, the target model/rule 1101 is obtained by training according to the training device 1120. The target model/rule 1101 can be an image processing system in the embodiment of the present application. Specifically, the image processing system provided in the embodiment of the present application may include a feature prediction model, an image generation model, and an audio generation model. The feature prediction model includes a motion displacement field prediction model, a motion feature extraction model, an image feature extraction model, and an adjustment model. The image generation model and the audio generation model, as well as the motion displacement field prediction model, the motion feature extraction model, and the image feature extraction model in the feature prediction model, can all be convolutional neural networks.

As mentioned in the previous basic concept introduction, convolutional neural network is a deep neural network with convolution structure, which is a deep learning architecture. Deep learning architecture refers to multiple levels of learning at different abstract levels through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, in which each neuron can respond to the data input into it.

The motion displacement field prediction model may be obtained by pre-training. The training device 1120 may train the motion displacement field prediction model based on the first pre-training data and the second pre-training data maintained in the database 1130 .

FIG. 15 is a schematic diagram of an image processing device provided in an embodiment of the present application.

The image processing device 1500 includes an acquisition unit 1510 and a processing unit 1520 .

In some embodiments, the image processing apparatus 1500 may execute the image processing method shown in FIG3 . The image processing apparatus 1500 may be the execution device 1110 .

The acquisition unit 1510 is used to acquire the image to be processed.

The processing unit 1520 is used to generate N target images in the target video and N target audios in the target video according to the image to be processed, wherein the N target images correspond one-to-one to the N target audios, N is an integer greater than 1, and each target audio is used to be played when the target image corresponding to the target audio is displayed.

Optionally, the processing unit 1520 is used to use an image processing system to process multiple first images in sequence to obtain at least one second image corresponding to each first image and the target audio corresponding to each second image, the multiple target images include the at least one second image corresponding to each first image, the at least one second image corresponding to each first image is an image after the first image in the target video, the first first image processed by the image processing system is the image to be processed, the i-th first image processed by the image processing system is an image in the at least one second image corresponding to the i-1-th first image processed by the image processing system, i is a positive integer greater than 1, and the image processing system includes a trained neural network model.

Optionally, the acquisition unit 1510 is further configured to acquire a target subject indicated by a user in the image to be processed.

In the case where the first image processed by the image processing system is the image to be processed, the image processing system is used to process the first image and target subject area information to obtain at least one second image corresponding to the first image, and the target audio corresponding to each second image, the target subject area information represents a target subject area, the target subject is recorded in the target subject area in the first image, and the contents recorded in other areas outside the target subject area in each second image are the same as the contents recorded in other areas in the first image.

Optionally, when the first image processed by the image processing system is the image to be processed, the image processing system is used to process the first image and the target subject area information to obtain at least one second image corresponding to the first image, and the target audio corresponding to each second image.

Optionally, the acquisition unit 1510 is further configured to acquire a target motion trend of the target subject indicated by the user.

In the case where the first image processed by the image processing system is the image to be processed, the image processing system is used to process the first image, target subject area information and motion information to obtain the target image corresponding to each second moment in the at least one second moment, and the target audio corresponding to each second moment, wherein the target subject in at least one target image corresponding to the at least one second moment moves according to the target motion trend represented by the motion information.

Optionally, the image processing system includes a feature prediction model, an image generation model and an audio generation model.

The feature prediction model is used to process the first image to obtain at least one prediction feature, different prediction features correspond to different second moments, and each second moment is a moment after the first moment corresponding to the first image in the target video.

The image generation model is used to process the at least one prediction feature respectively to obtain a second image corresponding to each second moment.

The audio generation model is used to process the at least one predicted feature respectively to obtain the target audio corresponding to each second moment.

Optionally, the feature prediction model includes a motion displacement field prediction model, a motion feature extraction model, an image feature extraction model and an adjustment model.

The motion displacement field prediction model is used to process the first image to obtain a motion displacement field corresponding to each second moment, and the motion displacement field corresponding to each second moment represents the displacement of multiple pixels in the first image at the second moment relative to the first moment.

The motion feature extraction model is used to perform feature extraction on the at least one motion displacement field respectively to obtain the motion feature corresponding to each second moment.

The image feature extraction model is used to extract features from the first image to obtain image features.

The adjustment model is used to adjust the image feature according to the motion feature corresponding to each second moment, so as to obtain the prediction feature corresponding to the second moment.

Optionally, the acquisition unit 1510 is further configured to acquire a target subject indicated by a user in the image to be processed.

The motion displacement field prediction model is used to process the first image and the target subject area information to obtain the motion displacement field corresponding to each second moment. The target subject area information represents the target subject area, and the target subject is recorded in the target subject area in the image to be processed.

The motion displacement field corresponding to each second moment indicates that the displacement of pixels outside the region is 0, and the pixels outside the region are located outside the target body region in the first image.

Optionally, the acquisition unit 1510 is further configured to acquire a target motion trend of the target subject indicated by a user.

In the case where the first image processed by the image processing system is the image to be processed, the motion displacement field prediction model is used to process the first image, the target subject area information and the motion information to obtain the motion displacement field corresponding to each second moment, and the target displacement of the target subject pixel represented by the motion displacement field corresponding to each second moment at the second moment relative to the target at the first moment conforms to the target motion trend represented by the motion information, and the target subject pixel is a pixel located on the target subject in the first image.

In some other embodiments, the image processing apparatus 1500 may execute the training method of the image processing system shown in FIG9 . The image processing apparatus 1500 may be the training device 1120 .

The acquisition unit 1510 is used to acquire training samples, label images, and label audio.

The processing unit 1520 is used to process the training samples using the initial image processing system to obtain training images and training audio.

The processing unit 1520 is also used to adjust the parameters of the initial image processing system according to the first difference between the training image and the label image, and the second difference between the training audio and the label audio, and the adjusted initial image processing system is the trained image processing system.

It should be noted that the above-mentioned image processing device 1500 is embodied in the form of a functional unit. The term "unit" here can be implemented in the form of software and/or hardware, and is not specifically limited to this.

For example, a "unit" may be a software program, a hardware circuit, or a combination of the two that implements the above functions. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a dedicated processor, or a group processor, etc.) and a memory for executing one or more software or firmware programs, a combined logic circuit, and/or other suitable components that support the described functions.

Therefore, the units of each example described in the embodiments of the present application can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

The present application also provides a chip, the chip comprising a data interface and one or more processors. When the one or more processors execute instructions, the one or more processors read instructions stored in a memory through the data interface to implement the image processing method and/or the training method of an image processing system described in the above method embodiment.

The one or more processors may be general-purpose processors or special-purpose processors. For example, the one or more processors may be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices such as discrete gates, transistor logic devices, or discrete hardware components.

The chip may be used as a component of a terminal device or other electronic devices. For example, the chip may be located in the electronic device 100 .

The processor and the memory may be provided separately or integrated together. For example, the processor and the memory may be integrated on a system on chip (SOC) of a terminal device. That is, the chip may also include a memory.

The memory may store a program, which may be executed by the processor to generate instructions, so that the processor executes the image processing method and/or the training method of the image processing system described in the above method embodiment according to the instructions.

Optionally, data may be stored in the memory. Optionally, the processor may read data stored in the memory, and the data may be stored at the same storage address as the program, or may be stored at a different storage address than the program.

Exemplarily, the memory can be used to store relevant programs of the image processing method provided in the embodiments of the present application, and the processor can be used to call relevant programs of the image processing method stored in the memory to implement the image processing method in the embodiments of the present application.

For example, the processor can be used to obtain an image to be processed; based on the image to be processed, generate N target images in a target video and N target audios in the target video, the N target images correspond one-to-one to the N target audios, N is an integer greater than 1, and each target audio is used to play when the target image corresponding to the target audio is displayed.

Exemplarily, the memory can be used to store relevant programs of the training method of the image processing system provided in the embodiments of the present application, and the processor can be used to call relevant programs of the training method of the image processing system stored in the memory to implement the training method of the image processing system in the embodiments of the present application.

For example, the processor can be used to obtain training samples and labeled images and labeled audio; use the initial image processing system to process the training samples to obtain training images and training audio; adjust the parameters of the initial image processing system based on the first difference between the training image and the labeled image, and the second difference between the training audio and the labeled audio, and the adjusted initial image processing system is the trained image processing system.

The chip can be arranged in an electronic device.

The present application also provides a computer program product, which, when executed by a processor, implements the image processing method and/or the training method of an image processing system described in any method embodiment of the present application.

The computer program product may be stored in a memory, for example, a program, which is finally converted into an executable target file that can be executed by a processor after preprocessing, compiling, assembling and linking.

The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a computer, the image processing method and/or the training method of the image processing system described in any method embodiment of the present application is implemented. The computer program can be a high-level language program or an executable target program.

The computer-readable storage medium is, for example, a memory. The memory may be a volatile memory or a nonvolatile memory, or the memory may include both a volatile memory and a nonvolatile memory. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

The embodiments of this application may involve the use of user data. In actual applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, subject to the requirements of applicable laws and regulations of the country where the user is located (for example, with the user's explicit consent, effective notification to the user, etc.).

In the description of this application, the terms "first", "second", etc. are used only for descriptive purposes and should not be understood as indicating or implying relative importance, or a specific order or sequence. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

In this application, "at least one" means one or more, and "more" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can mean: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic; for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims (12)

1. An image processing method, the method comprising:

Acquiring an image to be processed;

Generating N target images in a target video and N target audios in the target video according to the image to be processed, wherein the N target images are in one-to-one correspondence with the N target audios, N is an integer greater than 1, and each target audio is used for playing under the condition that the target image corresponding to the target audio is displayed.

2. The method of claim 1, wherein generating N target images in a target video and N target audio in the target video from the image to be processed comprises:

And processing the plurality of first images sequentially by using an image processing system to obtain at least one second image corresponding to each first image and the target audio corresponding to each second image, wherein the N target images comprise at least one second image corresponding to each first image, the at least one second image corresponding to each first image is an image after the first image in the target video, the first image processed by the image processing system is the image to be processed, the ith first image processed by the image processing system is an image in at least one second image corresponding to the ith-1 first image processed by the image processing system, i is a positive integer greater than 1, and the image processing system comprises a neural network model obtained through training.

3. The method according to claim 2, wherein the method further comprises: acquiring a target main body indicated by a user in the image to be processed;

And when the first image processed by the image processing system is the image to be processed, the image processing system is used for processing the first image and target main body area information to obtain at least one second image corresponding to the first image and the target audio corresponding to each second image, the target main body area information represents a target main body area, the target main body is recorded in the target main body area in the first image, and the content recorded in other areas except the target main body area in each second image is the same as the content recorded in the other areas in the first image.

4. A method according to claim 3, wherein, in the case where the first image processed by the image processing system is the image to be processed, the image processing system is configured to process the first image and target subject area information to obtain at least one second image corresponding to the first image, and the target audio corresponding to each second image.

5. The method according to claim 3 or 4, characterized in that the method further comprises: acquiring a target movement trend of the target main body indicated by the user;

And under the condition that the first image processed by the image processing system is the image to be processed, the image processing system is used for processing the first image, target main body area information and motion information to obtain the target image corresponding to each second moment in the at least one second moment and the target audio corresponding to each second moment, and the target main body in at least one target image corresponding to the at least one second moment moves according to the target motion trend represented by the motion information.

6. The method of claim 2, wherein the image processing system comprises a feature prediction model, an image generation model, and an audio generation model;

The feature prediction model is used for processing the first image to obtain at least one prediction feature, and second moments corresponding to different prediction features are different, wherein each second moment is a moment after a first moment corresponding to the first image in the target video;

The image generation model is used for respectively processing the at least one prediction feature to obtain a second image corresponding to each second moment;

the audio generation model is used for respectively processing the at least one prediction feature to obtain target audio corresponding to each second moment.

7. The method of claim 6, wherein the feature prediction model comprises a motion displacement field prediction model, a motion feature extraction model, an image feature extraction model, and an adjustment model;

The motion displacement field prediction model is used for processing the first image to obtain a motion displacement field corresponding to each second moment, and the motion displacement field corresponding to each second moment represents the displacement of a plurality of pixels in the first image relative to the first moment at the second moment;

The motion feature extraction model is used for extracting features of the at least one motion displacement field respectively to obtain motion features corresponding to each second moment;

The image feature extraction model is used for extracting features of the first image to obtain image features;

The adjustment model is used for adjusting the image characteristics according to the motion characteristics corresponding to each second moment so as to obtain the prediction characteristics corresponding to the second moment.

8. The method of claim 7, wherein the method further comprises: acquiring a target main body indicated by a user in the image to be processed;

The motion displacement field prediction model is used for processing the first image and target main body area information to obtain the motion displacement field corresponding to each second moment, the target main body area information represents a target main body area, and the target main body is recorded in the target main body area in the image to be processed;

And the motion displacement field corresponding to each second moment represents that the displacement of the pixel outside the area is 0, and the pixel outside the area is positioned outside the target main body area in the first image.

9. The method of claim 8, wherein the method further comprises: acquiring a target movement trend of the target main body indicated by a user;

And under the condition that the first image processed by the image processing system is the image to be processed, the motion displacement field prediction model is used for processing the first image, the target main body area information and the motion information to obtain the motion displacement field corresponding to each second moment, the target motion trend represented by the motion displacement field corresponding to each second moment is met by the target main body pixel corresponding to each second moment, and the target main body pixel is the pixel positioned on the target main body in the first image.

10. An electronic device, the electronic device comprising: one or more processors, and memory;

The memory being coupled to the one or more processors, the memory being for storing computer program code comprising computer instructions that the one or more processors invoke to cause the electronic device to perform the method of any of claims 1-9.

11. A chip system for application to an electronic device, the chip system comprising one or more processors to invoke computer instructions to cause the electronic device to perform the method of any of claims 1 to 9.

12. A computer readable storage medium comprising instructions that, when run on an electronic device, cause the electronic device to perform the method of any one of claims 1 to 9.