CN117593216A

CN117593216A - Training method of image restoration model, image restoration method and related device

Info

Publication number: CN117593216A
Application number: CN202311210523.4A
Authority: CN
Inventors: 宁本德
Original assignee: Beijing QIYI Century Science and Technology Co Ltd
Current assignee: Beijing QIYI Century Science and Technology Co Ltd
Priority date: 2023-09-19
Filing date: 2023-09-19
Publication date: 2024-02-23

Abstract

The invention provides a training method of an image restoration model, an image restoration method and a related device, and relates to the technical field of image processing. The method comprises the following steps: acquiring a sample image sequence to be repaired; inputting a sample image sequence into an image restoration model, wherein the image restoration model determines a target sample image containing a sample object and a reference sample image containing the sample object in the sample image sequence, and restores the sample object in the target sample image by using the sample object in the reference sample image to obtain a restored target sample image; the definition of the reference sample image is higher than that of the target sample image; calculating a loss value based on the repaired target sample image and the true value of the target sample image; and when the image restoration model is not converged, adjusting network parameters of the image restoration model, and returning to acquire a sample image sequence to be restored. The image restoration model trained by the scheme can restore objects losing a large amount of details in the video, and improves the watching experience of audiences.

Description

Training method of image restoration model, image restoration method and related device

Technical Field

The present invention relates to the field of image processing technologies, and in particular, to a training method for an image restoration model, an image restoration method, and a related apparatus.

Background

If the definition of the video containing any object is not high, the audience cannot see the clear object, and the viewing experience of the audience is affected.

In the related art, a digital image restoration technology is utilized to restore an object in a video, so that the definition of the object is improved as much as possible.

However, if many details of the object in the video are lost, the object repaired by applying the related technology may not conform to the performance of the real object in the video, and may not bring better viewing experience for the viewer.

Therefore, how to repair objects losing a great deal of details in a video so as to improve the viewing experience of viewers is a problem to be solved.

Disclosure of Invention

The embodiment of the invention aims to provide a training method of an image restoration model, an image restoration method and a related device, so as to restore objects losing a great deal of details in videos and improve the viewing experience of audiences. The specific technical scheme is as follows:

in a first aspect of the embodiment of the present invention, there is first provided a training method of an image restoration model, the training method including:

Acquiring a sample image sequence to be repaired;

inputting a sample image sequence to be repaired into an image repair model to be trained, so that the image repair model determines a target sample image containing a sample object and a reference sample image containing the sample object in the sample image sequence to be repaired, and repairing the sample object in the target sample image by using the sample object in the reference sample image to obtain a repaired target sample image; wherein the sharpness of the sample object in the reference sample image is higher than the sharpness of the sample object in the target sample image;

calculating a loss value based on the repaired target sample image and a true value of the target sample image;

and when judging that the image restoration model is not converged based on the loss value, adjusting network parameters of the image restoration model, and returning to the step of acquiring the sample image sequence to be restored.

Optionally, the image restoration model restores the sample object in the target sample image by using the sample object in the reference sample image to obtain a restored target sample image, including:

Performing feature extraction processing on the image content of the sample object in the reference sample image to obtain reference feature data, and performing feature extraction processing on the image content of the sample object in the target sample image to obtain feature data to be repaired;

determining joint feature data corresponding to the target sample image based on the extracted reference feature content and the feature content to be repaired;

performing image content restoration processing based on the joint feature data corresponding to the target sample image to obtain image content of the target sample image aiming at the sample object;

and repairing the image content of the sample object in the target sample image based on the obtained image content to obtain a repaired target sample image.

Optionally, the determining, based on the extracted reference feature content and the feature content to be repaired, joint feature data corresponding to the target sample image includes:

and carrying out weighting treatment and splicing treatment on the extracted reference characteristic content and the characteristic content to be repaired to obtain joint characteristic data corresponding to the target sample image.

Optionally, the target sample image is: and the true value of the target sample image is subjected to the sharpness reducing treatment to obtain the image.

In a second aspect of the embodiment of the present invention, there is also provided an image restoration method, including:

acquiring an image sequence to be repaired;

inputting an image sequence to be repaired into an image repairing model obtained by training in advance, so that the image repairing model determines an image to be repaired containing a target object and a reference image containing the target object in the image sequence to be repaired, and repairing the target object in the image to be repaired by utilizing the target object in the reference image to obtain a repaired image;

the definition of the target object in the reference image is higher than that of the target object in the image to be repaired;

the image restoration model is as follows: the training method based on the image restoration model trains the obtained model.

Alternatively to this, the method may comprise,

the image to be repaired and the reference image are determined according to the following modes:

performing pre-classification operation on each image containing a target object in the image sequence to be repaired to obtain an image to be repaired and the reference image; wherein the pre-sorting operation comprises: identification and definition detection of the identity.

Optionally, the pre-classifying the images including the target object in the image sequence to be repaired to obtain the image to be repaired and the reference image includes:

Determining the definition of each frame image in the image sequence to be repaired and the identity of the contained object;

and determining an image to be repaired and a reference image in the image sequence to be repaired based on the determined definition and the identity.

Optionally, the determining, based on the determined sharpness and the identity, the image to be repaired and the reference image in the image sequence to be repaired includes:

classifying each frame of image containing the object in the image sequence to be repaired according to the determined definition to obtain each first type of image and each second type of image; wherein the sharpness of the object in each second type of image is higher than the sharpness of the object in each first type of image;

and selecting images to be repaired from the first type images, and determining second type images with the same identity with the images to be repaired from the second type images to obtain reference images.

Optionally, the image restoration model restores the target object in the image to be restored by using the target object in the reference image to obtain a restored image, which includes:

Performing feature extraction processing on the image content of the target object in the reference image to obtain reference feature data, and performing feature extraction processing on the image content of the target object in the image to be repaired to obtain feature data to be repaired;

determining joint feature data corresponding to the image to be repaired based on the extracted reference feature content and the feature content to be repaired;

performing image content restoration processing based on the joint feature data corresponding to the image to be restored to obtain image content of the image to be restored aiming at the target object;

and repairing the image content of the target object in the image to be repaired based on the obtained image content to obtain a repaired image.

In a third aspect of the embodiment of the present invention, there is further provided a training apparatus for an image restoration model, the training apparatus including:

the first acquisition module is used for acquiring a sample image sequence to be repaired;

the first input module is used for inputting a sample image sequence to be repaired into an image repair model to be trained, so that the image repair model determines a target sample image containing a sample object and a reference sample image containing the sample object in the sample image sequence to be repaired, and repairs the sample object in the target sample image by utilizing the sample object in the reference sample image to obtain a repaired target sample image; wherein the sharpness of the sample object in the reference sample image is higher than the sharpness of the sample object in the target sample image;

The calculation module is used for calculating a loss value based on the repaired target sample image and the true value of the target sample image;

and the adjusting module is used for adjusting network parameters of the image restoration model when judging that the image restoration model is not converged based on the loss value, and returning to the step of acquiring the sample image sequence to be restored.

In a fourth aspect of the embodiment of the present invention, there is also provided an image restoration apparatus, including:

the second acquisition module is used for acquiring an image sequence to be repaired;

the second input module is used for inputting an image sequence to be repaired into an image repairing model obtained through pre-training, so that the image repairing model determines an image to be repaired containing a target object and a reference image containing the target object in the image sequence to be repaired, and the target object in the image to be repaired is repaired by utilizing the target object in the reference image, so that a repaired image is obtained;

the image restoration model is as follows: the training device based on the image restoration model trains the obtained model.

In yet another aspect of the embodiment of the present invention, there is also provided an electronic device including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus;

a memory for storing a computer program;

and the processor is used for realizing the training method and the image restoration method of the image restoration model when executing the program stored in the memory.

In still another aspect of the embodiments of the present invention, there is further provided a computer readable storage medium having a computer program stored therein, which when executed by a processor, implements a training method and an image restoration method of the image restoration model.

According to the training method of the image restoration model, a sample image sequence to be restored is input into the image restoration model to be trained, so that the image restoration model determines a target sample image containing a sample object and a reference sample image containing the sample object in the sample image sequence to be restored, the sample object in the target sample image is restored by utilizing the sample object in the reference sample image, the restored target sample image is obtained, a loss value is calculated based on a true value of the restored target sample image and the target sample image, and network parameters of the image restoration model are adjusted based on the loss value. Because the definition of the sample object in the reference sample image is higher than that of the sample object in the target sample image, the image restoration model can depend on the detail content of the sample object in the reference sample image when restoring the sample object in the target sample image, so that effective information is added to the restoration process of the sample object losing a large amount of detail in the video, and the accuracy of the image restoration model obtained through training is finally improved.

In addition, in the image restoration method provided by the embodiment of the invention, for the image sequence to be restored, the image restoration model obtained by training by using the training method provided by the embodiment of the invention is used for carrying out image restoration, after the input image sequence to be restored, the image to be restored containing the target object and the reference image containing the target object in the image sequence to be restored are determined, the target object in the reference image with higher definition is used for restoring the target object in the image to be restored, so that the restored image is obtained, the restoration of the object losing a large amount of details in the video is realized, and the viewing experience of audiences is improved.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.

FIG. 1 is a schematic diagram showing the effects of the related art;

FIG. 2 is a schematic flow chart of a training method of an image restoration model according to an embodiment of the present invention;

FIG. 3 is a flowchart of another training method of an image restoration model according to an embodiment of the present invention;

fig. 4 is a schematic flow chart of an image restoration method according to an embodiment of the present invention;

FIG. 5 is a flowchart of another image restoration method according to an embodiment of the present invention;

FIG. 6 is a flowchart of another image restoration method according to an embodiment of the present invention;

fig. 7 is a schematic diagram of a process of detecting, identifying and classifying an image sequence to be repaired according to an embodiment of the present invention;

FIG. 8 is a schematic diagram of a process for repairing an image sequence to be repaired according to an embodiment of the present invention;

FIG. 9 is a schematic diagram of a training process of an image restoration model according to an embodiment of the present invention;

FIG. 10 is a schematic diagram of a use process of an image restoration model according to an embodiment of the present invention;

FIG. 11 is a schematic structural diagram of a training device for an image restoration model according to an embodiment of the present invention;

fig. 12 is a schematic structural diagram of an image restoration device according to an embodiment of the present invention;

fig. 13 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be described below with reference to the accompanying drawings in the embodiments of the present invention.

For better understanding of the embodiments of the present invention, the following describes an application scenario with reference to fig. 1 and the related art.

With the progress of technology, photographic technology is also developing, and the quality of video is improved, but for various reasons, the quality of video is uneven; for example, due to the age of shooting or aging of transmission media, quality problems such as noise, blurring, distortion, etc. may occur in images of classical movie works, and these quality problems may result in poor definition of classical movie works. Any object contained in the video can help the audience understand the works, and as an important object in the video, the face of the person in the video can help the audience to better read the video, for example, the face of the person in the classical film and television works can show the film contents such as storyline, character and the like. However, the low quality of the video may affect the sharpness of objects in the video; for example, when the definition of classical movie works is not high, problems such as blurring, distortion and the like occur to objects in the video. As can be seen, blurring of objects in a video can have a significant impact on the viewing experience of the viewer.

However, in repairing a fuzzy object, the following three types of problems may be encountered:

the first type of problem is that the quality of the image of the original video is poor, for example, the resolution of the image is low, the noise is high, or the light is poor, and these quality problems have a great influence on the object repair result in the video, and the object repair result may be not ideal.

The second problem is that important structural components of the object are lost in the image, for example, the exact shape of eyes, nose or mouth of the face in the image is blurred, and the difficulty of recovering the face area is great.

A third class of problems is that the object itself has complex details, such as skin texture, hair texture, or facial expression of the face, which are challenges for restoration algorithms based on digital image restoration techniques.

If much detailed information of the object in the video is lost, for example, structural parts of a face or complex details are blurred, the object repaired by applying the related technology cannot conform to the performance of the real object in the video, and better viewing experience cannot be brought to the audience.

Illustratively, as shown in fig. 1, in a blurred image, eyebrows, eyes, nose, mouth, and french lines near the mouth of a face are blurred; in the truth image, the human face is a female old man; repairing the blurred image by using a digital image repairing technology to obtain two prediction results; comparison with the truth image shows that in the first prediction result, eyes are deeper, nose is straighter, and the first prediction result looks more like a foreigner; in the second prediction, there is a moustache near the mouth. It can be seen that, the related technology is applied to repair the fuzzy object, and the obtained repair result may not conform to the situation of the real object.

In order to repair objects losing a large amount of details in a video so as to improve the viewing experience of audiences, the embodiment of the invention provides a training method of an image repair model, an image repair method and a related device.

The following first describes a training method of an image restoration model provided by the embodiment of the present invention.

The training method of the image restoration model provided by the embodiment of the invention can be applied to electronic equipment, the electronic equipment is particularly used for training the model, and in the specific application, the electronic equipment can be a smart phone, a tablet personal computer and the like, which are all reasonable.

The embodiment of the invention provides a training method of an image restoration model, which comprises the following steps:

acquiring a sample image sequence to be repaired;

The following describes a training method of an image restoration model provided by the embodiment of the invention with reference to the accompanying drawings.

Fig. 2 is a flow chart of a training method of an image restoration model according to an embodiment of the present invention, as shown in fig. 2, the method may include steps S201 to S204.

S201, acquiring a sample image sequence to be repaired.

It may be understood that a video to be repaired including a sample object may be acquired, and the video may be used as a sample image sequence, or multiple frames of images to be repaired including a sample object may be acquired, and the video may be used as a sample image sequence, which is not specifically limited to the embodiment of the present invention. For example, a sample pair of a target sample image and a reference sample image to be repaired may be acquired, where the target sample image and the reference sample image each contain a sample object, and the sharpness of the contained sample object is different.

It should be noted that the obtained sample image sequence is an image sequence to be repaired, that is, the sharpness of each frame image in the sample image sequence is low relative to the sharpness image, where the sharpness image may be that the index value related to the sharpness index is higher than the preset sharpness index value. Optionally, in an implementation manner, the sharpness reducing process may be performed on the sharp image with the index value higher than the preset sharpness index value, for example, noise is added to the sharp image, so as to obtain an image with the reduced sharpness, which is used as an image in the sample image sequence.

Alternatively, in one implementation, the process of determining the sharpness of the image according to the index value of sharpness may include steps A1-A2.

A1, calculating an index value of the definition index of the image.

A2, determining the definition degree of the image according to the index value of the definition index of the image.

It can be understood that the index value of the image definition index can be calculated according to a preset definition evaluation algorithm, and the definition degree of the image can be determined according to the index value of the image definition index. Illustratively, the sharpness index value of the image is calculated by using the Laplace function and the image gray value of the image, and the sharpness index value can reflect the change of the pixel gray value of the image edge, so that the lower the sharpness index value is, the smaller the change of the pixel gray value of the image edge is, and the lower the sharpness is; accordingly, the higher the sharpness index value, the greater the change in the gray value of the image edge pixels, and the higher the sharpness.

It should be noted that when the index value of the sharpness index of an image is higher than a preset sharpness index, the image may be regarded as a sharp image, and when the index value of the sharpness index of an image is not higher than the preset sharpness index, the image may be regarded as a blurred image.

According to the scheme, the definition degree can be accurately and rapidly determined by calculating the index value of the specific definition index based on the content of the image, and a foundation is provided for the accurate classification processing of the image.

S202, inputting a sample image sequence to be repaired into an image repair model to be trained, so that the image repair model determines a target sample image containing a sample object and a reference sample image containing the sample object in the sample image sequence to be repaired, and repairing the sample object in the target sample image by using the sample object in the reference sample image to obtain a repaired target sample image.

Wherein the sharpness of the sample object in the reference sample image is higher than the sharpness of the sample object in the target sample image.

It can be appreciated that the acquired sample image sequence may be input into an image restoration model to be trained, the image restoration model may determine a target sample image including a sample object in the sample image sequence to be restored, the target sample image is an image to be restored, and the image restoration model may determine a reference sample image including the sample object according to the sample object, where the sharpness of the sample object in the reference sample image is higher than that of the sample object in the target sample image, so that the reference sample image may be used as a reference for restoring the target sample image, and the image restoration model may restore the sample object in the target sample image by using the sample object in the reference sample image based on the current network parameter, so as to obtain a restored target sample image. For any target sample image, the number of reference sample images used for repairing the target sample image may be one or more frames, which is not particularly limited in the embodiment of the present invention.

It should be noted that, in the process of determining the target sample image including the sample object in the sample image sequence to be repaired, the image repair model may identify the object included in each image in the sample image sequence, and then select the image including the sample object as the target sample image in the sample image sequence.

In the process of determining the reference sample image including the sample object, the image restoration model may use a clear image including the sample object and having an index value higher than a preset sharpness index value as the reference sample image. For example, a clear reference sample image containing sample objects may be directly acquired, where the same sample objects that the reference sample image and the target sample image have may be the same person, the same skin tone person, the same expression person, or the same person with the same facial features.

Alternatively, in one implementation, the sample image sequence to be repaired may be directly input into the image repair model, so that the image repair model itself identifies the target sample image and the reference sample image in the sample image sequence, thereby determining the target sample image and the reference sample image.

In another implementation, the sample image sequence may be identified with respect to the target sample image and the reference sample image and tag set for the identified target sample image and reference sample image before the sample image sequence is input to the image restoration model, such that the image restoration model may utilize the tag to determine the target sample image and the reference sample image in the sample image sequence after the sample image sequence is input to the image restoration model.

In addition, the sample image sequence may be identified by the same or different methods as the two methods described above, which are not specifically limited and only illustrated.

Optionally, in one implementation, the target sample image is: and the true value of the target sample image is subjected to the sharpness reducing treatment to obtain the image.

It is understood that a clear image containing a sample object whose index value exceeds a preset definition index value may be taken as a true value of the target sample image, and then the definition of the clear image containing the sample object is reduced to obtain the target sample image. The sharpness reducing process may be operations of adding noise, reducing resolution, or blurring to a sharp image of a sample object, which is not specifically limited and only illustrated.

In the scheme, the target sample image is obtained by processing the true value to reduce the definition, so that the number of required samples is reduced, and the training speed of the image restoration model is improved.

S203, calculating a loss value based on the restored target sample image and the true value of the target sample image.

It will be appreciated that the loss value may be calculated by comparing the repaired target sample image with the true value of the target sample image, and, for example, the loss value may be calculated by inputting the features of the repaired target sample image and the features of the true value of the target sample image into a preset loss function. The loss function may include a square error loss function, a perceptual loss function, an antagonistic loss function, and the like, and in the model training process, network parameters need to be adjusted according to each loss value to balance each loss value and optimize the model performance.

And S204, when judging that the image restoration model is not converged based on the loss value, adjusting network parameters of the image restoration model, and returning to the step of acquiring the sample image sequence to be restored.

It can be understood that, since the true value of the target sample image can reflect the situation of the target sample image in a real and clear state, the repairing effect of the repaired target sample image can be reflected based on the loss value calculated by the repaired target sample image and the true value of the target sample image. Under the condition that the loss value exceeds the preset loss threshold value, the reply restoration effect of the target sample image is not ideal, namely the restoration effect of the image restoration model is not good, at the moment, the network parameters of the image restoration model are adjusted, and the sample image sequence to be restored is acquired in a return mode, so that continuous training of the image restoration model is realized. For example, the network parameters may be updated over multiple iterations based on a back propagation algorithm to optimize the image restoration model.

It should be noted that, if the loss value does not exceed the preset loss threshold, the repairing effect of the target sample image is considered to be ideal, that is, the repairing effect of the image repairing model meets the requirement, and at this time, the trained image repairing model is obtained.

Optionally, in order to better understand the training method of the image restoration model provided by the embodiment of the present invention, a process for restoring the target sample image by the image restoration model is described below with reference to fig. 3.

As shown in fig. 3, on the basis of the embodiment shown in fig. 2, the image restoration model restores the sample object in the target sample image by using the sample object in the reference sample image, so as to obtain a restored target sample image, and may include steps S2021-S2024.

S2021, performing feature extraction processing on the image content of the sample object in the reference sample image to obtain reference feature data, and performing feature extraction processing on the image content of the sample object in the target sample image to obtain feature data to be repaired.

It may be appreciated that the image restoration model may be an encoder-decoder (encoder-decoder) structure, and specifically, the image restoration model may include a feature extraction layer, and based on the feature extraction layer, feature extraction processing may be performed on image contents of a sample object in a reference sample image to obtain reference feature data, and feature extraction processing may be performed on image contents of the sample object in a target sample image to obtain feature data to be restored. The feature extraction layer may be a convolutional neural network (Convolutional Neural Networks, CNN) or a residual neural network (res net) to extract features of the image, and the embodiment of the present invention is not limited in detail with respect to a specific form of the feature extraction layer.

Optionally, in one implementation, the input image may be preprocessed before feature extraction is performed on the image content, where the preprocessed image may be more easily feature-extracted, and exemplary, the input image may be normalized such that pixel data of the image falls within a range of 0 to 1. In the scheme, the image can be preprocessed, and the efficiency of feature extraction is improved.

And S2022, determining joint feature data corresponding to the target sample image based on the extracted reference feature content and the feature content to be repaired.

It can be appreciated that the image restoration model may combine the reference feature content with the feature content to be restored to obtain the joint feature data corresponding to the target sample image.

Optionally, in an implementation manner, the determining, based on the extracted reference feature content and the feature content to be repaired, joint feature data corresponding to the target sample image includes:

It can be understood that the image restoration model can perform weighting processing and splicing processing on the extracted reference feature content and the feature content to be restored so as to fuse and obtain joint feature data corresponding to the target sample image. The image restoration model may calculate an association weight ratio between features based on an attention mechanism (attention mechanism), and perform weighted fusion on the reference feature content and the feature content to be restored according to the association weight ratio, so as to splice and form joint feature data corresponding to the target sample image. In the scheme, the joint characteristic data corresponding to the target sample can be obtained rapidly and accurately through weighting processing and splicing processing.

S2023, performing image content restoration processing based on the joint feature data corresponding to the target sample image to obtain the image content of the target sample image for the sample object.

It can be appreciated that the image restoration model may perform image content restoration processing on the joint feature data, and map the joint feature data back to the original image space, so as to obtain the image content of the target sample image for the sample object. For example, the image restoration model may perform image content restoration processing based on a decoder included in itself or a generation countermeasure network (Generative Adversarial Network, GAN).

And S2024, repairing the image content of the sample object in the target sample image based on the obtained image content to obtain a repaired target sample image.

It can be appreciated that after obtaining the image content of the repaired target sample image for the sample content, the image content of the sample object in the target sample image can be repaired to obtain the repaired target sample image.

In the scheme, the target sample image after the sample object is repaired can be obtained rapidly and efficiently through the structure of the image repair model in the training process, so that the efficiency of training the image repair model is improved.

An image restoration method provided by the embodiment of the invention is described below.

The image restoration method provided by the embodiment of the invention can be applied to electronic equipment, the electronic equipment is particularly used for restoring images, and in the specific application, the electronic equipment can be a smart phone, a tablet personal computer and the like, which are all reasonable.

The embodiment of the invention provides an image restoration method, which comprises the following steps:

acquiring an image sequence to be repaired;

the image restoration model is as follows: the model obtained by training is based on the training method of the image restoration model described in the foregoing embodiment.

In the image restoration method provided by the embodiment of the invention, aiming at the image sequence to be restored, the image restoration model obtained by training by using the training method provided by the embodiment of the invention is used for carrying out image restoration, after the input image to be restored and the reference image containing the target object in the image sequence to be restored are determined, the target object in the reference image with higher definition is used for restoring the target object in the image to be restored, so that the restored image is obtained, the object losing a large amount of details in the video is restored, and the viewing experience of audiences is improved.

An image restoration method provided by the embodiment of the invention is described below with reference to the accompanying drawings.

Fig. 4 is a schematic flow chart of an image restoration method according to an embodiment of the present invention, as shown in fig. 4, the method may include steps S401 to S402.

S401, acquiring an image sequence to be repaired.

It will be appreciated that in the process of actually repairing an image, a sequence of images to be repaired, that is, a sequence of images to be repaired, may be acquired first, and for reference repair of an image, there is an image in the acquired sequence of images to be repaired that may be used as a reference image, where the reference image has the same characteristics as other images in the sequence of images to be repaired, for example, includes the same object.

For example, the acquired image sequence to be repaired may be a video with continuous content or a video with continuous content, instead of a video with a segment of video clips without any association, and the objects in the image sequence with continuous content have the same characteristics, for example, the objects in the image sequence to be repaired may be the same person with similar makeup shapes, and the image sequence to be repaired may be a set of television shows in which the same person has similar makeup shapes. The embodiments of the present invention are not particularly limited, but merely illustrative. Optionally, if the obtained image sequence to be repaired is video, extracting each video frame of the video, that is, performing frame disassembly processing on the video to obtain the image sequence to be repaired formed by each video frame.

S402, inputting an image sequence to be repaired into an image repair model obtained through pre-training, so that the image repair model determines an image to be repaired containing a target object and a reference image containing the target object in the image sequence to be repaired, and repairing the target object in the image to be repaired by utilizing the target object in the reference image to obtain a repaired image.

the image restoration model is as follows: the model obtained by training is based on the training method of the image restoration model provided by the embodiment of the invention.

It can be understood that the image sequence to be repaired is input into the image repair model, and the image repair model is obtained by training the image repair model by using the training method of the image repair model in the foregoing embodiment, so that the image repair model can repair the target object in the image to be repaired by using the target object in the reference image with higher definition after determining the image to be repaired containing the target object and the reference image containing the target object in the image sequence to be repaired, thereby obtaining the repaired image.

It should be noted that, the image restoration model may identify the object included in each image in the image sequence to be restored, and then, among the images including the target object, use the image whose index value exceeds the preset definition index value as the reference image and use the image whose index value does not exceed the preset definition index value as the target image.

It is emphasized that the restoration can be performed on the target object in the image to be restored, for example, the restoration can be performed on the face of the principal angle in the movie, the image to be restored containing the target object and the reference image can be determined according to the identity, that is, the image with the same identity is identified and clustered, so that the image restoration of the target object is realized, the viewing experience of the audience is improved, the restoration of irrelevant images is reduced, the calculation of irrelevant data is reduced, and the efficiency of image restoration is increased. It should be noted that this implementation is merely an illustration of one possible implementation, and the embodiments of the present invention are not limited in detail.

Optionally, in an implementation manner, the image to be repaired and the reference image are determined according to the following manner:

It will be appreciated that the pre-classification operation may include identification of an identity and detection of sharpness, so that the identity of each image in the image sequence to be repaired and the sharpness of each image may be identified by the pre-classification operation, and then the image to be repaired including the target object and the reference image may be obtained based on the identity of the object included in each image and the detected sharpness. In the scheme, the image to be repaired and the reference image can be accurately obtained through the pre-classification operation, and the accuracy of image repair is improved.

Optionally, in an implementation manner, the pre-classifying the images including the target object in the image sequence to be repaired to obtain the image to be repaired and the reference image includes steps B1-B2.

B1, determining the definition of each frame image in the image sequence to be repaired and the identity of the contained object.

It may be appreciated that an object identification model trained in advance may be used to identify an identity of an object included in each frame of image in the image sequence to be repaired, where the object identification model is used to identify an identity of each object in the image, and an index value of a sharpness index of each frame of image in the image sequence to be repaired may be calculated to determine sharpness of each frame of image.

It should be noted that, the manner of determining the definition of each frame image is described in the foregoing embodiments, and will not be described in detail herein.

And B2, determining an image to be repaired and a reference image in the image sequence to be repaired based on the determined definition and the identification.

It may be understood that after determining the object and the definition of the image included in each frame of the image sequence to be repaired, the image with the index value exceeding the preset definition index value and including the target object may be used as the reference image, and the image with the index value not exceeding the preset definition index value and including the target object may be used as the target image, according to the identity and the definition, thereby determining the image to be repaired and the reference image.

In the scheme, the image to be repaired and the reference image can be accurately obtained by determining the definition and the identity, so that the accuracy of image repair is improved.

Optionally, in an implementation manner, the determining the image to be repaired and the reference image in the image sequence to be repaired based on the determined definition and the identity includes steps B21-B22.

And B21, classifying each frame of image containing the object in the image sequence to be repaired according to the determined definition to obtain each first type of image and each second type of image.

Wherein the sharpness of the object in each of the second type of images is higher than the sharpness of the object in each of the first type of images.

It will be appreciated that images of low definition may be classified as first type images, images of high definition may be classified as second type images, based on the determined definition, resulting in respective first type images, and second type images. For example, an image whose index value of sharpness does not exceed a preset sharpness index value may be regarded as a first-type image, and an image whose index value exceeds a preset sharpness index value may be regarded as a second-type image, that is, the sharpness of an image in the second-type image is higher than that in the first-type image.

It should be noted that, for each first type image, the identity identifier may be used as a selection criterion, and according to the identity identifier of the first type image, a second type image identical to the identity identifier is selected from the second type images, and the selected second type image is used as a reference image. For example, if the identity of a first type image is 1, a second type image with the identity of 1 may be selected as the reference image.

It is emphasized that the second type of image has a higher sharpness than the first type of image, and correspondingly, the second type of image has more detailed information of the object retained in the image than the first type of image.

And B22, selecting images to be repaired from the first type images, and determining second type images with the same identity with the images to be repaired from the second type images to obtain reference images.

It can be understood that the definition of the first type of image is lower than that of the second type of image, the first type of image can be used as an image to be repaired, the image to be repaired is selected from the first type of images, the image in the second type of image can be used as a reference image, and in order to enable the selected reference image to have reference value relative to the image to be repaired, the second type of image with the same identity as the image to be repaired can be used as the reference image according to the identity. For example, the person in the episode with the third person can repair the face image with the lower definition of the third person in the episode by using the face image with the higher definition of the third person in the episode.

It should be noted that, in the same image sequence to be repaired, for the same identity, the reference image not only can retain more object details, but also can retain many same attributes with the image to be repaired, for example, for people appearing in a section of continuous video, the people of the same identity do not have too much age and shape differences, and the continuous video may also be consistent in shooting equipment and shooting scenes, so that the objects in the reference image of the same identity can have the same character attributes of makeup, hairstyle, clothing, etc., or scene attributes of light, background, etc., as the objects in the image to be repaired. Therefore, for the reference image and the image to be repaired from the same image sequence to be repaired, the target object in the image to be repaired is repaired based on the target object in the reference image and the target object in the image to be repaired, the front and back content in the video, namely the context information of the video, can be better utilized, the consistency of the target object in time and attribute is maintained, the repairing result is more natural and consistent, and the repairing effect is improved.

In addition, in the scheme, through determining the identity of the object, namely identifying and clustering the objects with the same identity, special restoration can be carried out aiming at the object with each identity, individual characteristics are reserved, the restored object is more in line with the real characteristics of the object, the restoration process is more accurate, the condition that the restored object is not real is avoided, and the reality and the credibility of the restoration result of the object are improved. In addition, by identifying and clustering the objects with the same identity, the method can repair only the object with a specific identity, and does not need to repair the object with other identities, thereby reducing the calculation of irrelevant data, improving the calculation efficiency and increasing the repair efficiency.

In addition, in the scheme, the images to be repaired can be repaired by means of the reference images, the front and rear contents in the video, namely the context information of the video, can be better utilized, the images to be repaired are repaired, the consistency of objects in the images to be repaired in time is maintained, the repairing result is more natural and consistent, the repairing effect is improved, and the visual discomfort of audiences is reduced.

Optionally, in another embodiment, based on the image restoration method shown in fig. 4, as shown in fig. 5, in step S402, the image restoration model restores the target object in the image to be restored by using the target object in the reference image, to obtain a restored image, including steps S4021 to S4024.

S4021, performing feature extraction processing on the image content of the target object in the reference image to obtain reference feature data, and performing feature extraction processing on the image content of the target object in the image to be repaired to obtain feature data to be repaired.

S4022, determining joint feature data corresponding to the image to be repaired based on the extracted reference feature content and the feature content to be repaired.

S4023, performing image content restoration processing based on the joint feature data corresponding to the image to be restored to obtain the image content of the image to be restored aiming at the target object.

S4024, repairing the image content of the target object in the image to be repaired based on the obtained image content to obtain a repaired image.

It can be understood that, in the foregoing embodiment, the implementation manner of applying the trained image restoration model and restoring the target object in the image to be restored by using the target object in the reference image is similar to the implementation manner of applying the trained image restoration model and restoring the sample object in the target sample image by using the sample object in the reference sample image in the process of training the image restoration model, and will not be described in detail herein.

In the scheme, the target image after repairing the target object can be obtained rapidly and efficiently through the structure of the image repairing model, so that the effect and efficiency of repairing the image by the image repairing model are improved.

Optionally, on the basis of the image restoration method shown in fig. 4, as shown in fig. 6, the image restoration method may further include step S601.

S601, inputting an image sequence to be repaired into an image processing model obtained through pre-training, so that the image processing model determines a reference image containing a target object in the image sequence to be repaired, and performing image repair on the target object in the reference image to obtain a repaired reference image.

The image processing model is used for repairing an image, and is: and training the obtained model based on the sample image and the true value of the sample image, wherein the true value of the sample image and the sample image belong to the same object, and the definition of the true value of the sample image is higher than that of the sample image.

It can be understood that the image processing model is another model which can be utilized in performing image restoration according to the embodiment of the present invention and is different from the image restoration model in the foregoing embodiment, and because the image processing model is a model obtained by training based on a sample image and a true value of the sample image, the true value of the sample image and the sample image belong to the same object, and the true value of the sample image has higher definition than the sample image, the trained image processing model can directly restore the input image based on the structure of the model itself.

It should be noted that, as in the foregoing embodiment, the definition of the reference image is higher than that of the image to be repaired, that is, the detail information of the object included in the reference image is more than that of the image to be repaired, and since the image processing model does not need to repair another image with one image, the repair speed of the image processing model is faster than that of the image repair model, so that in order to improve the repair efficiency of the image sequence to be repaired, the reference image does not need to be repaired with the image repair model, and the repair efficiency of the reference image itself based on the image processing model can be improved while the image repair effect is considered.

The image processing model may be a model trained by using a sample image and a true value of the sample image, and the training process of the image processing model may include:

acquiring a plurality of images with definition index values higher than a preset definition index and containing a target object as true values, and performing fuzzy processing on the acquired images, for example, adding noise points to the images to obtain a plurality of sample images with fuzzy effects; determining a plurality of groups of sample images and corresponding true values thereof; then, inputting the sample image into an image processing model to be trained aiming at each sample image so as to repair the sample image by the image processing model to obtain a repaired sample image; calculating a loss value based on the repaired sample image and the corresponding true value; judging whether the image processing model in training is converged based on the loss value, and ending training if the image processing model in training is converged to obtain the image processing model after training is completed; if not, adjusting network parameters of the image processing model, and continuing training the image processing model.

In the scheme, the reference image is restored based on the image processing model, so that the efficiency of restoring the image sequence to be restored can be improved under the condition of considering the image restoration effect.

In order to better understand the image restoration method provided by the embodiment of the present invention, a process of performing a pre-classification operation on each image in the image sequence to be restored is described below with reference to fig. 7.

As shown in fig. 7, the electronic device receives the image sequence to be repaired, performs object detection processing and object recognition processing on each image in the image sequence to be repaired, and determines an identity corresponding to each object, where the identities corresponding to each object may be: object 1, object 2, object 3, etc.; then, the definition degree of each image is identified by a definition judging device for identifying the definition degree of each image, wherein the definition judging device can be used for identifying the definition degree of the image, and likewise, a blurring judging device for judging the blurring degree of the image can be also used, so that the clearer the image is, the lower the blurring degree is; then classifying the images with the determined identity marks according to the definition degree of the identified images to obtain the first type images and the second type images; the sharpness of each first type of image is higher than that of each second type of image, that is, the second type of image is more blurred than the first type of image, and the first type of image is more sharp than the second type of image, so that after the images with determined identity are classified, the first type of image and the second type of image with determined identity can be determined: a clearer image of the object 1 and a blurred image of the object 1, a clearer image of the object 2 and a blurred image of the object 2, a clearer image of the object 3 and a blurred image of the object 3, and the like.

In the scheme, the images of the identity marks of the objects which are determined to be contained are classified into the first type of images and the second type of images, and a foundation is made for repairing according to different repairing methods in the subsequent classification, so that the efficiency and the accuracy of image repairing can be improved.

In order to better understand the image restoration method provided by the embodiment of the present invention, a restoration process is performed on an image to be restored in an image sequence to be restored in the following manner with reference to fig. 8.

As shown in fig. 8, for the image sequence to be repaired, determining an identity corresponding to each object contained in each image; identifying the sharpness of each image by using a sharpness determiner for identifying the sharpness of each image; then classifying each image according to the definition and the identity of each identified image to obtain each first type image and each second type image; wherein the definition of each first type of image is higher than the definition of each second type of image; then, each first type image is used as a reference image, and the reference image is repaired by using an image processing model, so that a repair result of the reference image is obtained; meanwhile, taking each second type image as an image to be repaired, and selecting the corresponding identity mark from each first type image as a reference image corresponding to the image to be repaired, wherein the identity mark is the same as that of the image to be repaired; and repairing the image to be repaired based on the image content of the image to be repaired and the image content of the corresponding reference image by using an image repairing model to obtain a repairing result of the image to be repaired.

By means of the scheme, expression details of actors can be better conveyed to the old movie works, artistic value of the movie works is fully displayed, the look and feel of the old movie works is improved, interests of audiences to the old movie works are stimulated to a certain extent, more audiences are attracted, audience range of the old movie works is enlarged, and inheritance of cultural heritage is facilitated. In addition, the repaired old film and television works can provide clearer and truer character images for directors, designers or producers, provide a basis for subsequent re-production and transmission, are beneficial to the repair work of film and television files, and can better protect cultural heritage. In addition, in the scheme, the reference image and the image to be repaired belong to the same episode, and as people in the same episode often have certain consistency in the aspects of makeup, clothing, hairstyle and the like, the reference image in the same episode can provide the reference information of the conditions such as light rays, scenes and the like under the episode for the image to be repaired, so that the image to be repaired is repaired based on the reference image and the image to be repaired, and the accuracy and quality of a repair result can be increased.

In order to better understand the training method of the image restoration model provided by the embodiment of the present invention, a training process of the image restoration model is described below with reference to fig. 9.

As shown in fig. 9, a target sample image including a sample object and a reference sample image including a sample object are input as input contents, and are input into an image restoration model together based on a concat method, so that the image restoration model restores the sample object in the target sample image by using the sample object in the reference sample image, and a restored target sample image is obtained as a restoration result.

Calculating a loss value based on the repaired target sample image and a true value of the target sample image; the true value of the target sample image is subjected to the definition reduction processing, so that a reference sample image can be obtained.

In the scheme, the reference sample image is used as one of input contents of model training, and the image restoration model can be trained into an image restoration model with reference information.

In order to better understand the image restoration method provided by the embodiment of the present invention, the following describes the use process of the image restoration model with reference to fig. 10.

As shown in fig. 10, an image to be repaired including a target object and a reference image including the target object are input as input contents, and are input into a pre-trained image repair model together based on a concat method, so that the image repair model repairs the target object in the image to be repaired by using the target object in the reference image, and a repaired image is obtained as a repair result.

Based on the foregoing training method of the image restoration model, the embodiment of the invention further provides a training device of the image restoration model, as shown in fig. 11, where the device includes:

a first acquisition module 1101, configured to acquire a sample image sequence to be repaired;

the first input module 1102 is configured to input a sample image sequence to be repaired into an image repair model to be trained, so that the image repair model determines a target sample image including a sample object and a reference sample image including the sample object in the sample image sequence to be repaired, and repair the sample object in the target sample image by using the sample object in the reference sample image to obtain a repaired target sample image; wherein the sharpness of the sample object in the reference sample image is higher than the sharpness of the sample object in the target sample image;

A calculating module 1103, configured to calculate a loss value based on the repaired target sample image and a true value of the target sample image;

and an adjustment module 1104, configured to adjust network parameters of the image restoration model when it is determined that the image restoration model is not converged based on the loss value, and return to the step of acquiring the sample image sequence to be restored.

Based on the foregoing image restoration method, an embodiment of the present invention further provides an image restoration device, as shown in fig. 12, where the device includes:

a second obtaining module 1201, configured to obtain an image sequence to be repaired;

the second input module 1202 is configured to input an image sequence to be repaired into an image repair model obtained by training in advance, so that the image repair model determines an image to be repaired including a target object and a reference image including the target object in the image sequence to be repaired, and repair the target object in the image to be repaired by using the target object in the reference image to obtain a repaired image;

the image restoration model is as follows: the training device based on the image restoration model provided by the embodiment of the invention trains the obtained model.

Optionally, the image to be repaired and the reference image are determined as follows:

The embodiment of the present invention further provides an electronic device, as shown in fig. 13, including a processor 1301, a communication interface 1302, a memory 1303 and a communication bus 1304, where the processor 1301, the communication interface 1302, and the memory 1303 complete communication with each other through the communication bus 1304,

a memory 1303 for storing a computer program;

processor 1301 is configured to implement the training method and the image restoration method of the image restoration model according to any of the above embodiments when executing the program stored in memory 1303.

The communication bus mentioned by the above terminal may be a peripheral component interconnect standard (Peripheral Component Interconnect, abbreviated as PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, abbreviated as EISA) bus, etc. The communication bus may be classified as an address bus, a data bus, a control bus, or the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.

The communication interface is used for communication between the terminal and other devices.

The memory may include random access memory (Random Access Memory, RAM) or non-volatile memory (non-volatile memory), such as at least one disk memory. Optionally, the memory may also be at least one memory device located remotely from the aforementioned processor.

The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), etc.; but also digital signal processors (Digital Signal Processor, DSP for short), application specific integrated circuits (Application Specific Integrated Circuit, ASIC for short), field-programmable gate arrays (Field-Programmable Gate Array, FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

In still another embodiment of the present invention, a computer readable storage medium is provided, where a computer program is stored, where the computer program, when executed by a processor, implements the training method and the image restoration method for an image restoration model according to any of the foregoing embodiments.

In yet another embodiment of the present invention, a computer program product comprising instructions, which when run on a computer, causes the computer to perform the training method and the image restoration method of the image restoration model according to any of the above embodiments is also provided.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present invention, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, by wired (e.g., coaxial cable, optical fiber, digital Subscriber Line (DSL)), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), etc.

It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for the device embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference is made to the description of the method embodiments in part.

The foregoing description is only of the preferred embodiments of the present invention and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method of training an image restoration model, the method comprising:

acquiring a sample image sequence to be repaired;

2. The method of claim 1, wherein the image restoration model restores the sample object in the target sample image using the sample object in the reference sample image to obtain a restored target sample image, comprising:

3. The method according to claim 2, wherein determining joint feature data corresponding to the target sample image based on the extracted reference feature content and the feature content to be repaired includes:

4. The method according to claim 1 or 2, wherein the target sample image is: and the true value of the target sample image is subjected to the sharpness reducing treatment to obtain the image.

5. An image restoration method, comprising:

acquiring an image sequence to be repaired;

the image restoration model is as follows: model trained based on the training method of any one of claims 1-4.

6. The method of claim 5, wherein the step of determining the position of the probe is performed,

7. The method of claim 6, wherein the pre-classifying each image of the sequence of images to be repaired including the target object to obtain the image to be repaired and the reference image comprises:

8. The method of claim 7, wherein determining the image to be repaired and the reference image in the sequence of images to be repaired based on the determined sharpness and the identity, comprises:

9. The method of claim 5, wherein the image restoration model restores the target object in the image to be restored by using the target object in the reference image to obtain a restored image, comprising:

10. A training device for an image restoration model, the training device comprising:

11. An image restoration device, comprising:

the image restoration model is as follows: the model trained based on the training apparatus of claim 10.

12. The electronic equipment is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are communicated with each other through the communication bus;

a memory for storing a computer program;

a processor for implementing the method of any of claims 1-9 when executing a program stored on a memory.

13. A computer readable storage medium, characterized in that the computer readable storage medium has stored therein a computer program which, when executed by a processor, implements the method of any of claims 1-9.