WO2023207778A1

WO2023207778A1 - Data recovery method and device, computer, and storage medium

Info

Publication number: WO2023207778A1
Application number: PCT/CN2023/089718
Authority: WO
Inventors: 赵远远
Original assignee: 腾讯科技（深圳）有限公司
Priority date: 2022-04-24
Filing date: 2023-04-21
Publication date: 2023-11-02
Also published as: CN114549369B; CN114549369A

Abstract

Disclosed in embodiments of the present application are a data recovery method and device, a computer, and a readable storage medium. The method comprises: acquiring a recovery image sample to be recovered, a recovery area label corresponding to the recovery image sample, and an original image sample; using a first area prediction model to predict an area to be recovered of the recovery image sample to obtain a sample prediction recovery area; using a first media recovery model to recover the sample prediction recovery area in the recovery image sample to obtain a sample optimization image corresponding to the recovery image sample; and jointly adjusting parameters of the first area prediction model and the first media recovery model according to the sample prediction recovery area, the recovery area label, the original image sample, and the sample optimization image to obtain a target area prediction model corresponding to the first area prediction model and a target media recovery model corresponding to the first media recovery model. By using the present application, the accuracy of data recovery can be improved.

Description

Data recovery method, device, computer and readable storage medium

This application claims the priority of the Chinese patent application with application number 202210448573.5 and the invention name "data repair method, device, computer and readable storage medium" submitted on April 24, 2022.

Technical field

The present application relates to the field of computer technology, and in particular, to a data repair method, device, computer and readable storage medium.

Background technique

With the development and application of deep learning, image repair has gradually begun to be implemented using deep learning. At present, image repair processing is generally performed by inputting the image to be repaired into the model. This also requires more comprehensive information recognition of the image to be repaired in this model. In other words, more parameters need to be trained for image repair.

Contents of the invention

Embodiments of the present application provide a data repair method, device, computer and readable storage medium, which can improve the accuracy of data repair.

On the one hand, embodiments of the present application provide a data repair method, which method includes:

Obtain the repaired image sample to be repaired, the repair area label corresponding to the repaired image sample, and the original image sample;

Use the first area prediction model to predict the area to be repaired of the repaired image sample, and obtain the sample predicted repair area;

Use the first media repair model to repair the sample predicted repair area in the repaired image sample to obtain a sample optimized image corresponding to the repaired image sample;

According to the sample predicted repair area, repair area label, original image sample and sample optimized image, the parameters of the first area prediction model and the first media repair model are jointly adjusted to obtain the target area prediction model corresponding to the first area prediction model, and a target media repair model corresponding to the first media repair model.

Obtain the image frame to be repaired, predict the image frame to be repaired based on the target area prediction model, and obtain the area to be repaired in the image frame to be repaired;

The area to be repaired in the image frame to be repaired is repaired based on the target media repair model to obtain the optimized image frame corresponding to the image frame to be repaired. The target area prediction model and the target media repair model are obtained through joint training.

On the one hand, embodiments of the present application provide a data repair device, which includes:

The sample acquisition module is used to obtain the repaired image sample to be repaired, the repair area label corresponding to the repaired image sample, and the original image sample;

The sample area prediction module is used to use the first area prediction model to predict the area to be repaired of the repaired image sample, and obtain the sample predicted repair area;

The sample repair module is used to use the first media repair model to repair the sample predicted repair area in the repair image sample, to obtain Optimize the image to the sample corresponding to the repaired image sample;

The model adjustment module is used to jointly adjust the parameters of the first area prediction model and the first media repair model based on the sample prediction repair area, repair area label, original image sample and sample optimized image, and obtain the corresponding parameters of the first area prediction model. a target area prediction model, and a target media repair model corresponding to the first media repair model.

Image acquisition module, used to acquire image frames to be repaired;

The area prediction module is used to predict the image frame to be repaired based on the target area prediction model and obtain the area to be repaired of the image frame to be repaired;

The data repair module is used to repair the area to be repaired in the image frame to be repaired based on the target media repair model, and obtain the optimized image frame corresponding to the image frame to be repaired. The target area prediction model and the target media repair model are jointly trained. owned.

On the one hand, embodiments of the present application provide a computer device, including a processor, a memory, and an input and output interface;

The processor is connected to the memory and the input and output interface respectively. The input and output interface is used to receive data and output data. The memory is used to store the computer program. The processor is used to call the computer program so that the computer device containing the processor executes the computer program. The data repair method in one aspect of the application embodiment.

On the one hand, embodiments of the present application provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. The computer program is adapted to be loaded and executed by a processor, so that a computer device having the processor executes the present application. The data repair method in one aspect of the embodiment.

In one aspect, embodiments of the present application provide a computer program product or computer program. The computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in various optional ways in one aspect of the embodiments of the present application.

Implementing the embodiments of this application will have the following beneficial effects:

In the embodiment of the present application, the repaired image sample to be repaired, the repaired area label corresponding to the repaired image sample, and the original image sample can be obtained; the first area prediction model is used to predict the area to be repaired of the repaired image sample, and the sample predicted repaired area is obtained ;Use the first media repair model to repair the sample predicted repair area in the repaired image sample to obtain the sample optimized image corresponding to the repaired image sample; based on the sample predicted repair area, repair area label, original image sample and sample optimized image, perform The parameters of the first region prediction model and the first media repair model are jointly adjusted to obtain a target region prediction model corresponding to the first region prediction model and a target media repair model corresponding to the first media repair model. Further, the image can be repaired based on the target area prediction model and the target media repair model. Through the above process, the multi-task joint training and use of the first regional prediction model and the first media repair model are realized to realize mutual adjustment and promotion between different tasks, and fully learn complementary information and similar information in different tasks, etc. , obtain mutual gain effects, improve the efficiency of model training, and save computing resources. Since different tasks can provide each other with enhanced effective information to promote model performance of different tasks, mutually improve the accuracy of the output results of different models, it is conducive to the improvement of model design and effects, thereby improving the accuracy of data repair.

Description of the drawings

In order to more clearly explain the technical solutions in the embodiments of the present application or the prior art, the following will describe the technical solutions required in the embodiments or the prior art. The drawings used are used for brief introduction. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without exerting creative efforts, they can also make drawings based on these drawings. Figure obtains additional drawings.

Figure 1 is a network interaction architecture diagram of data repair provided by an embodiment of the present application;

Figure 2 is a schematic diagram of a data repair scenario provided by an embodiment of the present application;

Figure 3 is a flow chart of a model training method provided by an embodiment of the present application;

Figure 4 is a schematic diagram of a multi-step training method provided by an embodiment of the present application;

Figure 5 is a flow chart of a data repair method provided by an embodiment of the present application;

Figure 6 is a schematic diagram of a regional prediction method provided by an embodiment of the present application;

Figure 7 is a schematic diagram of another regional prediction method provided by an embodiment of the present application;

Figure 8 is a schematic diagram of a repair method provided by an embodiment of the present application;

Figure 9 is a schematic diagram of another repair method provided by an embodiment of the present application;

Figure 10 is a schematic diagram of a data repair device provided by an embodiment of the present application;

Figure 11 is a schematic diagram of another data repair device provided by an embodiment of the present application;

Figure 12 is a schematic structural diagram of a computer device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of this application.

In this embodiment of the present application, please refer to Figure 1. Figure 1 is a network interaction architecture diagram of data repair provided by an embodiment of the present application. Among them, the computer device 101 can perform data exchange with the terminal device, and different terminal devices can also perform data exchange with each other. The number of terminal devices may be one or at least two. For example, the number of terminal devices is three, such as the terminal device 102a, the terminal device 102b, the terminal device 102c, etc. shown in FIG. 1 . It is also possible that only the computer device 101 is present. Among them, the computer device 101 can obtain the repaired image sample from the storage space of the computer device 101 itself, or can obtain the repaired image sample from any one or more terminal devices, etc., which is not limited here. The computer device 101 can perform model training based on the obtained repaired image samples. Specifically, the computer device 101 can jointly train the first region prediction model and the first media repair model to obtain the target region prediction model corresponding to the first region prediction model and the target media repair model corresponding to the first media repair model. wait. Further, the computer device 101 can perform data repair based on the trained target area prediction model and target media repair model. Data repair is, for example, video completion. Video completion refers to completing the missing position information or the area to be cut out in the video based on the texture information and timing information of the non-missing area.

Among them, this application may involve machine learning technology in the field of artificial intelligence, using machine learning technology to expand the training samples of the model, and to train the model, etc.

Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. . In other words, artificial intelligence is a comprehensive technology of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is the study of the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.

Artificial intelligence technology is a comprehensive subject that covers a wide range of fields, including both hardware-level technology and software-level technology. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation/interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, machine learning/deep learning, autonomous driving, smart transportation and other major directions.

Machine Learning (ML) is a multi-field interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers can simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications cover all fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, teaching learning and other technologies. For example, in this application, for the training and use of target area prediction models and target media repair models, the models are trained to continuously learn new knowledge or skills, and then the trained models are obtained for use in data repair.

With the research and progress of artificial intelligence technology, artificial intelligence technology has been researched and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, driverless driving, autonomous driving, and drones. , robots, smart medical care, smart customer service, Internet of Vehicles, autonomous driving, smart transportation, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

Specifically, please refer to Figure 2. Figure 2 is a schematic diagram of a data repair scenario provided by an embodiment of the present application.

As shown in Figure 2, this application conducts multi-task model joint training and uses the model based on the joint training. As shown in Figure 2, the computer device can obtain the repaired image, and based on the repaired image, model training and prediction use of the regional prediction model and the media repair model. For example, during model training, the repaired image refers to the repaired image sample to be repaired. The repaired image sample is input into the first region prediction model for prediction, and a sample predicted repairing area to be repaired corresponding to the repaired image sample is obtained. The regional prediction model in Figure 2 is used to represent the first regional prediction model, and the repair area is used to represent the sample prediction repair area. Further, the computer device can use the repair area as an input to the first media repair model. For example, the repair image sample and the sample predicted repair area are input into the first media repair model for repair, and a sample optimized image corresponding to the repair image sample is obtained. Among them, during model training, the part indicated by the dotted line in Figure 2 is also included. For example, the model training can further jointly adjust the parameters of the first region prediction model and the first media repair model to obtain a target region prediction model corresponding to the first region prediction model and a target media repair model corresponding to the first media repair model. Model. Among them, during the use of the model, the repaired image can be an image frame to be repaired, the area prediction model refers to the target area prediction model, the repair area refers to the area to be repaired, the media repair model refers to the target media repair model, and the optimized image is Refers to optimizing image frames. The computer device can input the image frame to be repaired into the target area prediction model, perform prediction based on the target area prediction model, and obtain the area to be repaired of the image frame to be repaired. The area to be repaired in the image frame to be repaired is repaired based on the target media repair model to obtain an optimized image frame corresponding to the image frame to be repaired. Through joint training of multi-task models, the accuracy of the model's output results can be mutually improved. Different tasks provide mutually reinforcing effective information to promote the model performance of different tasks, which can improve the efficiency of model training and save computing resources. , thereby improving the accuracy of data repair.

It can be understood that the computer equipment mentioned in the embodiments of this application includes but is not limited to terminal equipment or servers. In other words, the computer device can be a server or a terminal device, or it can be a system composed of a server and a terminal device. Among them, the above-mentioned terminal device can be an electronic device, including but not limited to a mobile phone, a tablet computer, a desktop computer, a notebook computer, a handheld computer, a vehicle-mounted device, an augmented reality/virtual reality (AR) /VR) equipment, helmet-mounted displays, wearable devices, smart speakers boxes, digital cameras, cameras and other mobile internet devices (mobile internet devices, MID) with network access capabilities, etc. Among them, the server mentioned above can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or it can provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, Cloud servers for basic cloud computing services such as network services, cloud communications, middleware services, domain name services, security services, vehicle-road collaboration, Content Delivery Network (CDN), and big data and artificial intelligence platforms.

The data involved in the embodiments of this application can be stored in any one or at least two devices of computer equipment or terminal equipment, or the data can be stored based on cloud storage technology or blockchain network, which will not be done here. limit.

Further, please refer to Figure 3, which is a flow chart of a model training method provided by an embodiment of the present application. As shown in Figure 3, the model training method includes the following steps:

Step S301: Obtain the repaired image sample to be repaired, the repair area label corresponding to the repaired image sample, and the original image sample.

In the embodiment of the present application, the computer device can obtain the repaired image sample. The repaired image sample refers to the image sample to be repaired. Wherein, the repaired image sample may be an image, or may be one sample image frame among N sample image frames that constitute the video sample, where N is a positive integer. When the repaired image sample is an image, the computer device can obtain the repaired area label and the original image sample corresponding to the repaired image sample. When the repaired image sample is one of the N sample image frames that make up the video sample, the computer device can, for example, directly obtain the repaired area label and the original image sample corresponding to the repaired image sample from the video sample. The repair area label is, for example, the area to be repaired in the repair image sample generated through manual annotation. In this application, the label may be a numerical value used to represent the position of a pixel in an image sample or image frame, and may be one of two values. For example, in the inpainted image sample, the numerical value of the pixel position in the area to be repaired is represented by a repair area label with one of the binary values (for example, 1), and the numerical value of the pixel position in the remaining areas is represented by the other of the binary values. represented by a value (for example, 0). The binary value can also be other values, for example, 1 and 100, etc.

Alternatively, the computer device may search the repaired area label corresponding to the repaired image sample from its internal memory or externally. If the repair area label is found, the computer device can directly obtain the repair area label. If the repair area label is not found, the computer device can predict the repair area label of the repair image sample based on the previous image sample of the repair image sample. For example, the video sample may include N sample image frames, in which there are repair area labels of the first image frame and repair area labels of the key image frames. The first image frame is the first image frame among N sample image frames, and the N sample image frames include key image frames. The repair area label of the first image frame and the repair area label of the key image frame are generated by manual annotation, for example, and are used to represent the area to be repaired in the first image frame and the area to be repaired in the key image frame. Among them, when the repaired image sample is one of the N sample image frames that make up the video sample, the preamble image sample of the repaired image sample in the N sample image frames is obtained, and the preamble image sample corresponding to the preamble image sample is obtained. Preamble sample repair area.

The computer equipment can directly obtain the repaired image sample from the data set, or can obtain the repaired image sample from the Internet, etc., or can generate the repaired image sample, etc., which are not limited here. That is, computer equipment can also obtain repaired image samples through other methods. In this application, any one of the above methods, or a combination of multiple methods, can be used to obtain the repaired image sample. For example, when generating a repaired image sample, the computer device can obtain the original image sample, perform damage processing on the original image sample, and obtain a repaired image sample. The damage processing may include but is not limited to adding watermarks, erasing part of the area, adding area noise or area blur processing, etc. One or at least two corresponding repaired image samples can be generated from an original image sample.

Wherein, when the repaired image sample is one image frame among N sample image frames that constitute the video sample, the computer device can generate the video sample through the following operations. The computer equipment can first obtain foreground object samples and conventional video data, and then perform simulation operations on the foreground object samples. Motion processing is performed to obtain the object motion trajectory. The foreground object sample may be, but is not limited to, area noise, area erasure mask, object object or area blur mask, etc. Then, the computer device fuses the foreground object samples with conventional video data based on the object's motion trajectory to obtain fused video data. Next, the computer device performs scene rendering optimization on the fused video data to generate video samples. The scene rendering optimization includes but is not limited to tone adjustment or lighting processing. By performing post-processing such as tone adjustment or lighting processing on the fused video data, the obtained video samples are more like real scenes and the authenticity of the video samples is improved. Among them, the conventional video data can be considered as the original sample corresponding to the video sample. The N regular video frames that make up the regular video data are the original image samples that make up the N sample image frames of the video sample. For example, the first regular video frame among N regular video frames is the original image sample of the first sample image frame among N sample image frames, etc.

Step S302: Predict the area to be repaired of the repaired image sample based on the first area prediction model to obtain the sample predicted repair area.

In this embodiment of the present application, the computer device can input the repaired image sample into the first area prediction model for prediction, and obtain the sample predicted repair area. The sample predicted repair area is the area to be repaired in the repair image sample, for example, the area to be removed foreground. When the repaired image sample is one of the N sample image frames that make up the video sample, the computer device can input the pre-order image sample, the repaired image sample and the pre-order sample repair area into the first region prediction model to predict the repaired image The area to be repaired of the sample is obtained to obtain the predicted repair area of the sample. The preamble sample repair area is, for example, the area to be repaired in the preamble image sample. The predicted value of the pixel position in the sample prediction repair area is, for example, represented by one of the two values corresponding to the label (for example, 1).

The number of preorder image samples may be p, where p is a natural number less than or equal to the preorder quantity threshold. For the first image frame among the N sample image frames, there is no preceding image sample. The computer device may determine a sample image frame located before the repaired image sample among the N sample image frames as a preceding image sample of the repaired image sample. Alternatively, the computer device may obtain the sample image frame number of the sample image frame located before the repaired image sample among the N sample image frames. If the number of sample image frames is less than or equal to the preamble number threshold, the computer device determines the sample image frame located before the repaired image sample as the preamble image sample of the repaired image sample. If the number of sample image frames is greater than the pre-order quantity threshold, the computer device will sequentially obtain the sample image frames corresponding to the pre-order quantity threshold (pre-order quantity threshold) based on the repaired image samples among the N sample image frames. sample image frame), as the preamble image sample of the repaired image sample. Alternatively, the computer device can perform semantic analysis on the video samples to obtain sample image semantic information corresponding to N sample image frames. The computer device divides the N sample image frames into one or at least two sample clusters based on the sample image semantic information. The sample image frames included in each sample cluster are continuous in the video sample and the similarity of the sample image semantic information is greater than the image similarity. threshold. The computer device can obtain the target sample cluster where the repaired image sample is located, and determine the sample image frame located before the repaired image sample in the target sample cluster as the preceding image sample of the repaired image sample.

For example, assume that the repaired image sample is the t-th sample image frame among N sample image frames. _The repaired _image sample can _be recorded as _t-1 ), p is a natural number less than or equal to the preorder quantity threshold. That is to say, when the repaired image sample is the first image frame of the video sample, there is no preceding image sample in the repaired image sample. When the repaired image sample is the second image frame of the video sample, the repaired image sample has a preamble image sample, etc. The above-mentioned pre-order image samples (X _tp , ..., X _t-2 , X _t-1 ) are only one possible expression form. In this example, the number of preamble image samples is at least three. _Let _the preamble _sample repair area _{of the} preamble image _sample The pre-sequence sample repair area of ₁ is denoted as B _t-1 and so on. The computer device can process the preamble image sample, the repaired image sample and the preamble sample repair area of the preamble image sample, that is, (X _tp ,..., X _t-2 , X _t-1 , X _t , B _tp ,..., B _t-2 , B _t-1 ), input the first area prediction model for prediction, and obtain the sample predicted repair area, denoted as The order in which the pre-order image samples, repaired image samples and pre-order sample repair areas of the pre-order image samples are arranged can be adjusted according to the needs of the model and will not be discussed here. Make restrictions. The pre-order sample repair area refers to the pre-order repair area label of the corresponding pre-order image sample. If there is a repair area label of the first image frame and a repair area label of the key image frame in the video sample, the repair area of the previous sample of the previous regular image sample can be predicted based on the first area prediction model. The preamble regular image sample is an image frame in the preamble image sample except the first image frame and the key image frame.

Step S303: Use the first media repair model to repair the sample predicted repair area in the repaired image sample to obtain a sample optimized image corresponding to the repaired image sample.

In this embodiment of the present application, the computer device can input the sample predicted repair area and the repaired image sample into the first media repair model for repair, and obtain a sample optimized image corresponding to the repaired image sample. When the repaired image sample is one of the N sample image frames that make up the video sample, the preamble image sample, the repaired image sample, the sample prediction repair area and the preamble sample repair area can be combined, for example (X _tp ,… ,X _t-2 ,X _t-1 ,X _t ,B _tp ,…,B _t-2 ,B _t-1 , ), input the first media repair model to repair the repaired image sample, and obtain a sample optimized image corresponding to the repaired image sample. Among them, the order of arrangement of the pre-order image samples, repaired image samples, sample prediction repair areas and pre-order sample repair areas of the pre-order image samples can be adjusted according to the needs of the model, and is not limited here.

Step S304: Jointly adjust the parameters of the first region prediction model and the first media repair model based on the sample predicted repair area, repair area label, original image sample, and sample optimized image to obtain the target area corresponding to the first area prediction model. a prediction model, and a target media repair model corresponding to the first media repair model.

In this embodiment of the present application, the computer device can generate a third loss function based on the sample prediction of the repair area and the repair area label, and a fourth loss function based on the original image sample and the sample optimized image. Wherein, the third loss function can be any one of h ₁ first model loss functions, or be obtained by a combination of at least two of h ₁ first model loss functions, or be obtained by h ₁ first model loss functions Obtained by weighted combination of at least two of . h ₁ is a positive integer. The h ₁ first model loss function can include the loss function shown in formula ①:

As shown in formula ①, L _CE is used to represent a first model loss function, B _gt is used to represent the repair area label of the repair image sample (used to represent the real area to be repaired in the repair image sample), Used to represent the sample predicted repair area.

The h ₁ first model loss function can also include the loss function shown in formula ②:

As shown in formula ②, L _focal is used to represent a first model loss function, B _gt is used to represent the repair area label of the repair image sample (used to represent the real area to be repaired in the repair image sample), Used to represent the sample predicted repair area. γ is an exponential parameter, which can be obtained based on empirical values or commonly used parameter values.

Among them, the above formula ① and formula ② are examples of possible first model loss functions. h ₁ first model loss function can also include other loss functions, such as intersection over union loss (IoU loss) and generalized intersection over union loss (GIoU loss), etc., which are not mentioned here. Make restrictions.

The repaired image sample is one sample image frame among the N sample image frames that make up the video sample. Any one of the h ₁ first model loss functions can be determined as the third loss function. Alternatively, at least two first model loss functions among h ₁ first model loss functions may be combined to obtain a third loss function. Alternatively, at least two first model loss functions among h ₁ first model loss functions can be weighted and summed to obtain a third loss function. Alternatively, a regional difference loss function can be generated based on the difference data between the sample prediction repair area and the repair area label, and the second discriminator is used to perform discriminant detection on the first regional prediction model to obtain an auxiliary loss function. Based on the regional difference loss function and The auxiliary loss function generates a third loss function. Specifically, the pre-sequence sample repair area can be And the sample optimized image is input into the first region prediction model for prediction (predicting the area to be repaired in the sample optimized image), and the first prediction region is obtained. The adjacent image sample adjacent to the repaired image sample can be obtained from the pre-order image sample, and the pre-order sample repair area and the sample optimized image of the adjacent image sample are input into the first area prediction model for prediction, and the first prediction area is obtained. The pre-sequence sample repair area and the original image sample are input into the first area prediction model for prediction (predicting the area to be repaired of the original image sample), and a second prediction area is obtained. The pre-sequence sample repair area and the original image sample of the adjacent image sample can be input into the first area prediction model for prediction to obtain the second prediction area. An auxiliary loss function is generated based on the first prediction area and the second prediction area. A possible way to generate this auxiliary loss function can be found in formula ③:

As shown in formula ③, L _DS is used to represent the auxiliary loss function, D _S is used to represent the second discriminator, NetS is used to represent the first region prediction model, Y _t is used to represent the original image sample, and B _t-1 is used to represent Represents the preorder sample repair area of adjacent image samples, Used to represent sample optimization images. That is to say, NetS(Y _t ,B _t-1 ) is used to represent the second prediction area, Used to represent the first prediction area. Further, the first prediction area can be input into the second discriminator for detection to obtain the first area detection result, and the second prediction area can be input into the second discriminator for detection to obtain the second area detection result. According to the first area detection result The difference data from the second region detection results generates an auxiliary loss function. The parameters of the first region prediction model are adjusted through the relevant data of the first media repair model, so that the output results of the first region prediction model are more applicable and beneficial to the task execution of the first media repair model, and the interaction between different models is realized. It promotes optimization, improves the speed of model training, saves computing resources, and thereby improves the accuracy of data repair.

Furthermore, a regional difference loss function can be generated based on the difference data between the sample prediction repair area and the repair area label. Wherein, the regional difference loss function may be generated based on h ₁ first model loss functions. For example, the regional difference loss function can be any one of h ₁ first model loss functions, or be obtained by a combination of at least two of h ₁ first model loss functions, or be obtained by a combination of h ₁ first model loss functions. At least two weighted combinations are obtained. The third loss function can be generated based on the auxiliary loss function and the regional difference loss function.

Wherein, the third loss function can be recorded as L _seg . For example, L _seg =L _CE , or L _seg =L _focal , or L _seg =L _CE +...+L _focal , or L _seg =λL _CE +...+μL _focal , etc., which are not limited here. Among them, λ, μ, etc. are used to represent the function weight of the corresponding first model loss function. When the repaired image sample is one of the N sample image frames that make up the video sample, the third loss function may also include an auxiliary loss function, such as L _seg =L _CE +...+L _focal +L _DS , etc.

Further, a fourth loss function can be generated according to h ₂ second model loss functions. The fourth loss function may be any one of the h ₂ second model loss functions, or be obtained by a combination of at least two of the h ₂ second model loss functions, or be obtained by a combination of the h ₂ second model loss functions. At least two weighted combinations are obtained. h ₂ is a positive integer. The h ₂ second model loss function can include the loss function shown in formula ④:

As shown in formula ④, L _sec is used to represent a second model loss function, Y _t is used to represent the original image sample, Used to represent sample optimization images. "|||| ₂ " is used to represent an operation symbol.

The h ₂ second model loss function can include a loss function as shown in formula ⑤:

As shown in formula ⑤, L _style is used to represent a second model loss function, and F can be a neural network, such as Visual Geometry Group Network (VGG), etc.

The h ₂ second model loss function can include a loss function as shown in formula ⑥:

As shown in formula ⑥, _Lgan is used to represent a second model loss function, and D is used to represent the first discriminator.

Among them, the above formula ④ to formula ⑥ are examples of possible second model loss functions. The h ₂ second model loss function can also include other loss functions, such as cross-entropy loss function or point-by-point difference loss function, etc., which are not limited here.

In an example fourth loss function generation method, the image difference data between the original image sample and the sample optimized image can be obtained, and the image difference loss function is generated based on the image difference data. See formula ④ and formula ⑤, etc. The original image sample is input into the first discriminator for detection, and the original discrimination result corresponding to the original image sample is obtained. The sample optimized image is input into the first discriminator for detection, and the optimized discrimination result corresponding to the sample optimized image is obtained. According to the original discrimination result and optimize the discrimination results to generate a discrimination loss function. It can be seen as shown in formula ⑥, where D(Y _t ) is used to represent the original discrimination result, Used to represent optimization discrimination results. The image difference loss function and the discrimination loss function are combined to obtain the fourth loss function. Among them, the fourth loss function can be recorded as L _input , for example, L _input =L _sec , or L _input =L _style , or L _input =L _sec +L _gan , etc.

Furthermore, the third loss function and the fourth loss function can be functionally combined to obtain a joint loss function, denoted as L _ALL . The parameters of the first region prediction model and the first media repair model are jointly adjusted through a joint loss function to obtain a target region prediction model corresponding to the first region prediction model and a target media repair model corresponding to the first media repair model.

In the embodiment of the present application, the repaired image sample to be repaired, the repaired area label corresponding to the repaired image sample, and the original image sample can be obtained; the first area prediction model is used to predict the area to be repaired of the repaired image sample, and the sample predicted repaired area is obtained ;Use the first media repair model to repair the sample predicted repair area in the repaired image sample to obtain the sample optimized image corresponding to the repaired image sample; based on the sample predicted repair area, repair area label, original image sample and sample optimized image, perform The parameters of the first region prediction model and the first media repair model are jointly adjusted to obtain a target region prediction model corresponding to the first region prediction model and a target media repair model corresponding to the first media repair model. Further, the image can be repaired based on the target area prediction model and the target media repair model. Through the above process, the joint training and use of multiple tasks is achieved to achieve mutual adjustment and promotion between different tasks, fully learn the complementary information and similar information in different tasks, and obtain the effect of mutual gain, that is to say, Different tasks can provide each other with enhanced effective information to promote model performance of different tasks, mutually improve the accuracy of the output results of different models, which is beneficial to the design and effect of the model, can improve the efficiency of model training, and save calculations resources while improving the accuracy of data repair.

The computer device can directly obtain the first regional prediction model and the first media repair model, or can perform preliminary adjustments to obtain the first regional prediction model and the first media repair model. Specifically, the computer device can obtain the second regional prediction model and the second media repair model, determine the second regional prediction model as the first regional prediction model, and determine the second media repair model as the first media repair model; or, it can The repaired image sample is used to adjust the parameters of the second region prediction model to obtain the first region prediction model, and the repaired image sample is used to adjust the parameters of the second media repair model to obtain the first media repair model, etc. The number of the first region prediction models may be d, and d is a positive integer.

The process of obtaining the first region prediction model and the first media repair model through preliminary adjustments can be seen in Figure 4. Figure 4 is a schematic diagram of a multi-step training method provided by an embodiment of the present application. As shown in Figure 4, the process may include the following steps:

Step S401: Obtain the repaired image sample to be repaired, the repair area label corresponding to the repaired image sample, and the original image sample.

In this embodiment of the present application, reference may be made to the relevant description of step S301 in Figure 3 , which will not be described again here.

Step S402: Perform preliminary adjustments to obtain a first regional prediction model and a first media repair model.

In this embodiment of the present application, the computer device can acquire the second regional prediction model and the second media repair model, use the repaired image sample to adjust parameters of the second regional prediction model, and obtain the first regional prediction model, and use the repaired image sample to adjust the parameters of the second regional prediction model. The parameters of the second media repair model are adjusted to obtain the first media repair model, etc. The second regional prediction model is, for example, the initial regional prediction model. The second media repair model is, for example, the initial media repair model.

Specifically, the repaired image sample is input into the second area prediction model for prediction, and the initial predicted repaired area in the repaired image sample is obtained; the first loss function is generated based on the initial predicted repaired area and the repaired area label, and the second loss function is calculated through the first loss function. The parameters of the regional prediction model are adjusted to obtain the first regional prediction model. Generating a first loss function based on the initial predicted repair area and the repair area label includes: determining the real area to be repaired in the repair image sample based on the repair area label, and based on the real area to be repaired and the initial area to be repaired. Predict the repair area and generate the first loss function. Wherein, the generation of the first loss function can refer to the generation method of the third loss function. The third loss function is obtained by predicting the repair area based on the repair area label and the sample, while the first loss function is obtained based on the repair area label and the initial prediction of the repair area, that is, the sample in the third loss function predicts the repair area. Changing to the initial predicted repair area can represent how the first loss function is generated. Input the repaired image sample and the initial predicted repair area into the second media repair model for repair, and obtain the initial optimized image corresponding to the repaired image sample; generate a second loss function based on the initial optimized image and the original image sample, and use the second loss function to The parameters of the second media repair model are adjusted to obtain the first media repair model. For the generation of the second loss function, please refer to the generation method of the fourth loss function, where the fourth loss function is obtained based on the sample optimized image and the original image sample, and the second loss function is based on the initial optimized image and the original image sample. what you get.

The number of the first region prediction models may be d, and d is a positive integer. For example, the first region prediction model may include a region separation model and a region identification model. The repaired image sample can be input into the initial region separation model for prediction to obtain a binary prediction image, and the separated repair area can be obtained from the binary prediction image. When inputting the repaired image sample into the initial region separation model, you can also input the pre-order repair area and the pre-order image frame at the same time. In the binary prediction image, the value of each pixel position in the area to be repaired can be represented by one of the two values (for example, 1), and the value of each pixel position in the remaining areas is represented by the other of the two values. value (for example, 0). Input the repaired image sample into the initial area recognition model for prediction, obtain the predicted border in the repaired image sample, and determine the area corresponding to the predicted border in the repaired image sample as the identified repair area. The first area loss function is generated based on the separation of the repair area and the repair area label, the second area loss function is generated based on the identification of the repair area and the repair area label, and the third area loss function is generated based on the separation of the repair area and the identification of the repair area. According to the first regional loss function, the second regional loss function and the third regional loss function, the parameters of the initial regional separation model and the initial regional identification model are jointly adjusted to obtain the regional separation model corresponding to the initial regional separation model, and the initial regional The region recognition model corresponding to the recognition model. The d first region prediction models may include any one or more of a region separation model, a region recognition model, an object detection model, etc. Since the function of the d first area prediction models is to identify the areas that need to be repaired in the repaired image samples, in theory, the results obtained by each first area prediction model for the repaired image samples have certain similarities. , the d first region prediction models can be jointly trained to make mutual adjustments based on the prediction results, thereby improving the prediction accuracy of the regions that need to be repaired.

Step S403: Use the first area prediction model to predict the area to be repaired of the repaired image sample, and obtain the sample predicted repair area.

In this embodiment of the present application, reference may be made to the relevant description of step S302 in Figure 3 , which will not be described again here.

Step S404: Use the first media repair model to repair the sample predicted repair area in the repaired image sample to obtain a sample optimized image corresponding to the repaired image sample.

In this embodiment of the present application, reference may be made to the relevant description of step S303 in Figure 3 , which will not be described again here.

Step S405: Predict the repair area, the repair area label, the original image sample and the sample optimized image based on the sample, and predict the first area The parameters of the model and the first media repair model are jointly adjusted to obtain a target area prediction model corresponding to the first area prediction model and a target media repair model corresponding to the first media repair model.

In this embodiment of the present application, reference may be made to the relevant description of step S304 in Figure 3 , which will not be described again here.

Further, please refer to FIG. 5 , which is a flow chart of a data repair method provided by an embodiment of the present application. As shown in Figure 5, the method may include the following steps:

Step S501: Obtain the image frame to be repaired, predict the image frame to be repaired based on the target area prediction model, and obtain the area to be repaired of the image frame to be repaired.

In this embodiment of the present application, the computer device can input the image frame to be repaired into the target area prediction model for prediction, and obtain the area to be repaired of the image frame to be repaired. Among them, in a region prediction method, k pooling parameters can be used in the target region prediction model to perform pooling processing on the image frames to be repaired respectively, and k pooling features corresponding to the image frames to be repaired are obtained, and k is Positive integer. Perform convolution processing on k pooled features respectively to obtain k convolution features. Perform feature fusion prediction on k convolution features to obtain the area to be repaired in the image frame to be repaired.

For example, please refer to FIG. 6 , which is a schematic diagram of a region prediction method provided by an embodiment of the present application. As shown in Figure 6, the computer device can input the image frame 601 to be repaired into the target area prediction model to obtain the initial image features 602. The computer equipment uses k pooling parameters to perform pooling processing on the initial image features 602 of the image frame to be repaired, and obtains k pooling features corresponding to the image frame to be repaired, such as pooling features 6031, pooling features 6032, and pooling features. Features 6033, etc. Further, the computer device can perform convolution processing on the k pooling features respectively to obtain k convolution features, such as the convolution feature 6041 corresponding to the pooling feature 6031 and the convolution feature 6042 corresponding to the pooling feature 6032. And the convolution feature 6043 corresponding to the pooling feature 6033, etc. The computer equipment can perform feature fusion prediction on k convolution features to obtain the area to be repaired of the image frame to be repaired. Specifically, the computer device can perform upsampling processing on k convolution features based on the initial feature size of the initial image feature 602 to obtain upsampling features corresponding to the k convolution features. The computer device performs feature fusion on the initial image feature 602 and k upsampling features to obtain a fused feature 605. Alternatively, the computer device performs feature fusion on the k upsampled features to obtain the fused feature 605. Predict the fused features 605 to obtain prediction results 606. The prediction result 606 includes the area to be repaired 6061 of the image frame to be repaired.

In a region prediction method, the initial image features of the image frame to be repaired can be obtained through the target region prediction model, and the initial image features are convolved to obtain the initial convolution features. The initial convolutional features are pooled to obtain the encoded pooling features to increase the receptive field. Among them, the receptive field refers to the area affected by a certain point on the feature map in the input space. For example, the pixel points on the feature map are mapped back to the size of the area on the input image. Further, the encoded pooling features are deconvolved to obtain the decoded convolution features. The decoded convolution features are upsampled to obtain the prediction feature map of the image frame to be repaired. The prediction feature map is activated to obtain the area to be repaired of the image frame to be repaired.

In a region prediction method, the initial image features of the image frame to be repaired can be obtained through the target region prediction model, and the initial image features are convolved to obtain the initial convolution features. The initial convolution feature is pooled to obtain the encoded pooling feature. Perform continuous convolution processing on the coding pooling features, that is, perform convolution processing on the coding pooling features sequentially through r convolution layers, and predict the area to be repaired of the image frame to be repaired, where r is a positive integer. For example, see FIG. 7 , which is a schematic diagram of another regional prediction method provided by an embodiment of the present application. As shown in Figure 7, the computer device can obtain the initial image feature 701 of the image frame to be repaired, and perform convolution processing on the initial image feature 701 to obtain the initial convolution feature 702. The computer device can then pool the initial convolutional features to obtain encoded pooled features 703. Then the computer device performs continuous convolution processing on the encoded pooling features, that is, through r convolution layers, sequentially The coding pooling feature is subjected to convolution processing, and the region to be repaired 7041 of the image frame 704 to be repaired is predicted.

In a region prediction method, the initial image features of the image frame to be repaired can be obtained through the target region prediction model, and s convolution sizes are used to perform atrous convolution sampling on the initial image features to obtain s atrous convolution features. s is a positive integer. Perform feature fusion on s dilated convolution features to obtain dilated fusion features. Multi-scale feature extraction is performed on the hole fusion feature to obtain global features and local features. Prediction is performed based on global features and local features to obtain the area to be repaired in the image frame to be repaired.

The above are just a few examples of region prediction methods. Other methods can also be used to predict the region to be repaired in the image frame to be repaired, and are not limited here.

The image frame to be repaired is one of the M image frames that make up the video data, and M is a positive integer. The computer device can obtain the pre-order image frame of the image frame to be repaired in the M image frames, and obtain the pre-order repair area corresponding to the pre-order image frame; input the pre-order repair area, the pre-order image frame and the pre-order image frame to the target The area prediction model predicts and obtains the area to be repaired corresponding to the image frame to be repaired. The number of pre-order image frames may be a natural number less than or equal to the pre-order number threshold. This is because there is no preceding image frame for the first image frame among the M image frames. Specifically, the computer device may determine the image frame located before the image frame to be repaired among the M image frames as the preceding image frame of the image frame to be repaired. Alternatively, the computer device may obtain the image frame number of the image frame located before the image frame to be repaired among the M image frames. If the number of image frames is less than or equal to the preamble number threshold, the computer device determines the image frame located before the image frame to be repaired as the preamble image frame of the image frame to be repaired. If the number of image frames is greater than the previous number threshold, then among the M image frames, based on the image frame to be repaired, the computer device sequentially acquires the image frames corresponding to the previous number threshold as the image frame to be repaired. Preamble image frame. Alternatively, the computer device can perform semantic analysis on the video samples to obtain image semantic information corresponding to the M image frames. The computer device divides the M image frames into one or at least two image clusters based on the image semantic information, the image frames included in each image cluster are continuous in the video sample and the similarity of the image semantic information is greater than the image similarity threshold. The computer device can obtain the target image cluster in which the image frame to be repaired is located, and determine the image frame in the target image cluster that is located before the image frame to be repaired as the preceding image frame of the image frame to be repaired.

Specifically, the computer device can input the previous repair area, the previous image frame, and the image frame to be repaired into the target area prediction model. In one area prediction method, the image frame to be repaired is predicted based on the image continuity between the previous image frame and the image frame to be repaired through the target area prediction model, and the initial prediction area corresponding to the image frame to be repaired is obtained. Through the target area prediction model, based on the regional continuity of the pre-order repair area, the initial prediction area is adjusted to obtain the area to be repaired corresponding to the image frame to be repaired. For example, in the target area prediction model, k pooling parameters are used to perform pooling processing on the pre-order repair area, the pre-order image frame and the image frame to be repaired respectively, and the pre-order repair area, the pre-order image frame and the image frame to be repaired are obtained. K pooling features corresponding to the image frames respectively, k is a positive integer; perform convolution processing on the k pooling features respectively to obtain k convolution features; perform feature fusion prediction on the k convolution features to obtain the image to be repaired The area of the frame to be repaired.

Alternatively, the computer device can use any of the above-mentioned area prediction methods to predict the area to be repaired corresponding to the image frame to be repaired. Specifically, in any region prediction method, when the computer device obtains the initial image features of the image frame to be repaired, the pre-repair region, the pre-order image frame and the image frame to be repaired can be processed through the target region prediction model. Feature fusion extraction is used to obtain initial image features. For example, the computer device can obtain the feature maps of the pre-order repair area, the pre-order image frame and the image frame to be repaired respectively, and perform feature fusion processing on the feature maps of the pre-order repair area, the pre-order image frame and the image frame to be repaired to obtain the initial Image features. Alternatively, the computer device can splice the pre-order repair area, the pre-order image frame and the image frame to be repaired to obtain the input data and obtain the initial image features of the input data.

The number of target area prediction models can be d, where d is a positive integer, such as a target area separation model or a target area identification model. The computer device can respectively predict the individual prediction areas of the image frame to be repaired based on d target area prediction models, and predict the d individual prediction areas. The areas are fused and adjusted to obtain the area to be repaired of the image frame to be repaired.

Step S502: Repair the area to be repaired in the image frame to be repaired based on the target media repair model to obtain an optimized image frame corresponding to the image frame to be repaired.

In this embodiment of the present application, the target area prediction model and the target media repair model are obtained through joint training. The computer device can input the image frame to be repaired and the area to be repaired into the target area prediction model for repair, and obtain an optimized image frame corresponding to the image frame to be repaired. Specifically, the computer device can determine the effective area in the image frame to be repaired based on the area to be repaired through the target media repair model, and perform repair processing on the area to be repaired based on the effective image information in the effective area to obtain the image frame to be repaired. The corresponding optimized image frame. Alternatively, the computer device can obtain the image features to be repaired of the image frame to be repaired through the target media repair model, perform feature analysis on the image features to be repaired based on the area to be repaired, and obtain the semantic features to be repaired and the rendering to be repaired of the image frame to be repaired. Features etc. The computer device can perform repair processing on the semantic features to be repaired and the rendering features to be repaired, to obtain optimized semantic features and optimized rendering features. The computer equipment can perform feature fusion processing on the optimized semantic features and the optimized rendering features to obtain an optimized feature map, and convert the optimized feature map into an optimized image frame. Among them, the semantic features to be repaired refer to the relevant features used to represent the image content in the image frame to be repaired. The rendering features to be repaired refer to related features used to represent the distribution and changes of lighting, hue, etc. in the image frame to be repaired.

Among them, the image frame to be repaired is one of the M image frames that make up the video data, and M is a positive integer. The pre-order image frame, the image frame to be repaired, the pre-order repair area and the area to be repaired can be input into the target media repair model for repair, and an optimized image frame of the image frame to be repaired can be obtained. In one repair method, the pre-order image frame, the image frame to be repaired, the pre-order repair area and the area to be repaired can be input into the target media repair model; in the target media repair model, the pre-order image frame and the pre-order repair area are Combine to obtain a pre-order combined image; obtain (for example, extract) the pixel feature map and semantic feature map of the pre-order combined image from the pre-order combined image, and obtain (e.g., extract) from the image frame to be repaired The pixel feature map and semantic feature map of the image frame to be repaired; the pixel feature map of the pre-order combined image and the pixel feature map of the image frame to be repaired are feature fused to obtain the attention map; based on the attention map, the pixel feature map of the pre-order combined image is obtained Obtain semantic repair data from the semantic feature map; obtain the semantic feature map of the area to be repaired from the semantic feature map of the image frame to be repaired, perform repair processing on the semantic feature map of the area to be repaired based on the semantic repair data, and obtain the optimization of the image frame to be repaired image frame.

For example, please refer to FIG. 8 , which is a schematic diagram of a repair method provided by an embodiment of the present application. As shown in Figure 8, in the target media repair model, the computer device can combine the pre-order image frame and the pre-order repair area to obtain a pre-order combined image 802, such as the pre-order combined image 8021 and the pre-order combined image shown in Figure 8. Sequentially combined image 8022, etc. Obtain the pixel feature map and semantic feature map of the preamble combined image 802, such as the pixel feature map and semantic feature map of the preamble combined image 8021, and the pixel feature map and semantic feature map of the preamble combined image 8022, etc. Obtain the pixel feature map 8031 and semantic feature map 8032 of the image frame 801 to be repaired. Feature fusion is performed on the pixel feature map 8031 of the pre-order combined image 802 and the pixel feature map 8031 of the image frame 801 to be repaired to obtain an attention map; according to the attention map, the semantic repair data is obtained from the semantic feature map of the pre-order combined image. Obtain the semantic feature map of the area to be repaired in the image frame to be repaired 804, perform repair processing on the semantic feature map of the area to be repaired based on the semantic repair data, and obtain the optimized image frame of the image frame to be repaired 805.

In one repair method, the computer device can obtain the forward optical flow and reverse optical flow of the pre-order image frame and the adjacent frames in the image frame to be repaired and a group of non-adjacent frames through the target media repair model, based on the to-be-repaired image frame. The forward optical flow and reverse optical flow are repaired in the repair area to obtain an optimized optical flow field. Further, the computer device can propagate candidate pixels for the pixels to be repaired in the area to be repaired based on the optical flow trajectory in the optimized optical flow field. Specifically, the optimized optical flow field may include a forward optical flow field and a reverse optical flow field. The computer equipment obtains a set of candidate pixels by connecting the forward optical flow field and the reverse optical flow field in series, and optimizes the candidate pixel set based on the optical flow trajectory to obtain the pixels to be repaired in the area to be repaired. Candidate pixels for points. Further, the candidate pixels of the pixels to be repaired in the area to be repaired can be fused with the effective pixels in the image frame to be repaired, and the pixels to be repaired in the area to be repaired can be pixel optimized to realize the repair of the area to be repaired. Obtain the optimized image frame corresponding to the image frame to be repaired.

In one repair method, see Figure 9 , which is a schematic diagram of another repair method provided by an embodiment of the present application. As shown in Figure 9, the computer device can obtain the image frame sequence 901 through the target media repair model, including b image frames, such as the preamble image frame 9011, the preamble image frame 9012, and the image frame to be repaired 9013 in Figure 9. , b is a positive integer. Using u block sizes, the image frame sequence 901 is processed respectively to obtain block fusion features corresponding to the u block sizes respectively, where u is a positive integer. Perform convolution fusion processing on u block fusion features to obtain sequence fusion features. Perform feature restoration processing on the sequence fusion features to obtain an optimized image frame sequence 904 corresponding to the image frame sequence 901, such as the optimization corresponding to the pre-order image frame 9011. The image frame 9041, the optimized image frame 9042 corresponding to the previous image frame 9012, the optimized image frame 9043 corresponding to the image frame 9013 to be repaired, and so on. Specifically, the computer device can obtain the first feature map, the second feature map, and the content feature map respectively corresponding to each image frame in the image frame sequence 901. Among them, the first feature map and the second feature map are used for attention processing. Take a block size as an example. The i-th block size can be used to obtain the first feature map 9021 from the first feature maps corresponding to the d image frames, and the i-th block size can be used to obtain the first feature map from the second feature map corresponding to the d image frames. Two block features 9022. Among them, the first feature 9021 can be considered to be composed of b*h/r ₁ *w/r ₂ features r ₁ *r ₂ , and the second feature 9022 can be considered to be composed of b*h/r ₁ *w /r consists of ₂ features of r ₁ * r ₂ . Among them, i is a positive integer less than or equal to u, h is the height of the image frame, w is the width of the image frame, and r ₁ * r ₂ refers to the corresponding block size. Regional similarity 903 is obtained through the first block feature 9021 and the second block feature 9022. Using the i-th block size, content block features 9023 are obtained from the content feature maps corresponding to the d image frames. Perform feature fusion processing on the regional similarity 903 and the content block feature 9023 to obtain the block fusion feature corresponding to the i-th block size. In the same way, block fusion features corresponding to u block sizes can be obtained.

Among them, the above are just a few examples of repair methods. Other methods can also be used to repair the image frame to be repaired to obtain an optimized image frame, which is not limited here.

To predict the area to be repaired of the repaired image sample and obtain the sample to predict the repaired area, the area prediction method shown in step S501 in Figure 5 can also be used; to repair the repaired image sample and obtain the sample optimized image, the method shown in Figure 5 can be used The repair method shown in step S502 in 5, etc.

In the embodiment of the present application, the image frame to be repaired can be obtained, the image frame to be repaired is predicted based on the target area prediction model, and the area to be repaired in the image frame to be repaired is obtained; the area to be repaired in the image frame to be repaired is based on the target media repair model Repair is performed to obtain the optimized image frame corresponding to the image frame to be repaired; the target area prediction model and the target media repair model are obtained through joint training. Through the above process, joint training and use of multi-tasks are achieved to achieve mutual adjustment and promotion between different tasks, fully learn complementary information and similar information in different tasks, obtain mutual gain effects, and improve model training. efficiency, saving computing resources. Since different tasks can provide each other with enhanced effective information to promote model performance of different tasks, mutually improve the accuracy of the output results of different models, it is conducive to the improvement of model design and effects, thereby improving the accuracy of data repair.

In this application, the computer device for model training (i.e., the computer device shown in Figure 3) and the computer device for model prediction (i.e., the computer device shown in Figure 5) may be the same device, or they may be different devices.

This application can be applied to any scenario that requires media repair, such as a video data repair scene or an image repair scene, etc. For example, the computer device can respond to a repair request for video data, obtain M image frames that make up the video data, use the above-mentioned processes shown in Figure 5 to repair the M image frames, and obtain the corresponding M image frames respectively. of optimized image frames, converting M Optimized image frames are composed of optimized video data. When the repair request for video data is sent by the business device to the computer device, when the computer device obtains the optimized video data, it can also send the optimized video data to the business device. Alternatively, assuming that the computer device obtains an upload request for video data, if an abnormality is detected in the video data, M image frames that make up the video data can be obtained, and the M images can be processed using the processes shown in Figure 5 above. The frames are repaired to obtain optimized image frames corresponding to the M image frames, the M optimized image frames are composed of optimized video data, and the optimized video data is uploaded.

Further, please refer to FIG. 10 , which is a schematic diagram of a data repair device provided by an embodiment of the present application. The data repair device may be a computer program (including program code, etc.) running in a computer device. For example, the data repair device may be an application software; the device may be used to perform corresponding steps in the method provided by the embodiments of the present application. . As shown in Figure 10, the data repair apparatus 1000 can be used in the computer equipment in the embodiment corresponding to Figure 3. The device may include: a sample acquisition module 11 , a sample area prediction module 12 , a sample repair module 13 and a model adjustment module 14 .

The sample acquisition module 11 is used to acquire the repaired image sample to be repaired, the repair area label corresponding to the repaired image sample, and the original image sample;

The sample area prediction module 12 is configured to use the first area prediction model to predict the area to be repaired of the repaired image sample, and obtain the sample predicted repair area;

The sample repair module 13 is configured to use the first media repair model to repair the sample predicted repair area in the repaired image sample, and obtain a sample optimized image corresponding to the repaired image sample;

The model adjustment module 14 is used to jointly adjust the parameters of the first regional prediction model and the first media repair model based on the sample predicted repair area, repair area label, original image sample and sample optimized image to obtain the corresponding parameters of the first regional prediction model. a target area prediction model, and a target media repair model corresponding to the first media repair model.

Among them, the device 1000 also includes: an initial prediction module 15, a first adjustment module 16, a first repair module 17, and a repair model generation module 18.

The initial prediction module 15 is used to input the repaired image sample into the second area prediction model for prediction, and obtain the initial predicted repaired area in the repaired image sample;

The first adjustment module 16 is configured to generate a first loss function based on the initial predicted repair area and the repair area label, and adjust the parameters of the second area prediction model through the first loss function to obtain the first area prediction model;

The first repair module 17 is used to input the repaired image sample and the initial predicted repair area into the second media repair model for repair, and obtain the initial optimized image corresponding to the repaired image sample;

The repair model generation module 18 is configured to generate a second loss function based on the initial optimized image and the original image sample, and adjust the parameters of the second media repair model through the second loss function to obtain the first media repair model.

Among them, the repaired image sample is one sample image frame among the N sample image frames that make up the video sample, and N is a positive integer.

The device 1000 also includes a preamble acquisition module 19, which is used to obtain the preamble image samples of the repaired image samples in the N sample image frames, and obtain the preamble sample repair area corresponding to the preamble image sample.

The sample area prediction module 12 is specifically used to input the pre-order image sample, the repair image sample and the pre-order sample repair area into the first area prediction model, predict the area to be repaired of the repair image sample, and obtain the sample predicted repair area.

The sample repair module 13 is specifically used to: input the pre-sequence image sample, the repair image sample, the sample prediction repair area and the pre-sequence sample repair area into the first media repair model to repair the repair image sample and obtain the repair image sample. Corresponding sample optimized image.

Among them, the device 1000 also includes: a trajectory generation module 20, a data fusion module 21, and a sample generation module 22.

The trajectory generation module 20 is used to obtain foreground object samples and conventional video data, perform simulated motion processing on the foreground object samples, and obtain object motion trajectories.

The data fusion module 21 is used to fuse foreground object samples and conventional video data based on object motion trajectories to obtain fused video data.

The sample generation module 22 is used to perform scene rendering optimization on the fused video data and generate video samples.

Among them, the model adjustment module 14 includes: a first loss generation unit 141, a second loss generation unit 142, a loss combination unit 143, and a joint adjustment unit 144.

The first loss generation unit 141 is used to predict the repair area and the repair area label according to the sample to generate a third loss function;

The second loss generation unit 142 is configured to generate a fourth loss function based on the original image sample and the sample optimized image;

The loss combining unit 143 is used to functionally combine the third loss function and the fourth loss function to obtain a joint loss function;

The joint adjustment unit 144 is configured to jointly adjust the parameters of the first region prediction model and the first media repair model through a joint loss function to obtain a target region prediction model corresponding to the first region prediction model and a target region prediction model corresponding to the first media repair model. target media repair model.

The preamble acquisition module 19 is also used to obtain the preamble image samples of the repaired image samples in the N sample image frames, and obtain the preamble sample repair area corresponding to the preamble image samples.

The first loss generation unit 141 includes: a first prediction sub-unit 1411, a second prediction sub-unit 1412, an auxiliary loss generation sub-unit 1413, and a region loss generation sub-unit 1414.

The first prediction subunit 1411 is used to input the pre-sample repair area and the sample optimized image into the first area prediction model, predict the area to be repaired in the sample optimized image, and obtain the first prediction area;

The second prediction sub-unit 1412 is used to input the repair area of the previous sample and the original image sample into the first area prediction model, predict the area to be repaired of the original image sample, and obtain the second prediction area;

The auxiliary loss generation subunit 1413 is used to generate an auxiliary loss function according to the first prediction area and the second prediction area.

The regional loss generation subunit 1414 is used to predict the difference data between the repair area and the repair area label based on the sample, and generate a regional difference loss function.

The first loss combination subunit 1415 is used to generate a third loss function based on the auxiliary loss function and the regional difference loss function.

The second loss generation unit 142 includes: an image loss generation subunit 1421, a result determination subunit 1422, a determination loss generation subunit 1423, and a second loss combination subunit 1424.

The image loss generation subunit 1421 is used to determine image difference data between the original image sample and the sample optimized image, and generate an image difference loss function based on the image difference data.

The result discrimination subunit 1422 is used to input the original image sample into the first discriminator for detection to obtain the original discrimination result corresponding to the original image sample, and input the sample optimized image into the first discriminator for detection to obtain the optimization corresponding to the sample optimized image. Discrimination results.

The discrimination loss generation subunit 1423 is used to generate a discrimination loss function based on the original discrimination result and the optimized discrimination result.

The second loss combination subunit 1424 is used to combine the image difference loss function and the discrimination loss function to obtain a fourth loss function.

Among them, the first regional prediction model includes a regional separation model and a regional identification model. The device 1000 also includes: a separation prediction module 23, Identify the prediction module 24, the loss acquisition module 25, and the model generation module 26.

The separation prediction module 23 is used to input the repaired image sample into the initial region separation model for prediction, obtain a binary prediction image, and obtain the separation repair region from the binary prediction image.

The recognition prediction module 24 is used to input the repaired image sample into the initial area recognition model for prediction, obtain the predicted border in the repaired image sample, and determine the area corresponding to the predicted border in the repaired image sample as the identified repair area.

The loss acquisition module 25 is configured to generate a first area loss function based on the separated repair area and the repair area label, generate a second area loss function based on the identified repair area and the repair area label, and generate a third area loss function based on the separated repair area and the identified repair area. .

The model generation module 26 is used to jointly adjust the parameters of the initial region separation model and the initial region identification model according to the first region loss function, the second region loss function and the third region loss function to obtain the region corresponding to the initial region separation model. separation model, and the region recognition model corresponding to the initial region recognition model.

Embodiments of the present application provide a data repair device, which can obtain a repaired image sample to be repaired, a repair region label corresponding to the repaired image sample, and an original image sample; and use a first region prediction model to predict the repaired image sample to be repaired. area to obtain the sample predicted repair area; use the first media repair model to repair the sample predicted repair area in the repair image sample, and obtain the sample optimized image corresponding to the repair image sample; predict the repair area, repair area label, and original image based on the sample Samples and sample optimization images are used to jointly adjust the parameters of the first area prediction model and the first media repair model to obtain the target area prediction model corresponding to the first area prediction model and the target media repair corresponding to the first media repair model. Model. Further, the image can be repaired based on the target area prediction model and the target media repair model. Through the above process, joint training and use of multi-tasks are achieved to achieve mutual adjustment and promotion between different tasks, fully learn complementary information and similar information in different tasks, obtain mutual gain effects, and improve model training. efficiency, saving computing resources. Since different tasks can provide each other with enhanced effective information to promote model performance of different tasks, mutually improve the accuracy of the output results of different models, it is conducive to the improvement of model design and effects, thereby improving the accuracy of data repair.

Further, please refer to Figure 11, which is a schematic diagram of another data repair device provided by an embodiment of the present application. The data repair device may be a computer program (including program code, etc.) running in the computer device. For example, the data repair device can be an application software. The device can be used to perform corresponding steps in the method provided by the embodiments of the present application. As shown in Figure 11, the data repair device 1100 can be used in the computer equipment in the embodiment corresponding to Figure 5. Specifically, the device can include: an image acquisition module 31, a region prediction module 32 and a data repair module 33.

The image acquisition module 31 is used to acquire image frames to be repaired;

The area prediction module 32 is used to predict the image frame to be repaired based on the target area prediction model, and obtain the area to be repaired of the image frame to be repaired;

The data repair module 33 is used to repair the area to be repaired in the image frame to be repaired based on the target media repair model, and obtain the optimized image frame corresponding to the image frame to be repaired, wherein the target area prediction model and the target media repair model are jointly trained owned.

Among them, the image frame to be repaired is one of the M image frames that make up the video data, and M is a positive integer.

The region prediction module 32 includes: a preorder acquisition unit 321 and a region prediction unit 322.

The preamble acquisition unit 321 is used to obtain the preamble image frame of the image frame to be repaired among the M image frames, and obtain the preamble repair area corresponding to the preamble image frame.

The area prediction unit 322 is used to input the previous repair area, the previous image frame and the image frame to be repaired into the target area prediction model for prediction, Obtain the area to be repaired corresponding to the image frame to be repaired.

The data repair module 33 is specifically used to: input the pre-order image frame, the image frame to be repaired, the pre-order repair area and the area to be repaired into the target media repair model for repair, and obtain an optimized image frame of the image frame to be repaired.

Among them, the region prediction unit 322 includes: a data input sub-unit 3221, an initial prediction sub-unit 3222, and a region adjustment sub-unit 3223.

The data input subunit 3221 is used to input the previous repair area, the previous image frame and the image frame to be repaired into the target area prediction model.

The initial prediction subunit 3222 is used to predict the image frame to be repaired based on the image continuity between the previous image frame and the image frame to be repaired through the target area prediction model, and obtain the initial prediction area corresponding to the image frame to be repaired.

The area adjustment subunit 3223 is used to adjust the initial prediction area through the target area prediction model and based on the area continuity of the previous repair area to obtain the area to be repaired corresponding to the image frame to be repaired.

Among them, the data repair module 33 includes: a model input unit 331, an image combination unit 332, an atlas acquisition unit 333, a feature fusion unit 334, a repair acquisition unit 335, and an image repair unit 336.

The model input unit 331 is used to input the previous image frame, the image frame to be repaired, the previous repair area, and the area to be repaired into the target media repair model.

The image combination unit 332 is used to combine the pre-order image frames and the pre-order repair area in the target media repair model to obtain the pre-order combined image.

The map acquisition unit 333 is configured to obtain the pixel feature map and semantic feature map of the previous combined image from the previous combined image, and obtain the pixel feature map and semantic feature map of the image frame to be repaired from the image frame to be repaired.

The feature fusion unit 334 is used to perform feature fusion on the pixel feature map of the pre-order combined image and the pixel feature map of the image frame to be repaired to obtain an attention map.

The repair acquisition unit 335 is configured to obtain semantic repair data from the semantic feature map of the pre-order combined image according to the attention map.

The image repair unit 336 is configured to obtain the semantic feature map of the region to be repaired from the semantic feature map of the image frame to be repaired, and perform repair processing on the semantic feature map of the region to be repaired based on the semantic repair data to obtain an optimized image frame of the image frame to be repaired.

Among them, the region prediction module 32 includes: a data pooling unit 323, a feature convolution unit 324, and a feature prediction unit 325.

The data pooling unit 323 is used to use k pooling parameters in the target area prediction model to perform pooling processing on the pre-order repair area, the pre-order image frame and the image frame to be repaired, respectively, to obtain the pre-order repair area, the pre-order repair area, and the pre-order image frame. The k pooling features corresponding to the image frame and the image frame to be repaired respectively, k is a positive integer.

The feature convolution unit 324 is used to perform convolution processing on k pooled features respectively to obtain k convolution features.

The feature prediction unit 325 is used to perform feature fusion prediction on k convolution features to obtain the area to be repaired of the image frame to be repaired.

Referring to Figure 12, Figure 12 is a schematic structural diagram of a computer device provided by an embodiment of the present application. As shown in Figure 12, the computer device in this embodiment of the present application may include: one or more processors 1201, a memory 1202, and an input and output interface 1203. The processor 1201, the memory 1202 and the input/output interface 1203 are connected through a bus 1204. The memory 1202 is used to store computer programs, which include program instructions. The input and output interface 1203 is used to receive data and output data, such as for data interaction between computer equipment and terminal equipment; the processor 1201 is used to execute the memory 1202 Stored program instructions.

Among them, the processor 1201 can perform the following operations:

Obtain the image frame to be repaired, predict the image frame to be repaired based on the target area prediction model, and obtain the area to be repaired of the image frame to be repaired;

When predicting the image frame to be repaired based on the target area prediction model and obtaining the area to be repaired of the image frame to be repaired, the processor 1201 can perform the following operations:

Obtain the preceding image frame of the image frame to be repaired from M image frames, and obtain the preceding repair area corresponding to the preceding image frame;

Input the pre-order repair area, the pre-order image frame and the image frame to be repaired into the target area prediction model for prediction, and obtain the area to be repaired corresponding to the image frame to be repaired;

Repair the area to be repaired in the image frame to be repaired based on the target media repair model, and obtain the optimized image frame corresponding to the image frame to be repaired, including:

The pre-order image frame, the image frame to be repaired, the pre-order repair area and the area to be repaired are input into the target media repair model for repair, and an optimized image frame of the image frame to be repaired is obtained.

Among them, when the pre-order repair area, the pre-order image frame and the image frame to be repaired are input into the target area prediction model for prediction, and the area to be repaired corresponding to the image frame to be repaired is obtained, the processor 1201 can perform the following operations:

Input the pre-order repair area, pre-order image frame and image frame to be repaired into the target area prediction model;

In the target area prediction model, based on the image continuity between the previous image frame and the image frame to be repaired, the image frame to be repaired is predicted to obtain the initial prediction area corresponding to the image frame to be repaired;

In the target area prediction model, based on the regional continuity of the pre-order repair area, the initial prediction area is adjusted to obtain the area to be repaired corresponding to the image frame to be repaired.

Among them, when the pre-order image frame, the image frame to be repaired, the pre-order repair area and the area to be repaired are input into the target media repair model for repair, and the optimized image frame of the image frame to be repaired is obtained, the processor 1201 can perform the following operations:

Input the pre-order image frame, the image frame to be repaired, the pre-order repair area and the area to be repaired into the target media repair model;

In the target media repair model, the pre-order image frame and the pre-order repair area are combined to obtain the pre-order combined image;

Obtain the pixel feature map and semantic feature map of the pre-order combined image from the pre-order combined image, and obtain the pixel feature map and semantic feature map of the image frame to be repaired from the image frame to be repaired;

Perform feature fusion on the pixel feature map of the pre-order combined image and the pixel feature map of the image frame to be repaired to obtain the attention map;

According to the attention map, the semantic repair data is obtained from the semantic feature map of the pre-order combined image;

The semantic feature map of the area to be repaired is obtained from the semantic feature map of the image frame to be repaired, and the area to be repaired is based on the semantic repair data. The semantic feature map is repaired to obtain the optimized image frame of the image frame to be repaired.

Wherein, when predicting the image frame to be repaired based on the target area prediction model and obtaining the area to be repaired of the image frame to be repaired, the processor 1201 can perform the following operations:

In the target area prediction model, k pooling parameters are used to pool the pre-order repair area, pre-order image frame and to-be-repaired image frame respectively to obtain the pre-order repair area, pre-order image frame and to-be-repaired image frame. The corresponding k pooling features respectively, k is a positive integer;

Perform convolution processing on k pooled features respectively to obtain k convolution features;

Perform feature fusion prediction on k convolution features to obtain the area to be repaired in the image frame to be repaired.

In some feasible implementations, the processor 1201 can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSP), special-purpose integrated processors, etc. Circuit (application specific integrated circuit, ASIC), off-the-shelf programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

The memory 1202 may include read-only memory and random access memory, and provides instructions and data to the processor 1201 and the input-output interface 1203. A portion of memory 1202 may also include non-volatile random access memory. For example, memory 1202 may also store device type information.

In specific implementation, the computer device can execute the implementation provided by each step in Figure 3 or Figure 5 through its built-in functional modules. For details, please refer to the implementation provided by each step in Figure 3 or Figure 5. I won’t go into details here.

Embodiments of the present application provide a computer device, including: a processor, an input and output interface, and a memory. The processor obtains the computer program in the memory and executes each step of the method shown in Figure 3 or Figure 5 to perform data repair. operate. The embodiment of the present application realizes that the repaired image sample to be repaired, the repaired area label corresponding to the repaired image sample and the original image sample can be obtained; the first area prediction model is used to predict the area to be repaired of the repaired image sample, and the sample predicted repaired area is obtained; Use the first media repair model to repair the sample predicted repair area in the repaired image sample to obtain the sample optimized image corresponding to the repaired image sample; based on the sample predicted repair area, repair area label, original image sample and sample optimized image, perform the repair on the first image sample Parameters of a region prediction model and a first media repair model are jointly adjusted to obtain a target region prediction model corresponding to the first region prediction model and a target media repair model corresponding to the first media repair model. Further, the image can be repaired based on the target area prediction model and the target media repair model. Through the above process, joint training and use of multi-tasks are achieved to achieve mutual adjustment and promotion between different tasks, fully learn complementary information and similar information in different tasks, obtain mutual gain effects, and improve model training. efficiency, saving computing resources. Since different tasks can provide each other with enhanced effective information to promote model performance of different tasks, mutually improve the accuracy of the output results of different models, it is conducive to the improvement of model design and effects, thereby improving the accuracy of data repair.

Embodiments of the present application also provide a computer-readable storage medium that stores a computer program. The computer program is adapted to be loaded by the processor and perform the data repair provided by each step in Figure 3 or Figure 5 For details of the method, please refer to the implementation provided by each step in Figure 3 or Figure 5, and will not be described again here. In addition, the description of the beneficial effects of using the same method will not be described again. For technical details not disclosed in the computer-readable storage medium embodiments involved in this application, please refer to the description of the method embodiments in this application. As examples, a computer program may be deployed to execute on one computer device, or on multiple computer devices located at one location, or on multiple computer devices distributed across multiple locations and interconnected by a communications network. implement.

The computer-readable storage medium may be the data repair device provided in any of the foregoing embodiments or an internal storage unit of the computer device. Elements, such as the hard drive or memory of a computer device. The computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card equipped on the computer device, Flash card, etc. Further, the computer-readable storage medium may also include both an internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been output or is to be output.

Embodiments of the present application also provide a computer program product or computer program. The computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in various optional ways in Figure 3 or Figure 5, thereby realizing the Joint training and use of multi-tasks to achieve mutual adjustment and promotion between different tasks, fully learn complementary information and similar information in different tasks, and obtain mutual gain effects, that is, different tasks can provide each other with The enhanced effective information can promote the performance of models for different tasks, mutually improve the accuracy of the output results of different models, and is conducive to the design and effect of the model, thereby improving the accuracy of data repair.

The terms “first”, “second”, etc. in the description, claims, and drawings of the embodiments of this application are used to distinguish different objects, rather than describing a specific sequence. Furthermore, the term "includes" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, device, product or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally also includes unlisted steps or modules, or optionally also includes Other step units inherent to such processes, methods, apparatus, products or equipment.

Those of ordinary skill in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of both. In order to clearly illustrate the relationship between hardware and software Interchangeability, in this description the composition and steps of each example have been generally described according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each specific application, but such implementations should not be considered beyond the scope of this application.

The methods and related devices provided by the embodiments of the present application are described with reference to the method flowcharts and/or structural schematic diagrams provided by the embodiments of the present application. Specifically, each process and/or the method flowcharts and/or structural schematic diagrams can be implemented by computer program instructions. or blocks, and combinations of processes and/or blocks in flowcharts and/or block diagrams. These computer program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data recovery device to produce a machine such that the instructions executed by the processor of the computer or other programmable data recovery device produce a use A device for realizing the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the structural diagram. These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data repair device to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction means, the instructions The device implements the functions specified in one process or multiple processes in the flowchart and/or in one block or multiple blocks in the structural diagram. These computer program instructions may also be loaded onto a computer or other programmable data recovery device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby executing on the computer or other programmable device. Instructions provide steps for implementing the functions specified in a process or processes of a flowchart and/or a block or blocks of a structural representation.

Claims

A data repair method, characterized in that the method includes:

Obtain the repaired image sample to be repaired, the repaired area label corresponding to the repaired image sample, and the original image sample;

Use the first area prediction model to predict the area to be repaired of the repaired image sample, and obtain the sample predicted repair area;

Use the first media repair model to repair the sample predicted repair area in the repaired image sample to obtain a sample optimized image corresponding to the repaired image sample;

According to the sample predicted repair area, the repair area label, the original image sample and the sample optimized image, the parameters of the first area prediction model and the first media repair model are jointly adjusted to obtain the a target area prediction model corresponding to the first area prediction model, and a target media repair model corresponding to the first media repair model.
The method of claim 1, further comprising:

Input the repaired image sample into a second area prediction model for prediction, and obtain an initial predicted repaired area in the repaired image sample;

Generate a first loss function according to the initial predicted repair area and the repair area label, and adjust parameters of the second area prediction model through the first loss function to obtain the first area prediction model;

Input the repaired image sample and the initial predicted repair area into a second media repair model for repair, and obtain an initial optimized image corresponding to the repaired image sample;

A second loss function is generated according to the initial optimized image and the original image sample, and parameters of the second media repair model are adjusted through the second loss function to obtain the first media repair model.
The method of claim 1, wherein the repaired image sample is one sample image frame among N sample image frames that make up the video sample, and N is a positive integer; the method further includes:

Obtain the preceding image sample of the repaired image sample in the N sample image frames, and obtain the preceding sample repair area corresponding to the preceding image sample;

The use of the first area prediction model to predict the area to be repaired of the repaired image sample to obtain the sample predicted repair area includes:

Input the preamble image sample, the repaired image sample and the preamble sample repair area into a first area prediction model, predict the area to be repaired of the repaired image sample, and obtain the sample predicted repair area;

The method of using the first media repair model to repair the sample predicted repair area in the repaired image sample to obtain a sample optimized image corresponding to the repaired image sample includes:

The preamble image sample, the repaired image sample, the sample predicted repair area and the preamble sample repair area are input into the first media repair model to repair the repaired image sample to obtain the repair The image sample corresponds to the sample optimized image.
The method of claim 3, further comprising:

Obtain foreground object samples and conventional video data, perform simulated motion processing on the foreground object samples, and obtain object motion trajectories;

Based on the object motion trajectory, fuse the foreground object sample with the conventional video data to obtain fused video data;

Perform scene rendering optimization on the fused video data to generate the video samples.
The method of claim 1, wherein the repair region is predicted based on the sample, the repair region label, the original image sample and the sample optimized image, and the first region prediction model and The parameters of the first media repair model are jointly adjusted to obtain a target area prediction model corresponding to the first area prediction model and a target media repair model corresponding to the first media repair model, including:

Generate a third loss function based on the sample to predict the repair area and the repair area label, and generate a fourth loss function based on the original image sample and the sample optimized image;

Functionally combine the third loss function and the fourth loss function to obtain a joint loss function;

The parameters of the first region prediction model and the first media repair model are jointly adjusted through the joint loss function to obtain a target region prediction model corresponding to the first region prediction model and the first media repair model. The target media repair model corresponding to the repair model.
The method of claim 5, wherein the repaired image sample is one sample image frame among N sample image frames that make up the video sample, and N is a positive integer; the method further includes:

Obtain the preceding image sample of the repaired image sample in the N sample image frames, and obtain the preceding sample repair area corresponding to the preceding image sample;

The predicting the repair area and the repair area label according to the sample to generate a third loss function includes:

Input the pre-sequence sample repair area and the sample optimized image into the first area prediction model, predict the area to be repaired in the sample optimized image, and obtain the first prediction area;

Input the preamble sample repair area and the original image sample into the first area prediction model, predict the area to be repaired of the original image sample, and obtain a second prediction area;

Generate an auxiliary loss function according to the first prediction area and the second prediction area;

Generate a regional difference loss function based on the difference data between the sample predicted repair area and the repair area label;

A third loss function is generated according to the auxiliary loss function and the regional difference loss function.
The method of claim 5, wherein generating a fourth loss function based on the original image sample and the sample optimized image includes:

Determine image difference data between the original image sample and the sample optimized image, and generate an image difference loss function based on the image difference data;

The original image sample is input into the first discriminator for detection, and the original discrimination result corresponding to the original image sample is obtained. The sample optimized image is input into the first discriminator for detection, and the sample optimized image is obtained. Corresponding optimization judgment results;

Generate a discrimination loss function based on the original discrimination result and the optimized discrimination result;

The image difference loss function and the discrimination loss function are combined to obtain a fourth loss function.
The method of claim 1, wherein the first region prediction model includes a region separation model and a region identification model, and the method further includes:

Input the repaired image sample into an initial region separation model for prediction to obtain a binary prediction image, and obtain the separated repair area from the binary prediction image;

Input the repaired image sample into an initial area recognition model for prediction, obtain a predicted border in the repaired image sample, and determine the area corresponding to the predicted border in the repaired image sample as the identified repair area;

A first area loss function is generated based on the separated repair area and the repair area label, a second area loss function is generated based on the identified repair area and the repair area label, and a second area loss function is generated based on the separated repair area and the identified repair area. Generate the third region loss function;

According to the first region loss function, the second region loss function and the third region loss function, the parameters of the initial region separation model and the initial region identification model are jointly adjusted to obtain the initial region The regional separation model corresponding to the separation model type, and the region recognition model corresponding to the initial region recognition model.
A data repair method, characterized in that the method includes:

Obtain the image frame to be repaired, predict the image frame to be repaired based on the target area prediction model, and obtain the area to be repaired of the image frame to be repaired;

Repair the area to be repaired in the image frame to be repaired based on the target media repair model to obtain the optimized image frame corresponding to the image frame to be repaired, wherein the target area prediction model and the target media repair model are Obtained through joint training.
The method of claim 9, wherein the image frame to be repaired is one of M image frames that make up the video data, and M is a positive integer;

The method of predicting the image frame to be repaired based on the target area prediction model and obtaining the area to be repaired of the image frame to be repaired includes: obtaining the preceding image of the image frame to be repaired from the M image frames. frame, obtain the pre-order repair area corresponding to the pre-order image frame; input the pre-order repair area, the pre-order image frame and the to-be-repaired image frame into the target area prediction model for prediction, and obtain the to-be-repaired area. Repair the area to be repaired corresponding to the image frame;

The method of repairing the area to be repaired in the image frame to be repaired based on the target media repair model to obtain the optimized image frame corresponding to the image frame to be repaired includes: combining the preceding image frame and the image frame to be repaired. The image frame, the pre-order repair area and the area to be repaired are input into the target media repair model for repair, and an optimized image frame of the image frame to be repaired is obtained.
The method according to claim 10, characterized in that the pre-order repair area, the pre-order image frame and the to-be-repaired image frame are input into a target area prediction model for prediction to obtain the to-be-repaired image. The area to be repaired corresponding to the frame includes:

Input the pre-order repair area, the pre-order image frame and the to-be-repaired image frame into a target area prediction model;

In the target area prediction model, based on the image continuity between the preceding image frame and the image frame to be repaired, the image frame to be repaired is predicted to obtain the image frame corresponding to the image frame to be repaired. Initial prediction area;

In the target area prediction model, based on the area continuity of the pre-order repair area, the initial prediction area is adjusted to obtain the area to be repaired corresponding to the image frame to be repaired.
The method of claim 10, wherein the preamble image frame, the image frame to be repaired, the preamble repair area and the to-be-repaired area are input into a target media repair model for repair, Obtaining the optimized image frame of the image frame to be repaired includes:

Input the preamble image frame, the image frame to be repaired, the preamble repair area and the to be repaired area into the target media repair model;

In the target media repair model, the preamble image frame and the preamble repair area are combined to obtain a preamble combined image;

Obtain the pixel feature map and semantic feature map of the pre-sequence combined image from the pre-sequence combined image, and obtain the pixel feature map and semantic feature map of the to-be-repaired image frame from the to-be-repaired image frame;

Perform feature fusion on the pixel feature map of the pre-order combined image and the pixel feature map of the image frame to be repaired to obtain an attention map;

According to the attention map, obtain semantic repair data from the semantic feature map of the pre-order combined image;

Obtain the semantic feature map of the region to be repaired from the semantic feature map of the image frame to be repaired, and perform repair processing on the semantic feature map of the region to be repaired based on the semantic repair data to obtain the image frame to be repaired. of optimized image frames.
The method of claim 10, wherein the step of predicting the image frame to be repaired based on a target area prediction model to obtain the area to be repaired of the image frame to be repaired includes:

In the target area prediction model, k pooling parameters are used to perform pooling processing on the pre-order repair area, the pre-order image frame and the image frame to be repaired, respectively, to obtain the pre-order repair area, the pre-order repair area, the pre-order image frame and the to-be-repaired image frame. The k pooling features corresponding to the preceding image frame and the image frame to be repaired respectively, k is a positive integer;

Perform convolution processing on the k pooled features respectively to obtain k convolution features;

Feature fusion prediction is performed on the k convolution features to obtain the area to be repaired of the image frame to be repaired.
A data repair device, characterized in that the device includes:

A sample acquisition module, used to acquire the repaired image sample to be repaired, the repair area label corresponding to the repaired image sample, and the original image sample;

A sample area prediction module, configured to use the first area prediction model to predict the area to be repaired of the repaired image sample, and obtain the sample predicted repair area;

A sample repair module, configured to use the first media repair model to repair the sample predicted repair area in the repaired image sample, and obtain a sample optimized image corresponding to the repaired image sample;

A model adjustment module configured to predict the repair area, the repair area label, the original image sample, and the sample optimized image based on the sample, and to predict the parameters of the first area prediction model and the first media repair model. Joint adjustment is performed to obtain a target area prediction model corresponding to the first area prediction model and a target media repair model corresponding to the first media repair model.
A data repair device, characterized in that the device includes:

Image acquisition module, used to acquire image frames to be repaired;

A region prediction module, used to predict the image frame to be repaired based on the target region prediction model, and obtain the region to be repaired of the image frame to be repaired;

The data repair module is used to repair the area to be repaired in the image frame to be repaired based on the target media repair model, and obtain the optimized image frame corresponding to the image frame to be repaired, wherein the target area prediction model is consistent with the image frame to be repaired. The above target media repair model is obtained through joint training.
A computer device, characterized by including a processor, a memory, and an input and output interface;

The processor is connected to the memory and the input-output interface respectively, wherein the input-output interface is used to receive data and output data, the memory is used to store computer programs, and the processor is used to call the computer Program, so that the computer device performs the method described in any one of claims 1-8, or performs the method described in any one of claims 9-13.
A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and the computer program is adapted to be loaded and executed by a processor, so that a computer device having the processor executes the claims The method described in any one of claims 1-8, or the method described in any one of claims 9-13.