CN112419216A

CN112419216A - Image interference removing method and device, electronic equipment and computer readable storage medium

Info

Publication number: CN112419216A
Application number: CN202011298217.7A
Authority: CN
Inventors: 彭健腾; 康斌
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2020-11-18
Filing date: 2020-11-18
Publication date: 2021-02-26

Abstract

The application provides an image interference removing method, an image interference removing device, electronic equipment and a computer readable storage medium; the method comprises the following steps: carrying out image decomposition on the image to be processed carrying the interference information to obtain at least two target images; performing feature extraction on the image to be processed through a first feature extraction layer of the image interference removal model to obtain image features of the image to be processed; respectively extracting interference features of each target image through a second feature extraction layer of the image interference removal model to obtain corresponding interference features; fusing the interference characteristics of each target image through a characteristic fusion layer of the image interference elimination model to obtain fused interference characteristics; and performing interference elimination processing on the image to be processed through an output layer of the image interference elimination model based on the fusion interference characteristic and the image characteristic to obtain a corresponding interference elimination image. By the method and the device, the interference information in the image can be accurately removed.

Description

Image interference removing method and device, electronic equipment and computer readable storage medium

Technical Field

The present disclosure relates to image processing technologies, and in particular, to an image interference removing method and apparatus, an electronic device, and a computer-readable storage medium.

Background

In the process of digital transmission, interference information such as image noise and image aliasing is easily generated due to interference of imaging equipment, external environment and the like, and in order to obtain an image with better image quality, the image is generally required to be subjected to interference elimination processing.

The image interference removing method adopted in the related technology is to directly input the image to be subjected to interference removal into a trained machine learning model, so that the machine learning model performs interference removing processing on the image. However, in this way, the image information that can be obtained by the machine learning model is very limited, so that the interference information removed when the interference removal processing is performed based on the limited image information is not accurate enough.

Disclosure of Invention

The embodiment of the application provides an image interference removing method and device, electronic equipment and a computer readable storage medium, which can accurately remove interference information in an image.

The technical scheme of the embodiment of the application is realized as follows:

the embodiment of the application provides an image interference removing method, which comprises the following steps:

carrying out image decomposition on the image to be processed carrying the interference information to obtain at least two target images;

performing feature extraction on the image to be processed through a first feature extraction layer of an image interference removal model to obtain image features of the image to be processed;

respectively extracting interference features of each target image through a second feature extraction layer of the image interference removal model to obtain corresponding interference features;

performing fusion processing on the interference features of the target images through a feature fusion layer of the image interference removal model to obtain fusion interference features;

and performing interference elimination processing on the image to be processed based on the fusion interference characteristic and the image characteristic through an output layer of the image interference elimination model to obtain a corresponding interference elimination image.

The embodiment of the application provides an image interference removing device, including:

the image decomposition module is used for carrying out image decomposition on the image to be processed carrying the interference information to obtain at least two target images;

the image feature extraction module is used for extracting features of the image to be processed through a first feature extraction layer of an image interference removal model to obtain image features of the image to be processed;

the interference feature extraction module is used for respectively extracting interference features of the target images through a second feature extraction layer of the image interference removal model to obtain corresponding interference features;

the characteristic fusion module is used for carrying out fusion processing on the interference characteristics of each target image through a characteristic fusion layer of the image interference removal model to obtain fusion interference characteristics;

and the interference removing module is used for performing interference removing processing on the image to be processed based on the fusion interference characteristic and the image characteristic through an output layer of the image interference removing model to obtain a corresponding interference removing image.

In the above scheme, the image decomposition module is further configured to perform multi-scale image decomposition on the image to be processed carrying the interference information to construct a gaussian pyramid corresponding to the image to be processed; and taking the image of each layer in the Gaussian pyramid as the target image.

In the above scheme, the image decomposition module is further configured to perform multi-scale image decomposition on the image to be processed carrying the interference information to construct a gaussian pyramid corresponding to the image to be processed; constructing a Laplacian pyramid corresponding to the image to be processed based on the Gaussian pyramid corresponding to the image to be processed; combining the images of all layers in the Gaussian pyramid with the images of corresponding layers in the Laplacian pyramid respectively to obtain at least two image combinations; combining the at least two images as the target image.

In the above scheme, the image decomposition module is further configured to perform multi-scale image decomposition on the image to be processed carrying the interference information to construct a gaussian pyramid corresponding to the image to be processed; constructing a Laplacian pyramid corresponding to the image to be processed based on the Gaussian pyramid corresponding to the image to be processed; respectively carrying out down-sampling processing on the images of all layers in the Gaussian pyramid to obtain corresponding down-sampled images; combining the images of all layers in the Gaussian pyramid, the images of the corresponding levels in the Laplacian pyramid and the downsampled images with the same size as the images of the corresponding levels in the Laplacian pyramid respectively to obtain at least two image combinations; combining the at least two images as the target image.

In the above scheme, the second feature extraction layer includes N sub-feature extraction layers, where the number of the target images is N, and N is a positive integer not less than 2; the interference feature extraction module is further configured to input each target image into one sub-feature extraction layer, and perform interference feature extraction on the target image through the sub-feature extraction layer to obtain corresponding interference features.

In the above scheme, the interference features include size features and channel features, and the feature fusion module is further configured to perform size transformation on the interference features of each target image, respectively, to obtain target interference features corresponding to each interference feature; wherein the size characteristics of each target interference characteristic are the same; performing fusion processing on the channel characteristics of the target interference characteristics to obtain fusion channel characteristics; and performing characteristic splicing on the size characteristic of the target interference characteristic and the fusion channel characteristic to obtain the fusion interference characteristic.

In the above scheme, when the number of the target images is m, the number of the interference features is m correspondingly, and the feature fusion module is further configured to fuse a jth interference feature and a (j + 1) th interference feature of the m interference features to obtain a jth fusion interference feature; wherein m is a positive integer not less than 3, j is a positive integer, and j belongs to [1, m-1 ]; fusing the jth fusion interference feature with the (j + 2) th interference feature to obtain a (j + 1) th fusion interference feature; and starting to take the value of j as 1, traversing j, and taking the j +1 th fusion interference feature as the fusion interference feature when the value of j +2 is the same as the value of m.

In the above scheme, the interference elimination module is further configured to determine a difference feature between the image feature and the fusion interference feature through an output layer of the image interference elimination model; and decoding the difference characteristic through an output layer of the image interference removal model to obtain the interference removal image.

In the above scheme, the image interference elimination apparatus further includes: the model training module is used for carrying out image decomposition on the sample interference image carrying the interference information to obtain at least two sample target images; the sample interference image is obtained by adding interference information to the original sample image; performing feature extraction on the sample interference image through a first feature extraction layer of the image interference removal model to obtain sample image features of the sample interference image; respectively extracting interference features of each sample target image through a second feature extraction layer of the image interference removal model to obtain corresponding sample interference features; performing fusion processing on the sample interference features corresponding to the sample target images through the feature fusion layer of the image interference removal model to obtain sample fusion interference features; performing interference elimination processing on the sample interference image based on the sample fusion interference characteristic and the sample image characteristic through an output layer of the image interference elimination model to obtain a corresponding prediction interference elimination image; updating model parameters of the image de-interference model based on a difference between the predicted de-interference image and the sample original image.

In the above scheme, the sample original image corresponds to at least two sample interference images, each sample interference image is obtained by adding interference information to the sample original image, and correspondingly, the image decomposition module is further configured to perform, for each sample interference image, the operation from performing image decomposition on the sample interference image carrying the interference information to performing interference removal processing on the sample interference image, so as to obtain a predicted interference removal image corresponding to each sample interference image; correspondingly, the model training module is further configured to update the model parameters of the image interference elimination model based on the difference between each predicted interference elimination image and the sample original image.

In the above scheme, the interference information includes at least one of image aliasing and image noise; correspondingly, the image interference removing device further comprises: the interference information adding module is used for adding image saw teeth to the sample original image to obtain a sample saw tooth image carrying the image saw teeth, and the sample saw tooth image is used as the sample interference image; or adding image noise to the sample original image to obtain a sample noise image carrying the image noise, and taking the sample noise image as the sample interference image; or adding image saw-teeth and image noise to the sample original image to obtain a sample saw-tooth noise image simultaneously carrying the image saw-teeth and the image noise, and taking the sample saw-tooth noise image as the sample interference image.

The embodiment of the application provides a training method of an image interference removal model, which comprises the following steps:

carrying out image decomposition on the sample interference image carrying the interference information to obtain at least two sample target images; the sample interference image is obtained by adding interference information to the original sample image;

performing feature extraction on the sample interference image through a first feature extraction layer of an image interference removal model to obtain sample image features of the sample interference image;

respectively extracting interference features of each sample target image through a second feature extraction layer of the image interference removal model to obtain corresponding sample interference features;

performing fusion processing on the sample interference features corresponding to the sample target images through the feature fusion layer of the image interference removal model to obtain sample fusion interference features;

performing interference elimination processing on the sample interference image based on the sample fusion interference characteristic and the sample image characteristic through an output layer of the image interference elimination model to obtain a corresponding prediction interference elimination image;

and updating the parameters of the first feature extraction layer, the parameters of the second feature extraction layer, the parameters of the feature fusion layer and the parameters of the output layer respectively based on the difference between the predicted interference-removed image and the sample original image.

An embodiment of the present application provides an electronic device, including:

a memory for storing executable instructions;

and the processor is used for realizing the image interference removing method provided by the embodiment of the application or the training method of the image interference removing model provided by the embodiment of the application when the executable instructions stored in the memory are executed.

The embodiment of the present application provides a computer-readable storage medium, which stores executable instructions and is used for implementing, when executed by a processor, an image interference removal method provided by the embodiment of the present application or a training method of an image interference removal model provided by the embodiment of the present application.

The embodiment of the application has the following beneficial effects:

according to the method and the device, the image to be processed is decomposed into the multiple target images, it can be understood that each target image contains the image information of the image to be processed, the image information of the image to be processed is also decomposed into multiple representations along with the decomposition of the image to be processed, therefore, the image information of the image to be processed can be more clearly and comprehensively represented through the multiple target images, then the interference characteristics of each target image are extracted through the image interference removing model, the interference characteristics are fused, the obtained fusion interference characteristics correspondingly cover more and more accurate interference information in the image to be processed, and therefore the interference information in the image to be processed can be accurately removed based on the fusion interference characteristics.

Drawings

FIG. 1 is a schematic diagram of an alternative configuration of an image de-interference system according to an embodiment of the present disclosure;

fig. 2 is an alternative structural schematic diagram of an electronic device provided in an embodiment of the present application;

FIG. 3 is a schematic flow chart of an alternative image de-interference method according to an embodiment of the present disclosure;

FIG. 4 is a schematic flow chart of an alternative image de-interference method provided by the embodiment of the present application;

FIG. 5 is a schematic diagram of an alternative structure of a Gaussian pyramid provided by an embodiment of the present application;

FIG. 6 is an alternative schematic diagram of a downsampling process provided by an embodiment of the present application;

FIG. 7 is a schematic flow chart of an alternative image de-interference method according to an embodiment of the present disclosure;

FIG. 8 is an alternative diagram of a process for constructing a Gaussian pyramid and a Laplacian pyramid according to an embodiment of the present application;

FIG. 9 is a schematic diagram of an alternative structure of a Gaussian pyramid and a corresponding Laplacian pyramid provided in an embodiment of the present application;

FIG. 10 is an alternative schematic diagram of an upsampling process provided by an embodiment of the present application;

FIG. 11 is an alternative schematic diagram of an image combination pyramid provided by an embodiment of the present application;

FIG. 12 is a schematic flow chart of an alternative image de-interference method according to an embodiment of the present application;

FIG. 13 is an alternative schematic diagram of a downsampled image pyramid provided by an embodiment of the present application;

FIG. 14 is an alternative schematic diagram of an image combination pyramid provided by an embodiment of the present application;

FIG. 15 is an alternative schematic diagram of a downsampling process provided by an embodiment of the present application;

FIG. 16 is an alternative schematic diagram of an image combination pyramid provided by an embodiment of the present application;

FIG. 17 is an alternative schematic diagram of an image combination pyramid provided by an embodiment of the present application;

FIG. 18 is an alternative structural diagram of an image de-interference model provided by an embodiment of the present application;

FIG. 19 is an alternative structural diagram of an image de-interference model provided by an embodiment of the present application;

FIG. 20 is a schematic flow chart of an alternative image de-interference method according to an embodiment of the present application;

FIG. 21 is an alternative schematic diagram of image features provided by embodiments of the present application;

FIG. 22 is a schematic flow chart of an alternative image de-interference method according to an embodiment of the present application;

FIG. 23 is an alternative schematic diagram of an image de-interference process provided by an embodiment of the present application;

FIG. 24 is an alternative flowchart of a training method for an image de-interference model according to an embodiment of the present disclosure;

FIG. 25 is an alternative schematic diagram of a sample set provided by an embodiment of the present application;

FIG. 26 is a schematic flow chart of an alternative image de-interference method according to an embodiment of the present application;

fig. 27 is an alternative schematic diagram of an image interference elimination device according to an embodiment of the present application.

Detailed Description

In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be described in further detail with reference to the attached drawings, the described embodiments should not be considered as limiting the present application, and all other embodiments obtained by a person of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict.

In the following description, references to the terms "first \ second \ third" are only to distinguish similar objects and do not denote a particular order, but rather the terms "first \ second \ third" are used to interchange specific orders or sequences, where appropriate, so as to enable the embodiments of the application described herein to be practiced in other than the order shown or described herein.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of the present application only and is not intended to be limiting of the application.

Before further detailed description of the embodiments of the present application, terms and expressions referred to in the embodiments of the present application will be described, and the terms and expressions referred to in the embodiments of the present application will be used for the following explanation.

1) The interference information refers to unnecessary or redundant information that affects image quality, such as image noise and image aliasing, present in the image data.

2) And (3) image decomposition, namely decomposing the original image into sub-images with different image characteristics, wherein the image characteristics can be the structure, texture, noise, sawtooth, scale and the like of the image.

The scale of the image comprises a spatial scale, a time scale, a semantic scale and the like, and the spatial scale comprises a drawing scale, a geographic scale, an operation scale and a resolution measurement scale. In the field of computer vision, the scale of an image generally refers to the resolution measurement scale.

Artificial Intelligence (AI) is a theory, method, technique and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend and expand human Intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a manner similar to human intelligence. Artificial intelligence is the research of the design principle and the realization method of various intelligent machines, so that the machines have the functions of perception, reasoning and decision making.

The artificial intelligence technology is a comprehensive subject and relates to the field of extensive technology, namely the technology of a hardware level and the technology of a software level. The artificial intelligence infrastructure generally includes technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and the like.

Machine Learning (ML) is a multi-domain cross discipline, and relates to a plurality of disciplines such as probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and the like. The special research on how a computer simulates or realizes the learning behavior of human beings so as to acquire new knowledge or skills and reorganize the existing knowledge structure to continuously improve the performance of the computer. Machine learning is the core of artificial intelligence, is the fundamental approach for computers to have intelligence, and is applied to all fields of artificial intelligence. Machine learning and deep learning generally include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and formal education learning.

The scheme provided by the embodiment of the application relates to the technologies such as machine learning of artificial intelligence, and the like, and is specifically explained by the following embodiment.

The inventor finds that when image de-interference is performed in a mode that an image needing de-interference is input into a machine learning model, the image is down-sampled by the machine learning model to extract image features, then deconvolution is performed on the image features to enlarge the sizes of the image features, interference information with the same size as the input image is obtained, and then the image and the interference information are subtracted to obtain a predicted de-interference image, the image features are easy to generate saw teeth in the up-sampling process, and the image de-interference effect is poor.

Based on this, the inventors have found that the operation of upsampling can be avoided while increasing the field of view by extracting image features using a hole convolution instead of downsampling. However, the hole convolution may cause the generated image to have a certain degree of holes, which is also disadvantageous to the image de-interference, so that the image de-interference effect is not good.

Based on this, embodiments of the present application provide an image interference removing method, an image interference removing device, an electronic device, and a computer-readable storage medium, which can accurately remove interference information in an image.

First, an image interference removing system provided in the embodiment of the present application is described, fig. 1 is an optional schematic diagram of the image interference removing system 100 provided in the embodiment of the present application, in order to support an image interference removing application, an image interference removing client 410 is disposed on a terminal 400, the terminal 400 is connected to a server 200 through a network 300, the network 300 may be a wide area network or a local area network, or a combination of the two, and data transmission is achieved by using a wireless link. In some embodiments, the terminal 400 may be, but is not limited to, a laptop, a tablet, a desktop computer, a smart phone, a dedicated messaging device, a portable gaming device, a smart speaker, a smart watch, and the like. The server 200 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a network service, cloud communication, middleware service, a domain name service, a security service, a CDN, a big data and artificial intelligence platform, and the like. The network 300 may be a wide area network or a local area network, or a combination of both. The terminal 400 and the server 200 may be directly or indirectly connected through wired or wireless communication, and the embodiment of the present application is not limited thereto.

The terminal 400 is configured to, in response to an upload operation for the image to be processed, acquire the image to be processed, and in response to an interference elimination instruction for the image to be processed, send the image to be processed to the server.

The server 200 is configured to perform image decomposition on the image to be processed to obtain at least two target images, input the image to be processed and the at least two target images into the image interference elimination model, extract the target images through the first feature extraction layer of the image interference elimination model, extracting the characteristics of the image to be processed to obtain the image characteristics of the image to be processed, extracting the interference characteristics of each target image through a second characteristic extraction layer of the image interference removal model to obtain corresponding interference characteristics, performing image interference removal on the target image through a characteristic fusion layer of the image interference removal model, the interference characteristics of each target image are fused to obtain fused interference characteristics, and the fused interference characteristics and the image characteristics are obtained through an output layer of an image interference removing model, and performing interference elimination processing on the image to be processed to obtain a corresponding interference elimination image, and sending the interference elimination image to the terminal 400.

The terminal 400 is further configured to receive and output the interference-free image sent by the server 200.

Next, an electronic device for implementing the image interference removing method according to the embodiment of the present application is described, referring to fig. 2, fig. 2 is a schematic structural diagram of the electronic device 500 according to the embodiment of the present application, and in practical applications, the electronic device 500 may be implemented as the terminal 400 or the server 200 in fig. 1, and the electronic device implementing the image interference removing method according to the embodiment of the present application is described by taking the electronic device as the server 200 shown in fig. 1 as an example. The electronic device 500 shown in fig. 2 includes: at least one processor 510, memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together by a bus system 540. It is understood that the bus system 540 is used to enable communications among the components. The bus system 540 includes a power bus, a control bus, and a status signal bus in addition to a data bus. For clarity of illustration, however, the various buses are labeled as bus system 540 in fig. 2.

The Processor 510 may be an integrated circuit chip having Signal processing capabilities, such as a general purpose Processor, a Digital Signal Processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like, wherein the general purpose Processor may be a microprocessor or any conventional Processor, or the like.

The user interface 530 includes one or more output devices 531 enabling presentation of media content, including one or more speakers and/or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components to facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid state memory, hard disk drives, optical disk drives, and the like. Memory 550 optionally includes one or more storage devices physically located remote from processor 510.

The memory 550 may comprise volatile memory or nonvolatile memory, and may also comprise both volatile and nonvolatile memory. The nonvolatile Memory may be a Read Only Memory (ROM), and the volatile Memory may be a Random Access Memory (RAM). The memory 550 described in embodiments herein is intended to comprise any suitable type of memory.

In some embodiments, memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or subsets or supersets thereof, as exemplified below.

An operating system 551 including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

a network communication module 552 for communicating to other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: bluetooth, wireless compatibility authentication (WiFi), and Universal Serial Bus (USB), etc.;

a presentation module 553 for enabling presentation of information (e.g., a user interface for operating peripherals and displaying content and information) via one or more output devices 531 (e.g., a display screen, speakers, etc.) associated with the user interface 530;

an input processing module 554 to detect one or more user inputs or interactions from one of the one or more input devices 532 and to translate the detected inputs or interactions.

In some embodiments, the image interference elimination apparatus provided in the embodiments of the present application may be implemented in software, and fig. 2 shows an image interference elimination apparatus 555 stored in a memory 550, which may be software in the form of programs and plug-ins, and includes the following software modules: the image decomposition module 5551, the image feature extraction module 5552, the interference feature extraction module 5553, the feature fusion module 5554, and the interference elimination module 5555 are logical and thus may be arbitrarily combined or further separated according to the functions implemented. The functions of the respective modules will be explained below.

In other embodiments, the image de-interference apparatus provided in the embodiments of the present Application may be implemented in hardware, and for example, the image de-interference apparatus provided in the embodiments of the present Application may be a processor in the form of a hardware decoding processor, which is programmed to perform the image de-interference method provided in the embodiments of the present Application, for example, the processor in the form of the hardware decoding processor may be one or more Application Specific Integrated Circuits (ASICs), DSPs, Programmable Logic Devices (PLDs), Complex Programmable Logic Devices (CPLDs), Field Programmable Gate Arrays (FPGAs), or other electronic components.

Next, the image interference removing method provided by the embodiment of the present application will be described in conjunction with an exemplary application and implementation of the server provided by the embodiment of the present application.

Referring to fig. 3, fig. 3 is an alternative flowchart of an image de-interference method provided in the embodiment of the present application, which will be described with reference to the steps shown in fig. 3.

Step 101, the server performs image decomposition on the to-be-processed image carrying the interference information to obtain at least two target images.

Here, each target image has different image characteristics, and the server obtains a plurality of target images having different image characteristics by performing image decomposition processing on the image set to be processed. Here, the target image may include an image to be processed. In actual implementation, the server may process a plurality of pixel points constituting the image to be processed to obtain at least two target images. Specifically, the server may screen a plurality of target pixel points from the image to be processed, and construct a target image with the plurality of target pixel points.

In some embodiments, referring to fig. 4, fig. 4 is an optional flowchart of an image interference removing method provided in the embodiment of the present application, and based on fig. 3, step 101 may also be implemented as follows:

step 201, a server carries out multi-scale image decomposition on an image to be processed carrying interference information to construct a Gaussian pyramid corresponding to the image to be processed;

and step 202, taking the image of each layer in the Gaussian pyramid as a target image.

In practical implementation, the server may perform at least one image decomposition on the image to be processed to obtain a target image with a different scale from that of the image to be processed, and then construct a gaussian pyramid based on the image to be processed and the target image, where the obtained gaussian pyramid includes images with two scales, such as the image to be processed and the target image. Referring to fig. 5, fig. 5 is an alternative structural schematic diagram of a gaussian pyramid provided in the embodiment of the present application, and the gaussian pyramid includes i +2 layer images in total from layer 0 to layer i +1, where i is an integer greater than or equal to 0. The process of constructing the gaussian pyramid in the embodiment of the application specifically comprises the following steps: the server will treat the image I₀The image I to be processed is taken as the bottom layer image of the Gaussian pyramid, namely the 0 th layer image₀Performing convolution operation by using Gaussian kernel to obtain Gaussian image G (I) after Gaussian convolution₀) Then to the Gaussian image G (I)₀) Down-sampling processing is performed to obtain a down-sampled image down _ sample (G (I)₀) Taking the image obtained by down-sampling as the image I to be processed₀Upper layer image in gaussian pyramid I₁I.e. I₁＝down_sample(G(I₀) In the same way, the image I of the I +1 th layer of the Gaussian pyramid is obtained_i+1＝down_sample(G(I_i))。

In addition, the downsampling processing mode involved in the process of constructing the gaussian pyramid is specifically as follows: and extracting pixel points with odd numbers of lines and columns in the original image, constructing a new image based on the extracted pixel points, and taking the constructed image as a down-sampling image. Exemplarily, referring to fig. 6, fig. 6 is an optional schematic diagram of a downsampling process provided in this embodiment of the present application, where a pixel point whose number of lines and number of columns in an original image P are odd is denoted as 1, and the remaining pixel points are denoted as 0, and through the downsampling process described in this embodiment of the present application, a pixel point whose number of lines and number of columns are odd is extracted from the original image P, and then an image, that is, a downsampled image down _ sample (P), is constructed based on these pixel points. It is understood that the size of the down-sampled image obtained by the down-sampling method is 1/4 of the original image. That is, the size of the image of the upper layer in the Gaussian pyramid is 1/4, i.e., I, of the image of the lower layer_i+1Has a dimension of I_i1/4 of (1). For example, FIG. 5 shows the image I of layer 0 in the Gaussian pyramid₀Is (W, H), where W is the image width and H is the image height, then layer 1 image I₁The size of (B) is (W/2, H/2), i.e. I₁The size of (A) is W/2H/2-1/4W H, is I₀1/4, corresponding, I-th layer image I_iHas a size of (W/2)ⁱ,H/2ⁱ)，

In practical implementation, the server takes each layer of image in the constructed gaussian pyramid as a target image, the target image is input into an image interference elimination model to be subjected to interference elimination processing on the image to be processed, the image to be processed is decomposed into the target images of multiple scales in this way, the target image is a low-frequency signal of the image to be processed on different scales, and has image information of the image to be processed with different intensities, for example, a higher-level image in the gaussian pyramid has lower image resolution, weaker detail information, more interference information and stronger contour information than a higher-level image, so that the image interference elimination model can learn the image information of multiple scales, and therefore the interference elimination processing can be performed more accurately, and the image quality of the obtained interference elimination image is higher.

In some embodiments, referring to fig. 7, fig. 7 is an optional flowchart of an image interference removing method provided in the embodiment of the present application, and based on fig. 3, step 101 may also be implemented as follows:

step 301, the server performs multi-scale image decomposition on the image to be processed carrying the interference information to construct a gaussian pyramid corresponding to the image to be processed.

Step 302, constructing a laplacian pyramid corresponding to the image to be processed based on the gaussian pyramid corresponding to the image to be processed.

Referring to fig. 8 and 9, fig. 8 is an optional schematic diagram of a process for constructing a gaussian pyramid and a laplacian pyramid provided in the embodiment of the present application, and fig. 9 is an optional structural schematic diagram of a gaussian pyramid and a corresponding laplacian pyramid provided in the embodiment of the present application. In practical implementation, the process of constructing the laplacian pyramid specifically includes: the server converts the layer 0 image I of the Gaussian pyramid₀Performing upsampling processing to obtain an upsampled image up _ sample (I)₀) And the up-sampled image is convolved by a Gaussian kernel to obtain an image G' (up _ sample (I)₀) The structure of the gaussian kernel used here is the same as that used when the gaussian pyramid is constructed, and the numerical value is 4 times that of the gaussian kernel used when the gaussian pyramid is constructed, and then, the server bases on the layer 0 image I of the gaussian pyramid₀And the image G' (up _ sample (I)₀) Obtain the 0 th layer image L of the Laplacian pyramid₀Here, the layer 0 image I of the Gaussian pyramid is taken₀And the image G' (up _ sample (I)₀) Are subtracted) to obtain an image L₀I.e. L₀＝I₀-G’(up_sample(I₀) In the same manner, an image L of the ith layer of the laplacian pyramid is obtained_i＝I_i-G’(up_sample(I_i+1) Wherein i is an integer of 0 or more. It can be understood that each layer of image in the laplacian pyramid is a residual image, which is a high-frequency signal of the image to be processed on different scales, and the laplacian is a high-frequency signalThe number of layers of the character pyramid is one layer less than that of the corresponding Gaussian pyramid, and the Laplacian images with the same level are the same as the Gaussian images in size.

In addition, the upsampling processing mode involved in the process of constructing the laplacian pyramid may specifically be: for image I in Gaussian pyramid_iAdding new pixel points, expanding the number of pixel line rows to twice of the original number, expanding the number of pixel line columns to twice of the original number, and inserting the newly added pixel points in each row into the image I at intervals_iAnd inserting newly added pixel points of each row into the image I at intervals_iAnd then, filling the channel value of the newly added pixel point, for example, the channel value corresponding to white can be completely filled. Illustratively, referring to fig. 10, fig. 10 is an optional schematic diagram of an upsampling process provided by an embodiment of the application, where a server is in an image I_iAdding a row of pixel points below each row of pixel points and adding a row of pixel points on the right side of each row of pixel points, wherein the channel value is 0 as an example for filling the added pixel points, and obtaining an up-sampling image up _ sample (I)_i) As shown in fig. 10, its size is enlarged for image I_iFour times that of the prior art. In some embodiments, the server may also be in image I_iA row of pixel points is newly added at the lower side of each row of pixel points, and a column of pixel points is newly added at the left side of each column of pixel points, or in the image I_iThe upper side of each row of pixel points is newly added with a row of pixel points and the left side of each column of pixel points is newly added with a column of pixel points, or in the image I_iThe method comprises the following steps of performing upsampling processing on the upper side of each row of pixel points by adding a row of pixel points on the upper side of each row of pixel points, adding a row of pixel points on the right side of each row of pixel points, and the like, or adding pixel points at intervals of multiple rows or multiple columns, wherein the upsampling mode is not specifically limited in the embodiment of the application.

Step 303, combining the images of each layer in the gaussian pyramid with the images of the corresponding layer in the laplacian pyramid, respectively, to obtain at least two image combinations.

At step 304, at least two images are combined as a target image.

See the figureFig. 11 is an optional schematic diagram of an image combination pyramid provided in the embodiment of the present application, where the image combination may be represented by a set, and for example, the server combines the ith layer image I in the gaussian pyramid_iAnd the ith layer image L in the Laplacian pyramid_iCombining to obtain an image combination B_i＝{I_i,L_i}. In actual implementation, the server combines B the images during image processing_iThe form of the constituent image blocks is processed. The image block is obtained by superposing more than two images, and the size of the image block comprises three dimensions, namely width, height and image quantity. Here, if the image is an RGB (Red, Green, Blue, Red Green Blue) image, the number of channels of the pixel in the image is 3, and the channels are three channels of Red, Green, Blue, and the like, respectively, and if the image is an RGGB (Red, Green, Blue, Red Green Blue) image, the number of channels of the pixel in the image is 4, and the channels are four channels of Red, Green, Blue, and the like, respectively. In the embodiment of the present application, the number of channels is 3, and for example, in the image combination shown in fig. 11, the image combination B is described_iIs characterized by the dimension of (W/2)ⁱ,H/2ⁱ,2×3)。

After the server combines the images of each layer in the gaussian pyramid with the images of the corresponding layer in the laplacian pyramid, the sizes of the obtained image combinations are different from each other, and an image pyramid, such as the image combination pyramid shown in fig. 11, may be formed. Because the number of layers of the laplacian pyramid is one less than that of the laplacian pyramid, the server combines the images of the last layer of the laplacian pyramid, namely the image I of the topmost layer, when performing image combination_i+1Alone as one image combination. It can be understood that, in the embodiment of the application, after the server performs image combination on the i +2 layers of gaussian pyramids including 0 to i +1 layers and the corresponding laplacian pyramids, an image group including 0 to i +1 layers and i +2 layers is obtainedA pyramid is combined, wherein the ith layer image is combined with B_i＝{I_i,L_iH, i +1 th layer image combination B_i+1＝{I_i+1}。

In practical implementation, the server combines each image as a target image, the target image is input into an image interference elimination model to be subjected to interference elimination processing, the image to be processed is decomposed into a gaussian pyramid containing a plurality of scale images and a laplacian pyramid containing a plurality of scale images in this way, then the gaussian pyramid and the laplacian pyramid are combined, and the combined image combination is used as the target image, so that a low-frequency signal and a high-frequency signal in the image to be processed can be considered simultaneously when the image processing is performed on the basis of the target image, and interference information in the low-frequency signal and the high-frequency signal can be removed simultaneously, and therefore an interference elimination image with higher image quality can be obtained.

In some embodiments, referring to fig. 12, fig. 12 is an optional flowchart of an image interference removing method provided in the embodiment of the present application, and based on fig. 3, step 101 may also be implemented as follows:

step 401, the server performs multi-scale image decomposition on the image to be processed carrying the interference information to construct a gaussian pyramid corresponding to the image to be processed.

And 402, constructing a Laplacian pyramid corresponding to the image to be processed based on the Gaussian pyramid corresponding to the image to be processed.

And step 403, respectively performing down-sampling processing on the images of each layer in the Gaussian pyramid to obtain corresponding down-sampled images.

Referring to fig. 13, fig. 13 is an alternative schematic diagram of a downsampled image pyramid provided in the embodiment of the present application. Here, the server directly performs downsampling processing on the images of the respective layers in the gaussian pyramid to obtain downsampled images corresponding to the images of the respective layers. In practical implementation, the server performs down-sampling processing on the images of the layers except the top layer in the Gaussian pyramid respectively to obtain corresponding down-sampled images. Illustratively, the server pairs the 0 th level in a Gaussian pyramid with i +2 level imagesTotal I +1 layer image I to ith layer₀To I_iDown-sampling processing is carried out to obtain corresponding down-sampled images which are ds (I) respectively₀)、ds(I₁)、…、ds(I_i). It should be appreciated that the downsampled image ds (I)₀) Is compared with the image I before down-sampling₀Smaller, in particular down-sampled, image ds (I)₀) Has a size of I ₀1/4 of (1).

Step 404, combining the images of the layers in the gaussian pyramid, the images of the corresponding levels in the laplacian pyramid, and the downsampled images with the same size as the images of the corresponding levels in the laplacian pyramid, respectively, to obtain at least two image combinations.

At step 405, at least two images are combined as a target image.

Referring to fig. 14, fig. 14 is an optional schematic diagram of the image combination pyramid provided in the embodiment of the present application, and in practical implementation, the server combines the ith layer image I in the gaussian pyramid_iAnd the ith layer image L in the Laplacian pyramid_iAnd with the image I_iIs combined with the down-sampled image of the same size as the image I, here_iIs the same size down-sampled image as image I_iThe down-sampling image corresponding to the next layer image in the Gaussian pyramid, namely the I-1 layer image I_i-1Corresponding down-sampled image ds (I)_i-1) That is, the server will image I_iImage L_iAnd down-sampled image ds (I)_i-1) Combining to obtain corresponding image combination B_i＝{I_i,L_i,ds(I_i-1)}。

In the embodiment of the application, on the basis of the gaussian pyramid and the laplacian pyramid, down-sampling processing is further performed on the gaussian pyramid so as to perform further image decomposition on the gaussian pyramid and obtain a down-sampled image pyramid, and thus, an image combination pyramid obtained by combining the gaussian pyramid, the image combination pyramid and the image combination pyramid has image information with more dimensions, so that the model effect of the interference removing model obtained by model training based on the image combination pyramid is better, and the interference removing image with higher quality can be obtained.

In some embodiments, based on fig. 12, step 403 may also be implemented as follows:

the server respectively executes the following processing aiming at the Gaussian images of each layer of the Gaussian pyramid: determining row codes and column codes of pixel points in the Gaussian image; selecting a plurality of first pixel points of which row codes and column codes are odd numbers from a plurality of pixel points forming the Gaussian graph, and constructing a corresponding first down-sampling image based on the plurality of first pixel points; selecting a plurality of second pixel points with odd row codes and even column codes from a plurality of pixel points forming the Gaussian graph, and constructing a corresponding second down-sampling image based on the plurality of second pixel points; selecting a plurality of third pixel points with even-numbered row codes and odd-numbered column codes from a plurality of pixel points forming the Gaussian graph, and constructing a corresponding third down-sampling image based on the plurality of third pixel points; selecting a plurality of fourth pixel points of which the row codes and the column codes are even numbers from a plurality of pixel points forming the Gaussian graph, and constructing a corresponding fourth down-sampling image based on the plurality of fourth pixel points; the first downsampled image, the second downsampled image, the third downsampled image, and the fourth downsampled image are used as the downsampled images.

Exemplarily, referring to fig. 15 and 16, fig. 15 is an optional schematic diagram of a downsampling processing procedure provided by an embodiment of the present application, and fig. 16 is an optional schematic diagram of a downsampling image pyramid provided by an embodiment of the present application. In the embodiment of the present application, if the first pixel point is marked as 0, the second pixel point is marked as 1, the third pixel point is marked as 2, and the fourth pixel point is marked as 3, the ith layer image I of the gaussian pyramid is obtained_iThe pixel points (b) may be represented in the form shown in fig. 15, and the server may obtain the first to fourth downsampled images ds0 (I) through the four downsampling methods described above_i)、ds1(I_i)、ds2(I_i) And ds3 (I)_i) The pixel points in the four down-sampling images are respectively 0, 1, 2 and 3, and the sizes of the four down-sampling images are all the image I_i1/4 of (1).

In practical implementation, after performing the four types of downsampling processing on the images except for the top layer in the gaussian pyramid, the server obtains corresponding four types of downsampling image pyramids, which are respectively a first downsampling image pyramid, a second downsampling image pyramid, a third downsampling image pyramid and a fourth downsampling image pyramid. Referring to fig. 17, fig. 17 is an alternative schematic diagram of an image combination pyramid provided in the embodiment of the present application, where the image combination pyramid includes a gaussian pyramid, a laplacian pyramid, and images in four downsampled image pyramids.

In the embodiment of the application, the four down-sampling processes are performed on the gaussian pyramid, one layer of image in the gaussian pyramid is decomposed into the four down-sampling images, and the pixel points of the four down-sampling images jointly form the pixel points of the corresponding image in the gaussian pyramid, so that all the pixel point information in the gaussian pyramid is kept in the four down-sampling images, and an image combination pyramid formed by the gaussian pyramid, the laplacian pyramid and the four down-sampling image pyramids has more and more perfect original image information of the image to be processed.

And 102, performing feature extraction on the image to be processed through a first feature extraction layer of the image interference removal model to obtain image features of the image to be processed.

Referring to fig. 18, fig. 18 is an alternative structural diagram of an image interference elimination model provided in the embodiment of the present application. Here, the image interference removal model is a Residual Network model, where the first feature extraction layer and the second feature extraction layer are both composed of a series of convolutional layers, and each convolutional layer may adopt a convolutional structure of a Residual Network (Residual Network), a dense Connected Network (densnet), a SEResnet Network, or an initiation Network. In the embodiment of the present application, the convolution operation performed by the first feature extraction layer is a long term convolution operation (LTCP), and the convolution operation performed by the second feature extraction layer is a short term convolution operation (STCP). The feature fusion layer is also composed of a series of convolution layers, and two or more features are fused by convolution operation.

In practical implementation, after the server performs image decomposition on the image to be processed to obtain at least two target images, the image to be processed and the at least two target images are respectively input into the image interference removal model. Specifically, the server inputs the image to be processed to a first feature extraction layer of the image interference removal model, and inputs the target image to a second feature extraction layer of the image interference removal model. And then, performing feature extraction on the image to be processed through a first feature extraction layer of the image interference removal model to obtain the image features of the image to be processed. Here, the image feature includes image information of the image to be processed and interference information.

And 103, respectively extracting interference features of each target image through a second feature extraction layer of the image interference removal model to obtain corresponding interference features.

In practical implementation, the server inputs each target image to the image interference elimination model respectively, so that the image interference elimination model extracts the interference features of the target image. Here, the interference characteristic is an encoded representation of interference information present in the target image. In the embodiment of the application, the image interference removing model extracts interference information in the target image by learning the residual error of the target image, so as to obtain corresponding interference characteristics.

In some embodiments, the second feature extraction layer includes N sub-feature extraction layers, and the number of target images is N, where N is a positive integer not less than 2. Based on fig. 3, step 103 can also be implemented as follows: and the server inputs each target image into a sub-feature extraction layer, and performs interference feature extraction on the target images through the sub-feature extraction layer to obtain corresponding interference features.

Referring to fig. 19, fig. 19 is an alternative structural diagram of an image interference elimination model provided in the embodiment of the present application. It should be noted that the structures of the sub-feature extraction layers may not be completely consistent, and the parameters may also be different for two sub-feature extraction layers with consistent structures. In practical implementation, the server inputs each target image into one sub-feature extraction layer respectively, and extracts different target images through different sub-feature extraction layers respectively. According to the method and the device, each target image obtained by image decomposition of the image to be processed has different scale information, so that the target images with different scales are respectively extracted through the sub-feature extraction layers with different structures or parameters, interference features with different scales can be pertinently extracted, and the accuracy of feature extraction is improved.

And step 104, carrying out fusion processing on the interference characteristics of each target image through the characteristic fusion layer of the image interference removal model to obtain fusion interference characteristics.

In practical implementation, the server performs feature fusion processing on the interference features of all target images, and fuses the interference features corresponding to each target image into a fusion interference feature. It is understood that the fused interference features include interference information for all target images. Specifically, the server performs convolution operation on each interference feature through a feature fusion layer of the image interference removal model, and fuses all the interference features into fusion interference features through the convolution operation.

In some embodiments, the interference features include size features and channel features. Referring to fig. 20, fig. 20 is an optional flowchart of an image interference removing method according to an embodiment of the present application, and based on fig. 3, step 104 may also be implemented as follows:

step 501, the server performs size transformation on the interference features of each target image through a feature fusion layer of an image interference removal model to obtain target interference features corresponding to each interference feature; wherein the size characteristics of each target interference characteristic are the same.

It should be noted that, in the embodiment of the present application, an image includes information of two dimensions, i.e., an image feature of the image includes a size feature and a channel feature, where the size feature includes a width feature and a height feature, for example, the image combination B shown in fig. 11_iIs characterized by the dimension of (W/2)ⁱ,H/2ⁱ)，W/2ⁱFor width characteristics, H/2ⁱIs a height feature. The channel characteristics of the image comprise sub-channel characteristics corresponding to each pixel point of the image, wherein the sub-channel characteristics comprise color of a pixel pointThe channel value of each monochromatic channel in the color mode may be an RGB (Red, Green, Blue, Red, Green, Blue) color mode, and then the sub-channel feature of the pixel point includes the channel values of three channels, i.e., Red, Green, Blue, and the like. Illustratively, referring to fig. 21, fig. 21 is an optional schematic diagram of image features provided in this embodiment of the present application, where fig. 21 shows an image with a width feature of W and a height feature of H, where the subchannel feature of the pixel point in the first row and the first column is (0,0,0), the subchannel feature of the pixel point in the second row and the second column is (56,56,56), the subchannel feature of the pixel point in the first row and the first column is (56,56,56), the subchannel feature of the pixel point in the first column and the second row is (255,255,255,255), and so on.

In the embodiment of the application, the feature dimension of the image feature of the interference feature is consistent with that of the corresponding target image, and the image feature of the interference feature also has a size feature and a channel feature, and the size feature of the interference feature is consistent with that of the corresponding target image. For example, if the size characteristic of the target image is (W/2)ⁱ,H/2ⁱ2 × 3), then the size feature of the interference feature obtained after the interference feature extraction on the target image is also (W/2)ⁱ,H/2ⁱ2 × 3). And the channel characteristic of the interference characteristic corresponds to the channel characteristic corresponding to the interference information in the target image.

In actual implementation, the scales of the target images are different, the size characteristics of the target images are also different, and when the characteristics are fused, the server firstly carries out size transformation on the interference characteristics of the target images through a characteristic fusion layer of an image interference removal model, so that the size characteristics of the interference characteristics corresponding to the target images are consistent. In this embodiment, the size conversion is realized by an upsampling method.

And 502, performing fusion processing on the channel characteristics of the target interference characteristics to obtain fusion channel characteristics.

In practical implementation, the server can perform feature splicing on the channel features of all the target interference features to obtain a multi-dimensional splicing channel feature, and then fuse the splicing channel feature into a fusion channel feature of one dimension through convolution operation. The server can also directly carry out convolution operation on all target interference characteristics so as to directly fuse the fusion channel characteristics of one dimension.

And 503, performing feature splicing on the size feature of the target interference feature and the fusion channel feature to obtain a fusion interference feature.

The size features and the fusion channel features are subjected to feature splicing, so that the obtained fusion interference features have the features of size dimension and channel dimension, the feature fusion of all target interference features is completed, and the obtained fusion interference features have interference information in target images of all scales.

In some embodiments, when the number of target images is m, the corresponding number of interference features is m. Referring to fig. 22, fig. 22 is an optional flowchart of an image interference removing method according to an embodiment of the present application, and based on fig. 3, step 104 may also be implemented as follows:

601, fusing a jth interference feature and a (j + 1) th interference feature in m interference features by a server through a feature fusion layer of the image interference removal model to obtain a jth fusion interference feature; wherein m is a positive integer not less than 3, j is a positive integer, and j belongs to [1, m-1 ].

And step 602, fusing the jth fusion interference feature with the (j + 2) th interference feature to obtain a (j + 1) th fusion interference feature.

And 603, beginning to take the value of j as 1, traversing j, and taking the j + 1-th fusion interference feature as the fusion interference feature when the value of j +2 is the same as the value of m.

In the embodiment of the application, the server sorts the interference features from small to large in size, and it can be understood that the size of the jth interference feature is smaller than that of the (j + 1) th interference feature. In practical implementation, the server performs feature fusion on each interference feature step by step from the interference feature with the smallest size through a feature fusion layer of the image interference elimination model. In practical implementation, the server traverses j from the value of j being 1 through a feature fusion layer of the image interference removal model, performs feature fusion on the traversed jth interference feature and the jth +1 interference feature to obtain a jth fusion interference feature, then continues traversing to obtain a jth +2 interference feature, fuses the jth fusion interference feature and the jth +2 interference feature to obtain a jth +1 fusion interference feature, and then continues traversing j to execute the steps until traversing is completed to obtain a final fusion interference feature. Here, for the fusion process of the jth interference characteristic and the jth +1 th interference characteristic, reference may be made to the embodiment of the present application shown in fig. 19, which is not described herein again.

Exemplarily, referring to fig. 23, fig. 23 is an optional schematic diagram of an image interference elimination process provided by the embodiment of the present application. In practical implementation, the feature fusion layer of the image interference elimination model performs feature fusion on the interference features from the interference features corresponding to the target image with the smallest size. Here the target image B₀、B₁、……、B_i、B_i+1Referring to the image combination pyramid shown in fig. 17, the size of the target image decreases sequentially as the hierarchy increases. In practical implementation, the image de-interference model is subjected to feature fusion layer extraction from the target image B_i+1Corresponding interference characteristic P (B)_i+1) And starting to perform feature fusion on each interference feature layer by layer. Specifically, the feature fusion layer of the image interference elimination model is used for interfering with the feature P (B)_i+1) Upsampling is performed to obtain the corresponding upsampling characteristic up (P (B)_i+1) Here the upsampling operation is to deconvolve the interference features to better predict the more scaled features, the upsampling feature up (P (B)_i+1) The size characteristic of) is compared with the target image B_iCorresponding interference characteristic P (B)_i) Is consistent, and then the upsampling feature up (P (B) with consistent size feature is used_i+1) And interference characteristic P (B)_i) Performing characteristic splicing of channel characteristics in channel dimension to obtain splicing interference characteristics M (up (P (B) after characteristic splicing_i+1) )) and then the splice disturbance signature M (up (P (B)) is applied_i+1) ) is upsampled such that its dimensional characteristics are associated with the target image B_i-1Corresponding interference characteristic P (B)_i-1) And then performing feature splicing on the two images until all target images are processed to obtain the final target splicing interference feature, and then performing feature splicing on the two imagesThe target splicing interference feature is convolved into a one-dimensional channel feature by a convolution operation, so that a fusion interference feature P (I) with the size of (W, H,3) is obtained₀)。

And 105, performing interference elimination processing on the image to be processed through an output layer of the image interference elimination model based on the fusion interference characteristic and the image characteristic to obtain a corresponding interference elimination image.

In some embodiments, based on fig. 3, step 105 may also be implemented by: the server determines the difference characteristic between the image characteristic and the fusion interference characteristic through an output layer of the image interference removal model; and decoding the difference characteristic through an output layer of the image interference removal model to obtain an interference removal image.

In practical implementation, the server outputs the image I to be processed through the output layer of the image interference elimination model₀Image feature F (I)₀) With fusion of the interference feature P (I)₀) Subtracting to obtain the difference characteristic pred (I) of the two₀) And then, decoding the difference characteristic through an output layer of the image interference removal model, converting the difference characteristic of the coding type into an image, wherein the image is the interference removal image corresponding to the image to be processed. Thus, the interference removing processing of the image to be processed is completed.

In the embodiment of the application, the server decomposes the image to be processed into a plurality of target images, each target image contains the image information of the image to be processed, and along with the decomposition of the image to be processed, the image information of the image to be processed is also decomposed into a plurality of representations, so that the image information of the image to be processed can be more clearly and comprehensively represented by the plurality of target images, then the interference characteristics of each target image are extracted by using an image interference removing model, the interference characteristics are fused, the obtained fusion interference characteristics correspondingly cover more and more accurate interference information in the image to be processed, and therefore the interference information in the image to be processed can be accurately removed based on the fusion interference characteristics.

In some embodiments, based on fig. 3, prior to step 102, the image de-interference model is also trained. Here, referring to fig. 24 specifically for a training process of an image interference elimination model, fig. 24 is an optional flowchart of a training method of an image interference elimination model provided in an embodiment of the present application, and an embodiment of the present application provides a training method of an image interference elimination model, including:

step 701, a server carries out image decomposition on a sample interference image carrying interference information to obtain at least two sample target images; and adding interference information to the original sample image to obtain the sample interference image.

In practical implementation, the server acquires a sample original image with high image quality, and adds interference information to the sample original image to obtain a corresponding sample interference image. In some embodiments, the interference information includes at least one of image aliasing and image noise, and based on fig. 24, before step 701, the following may be further performed: adding image sawteeth to a sample original image by a server to obtain a sample sawteeth image carrying the image sawteeth, and taking the sample sawteeth image as a sample interference image; or adding image noise to the original sample image to obtain a sample noise image carrying the image noise, and taking the sample noise image as a sample interference image; or adding image saw-teeth to the original sample image to obtain a sample saw-tooth image carrying the image saw-teeth, adding image noise to the sample saw-tooth image to obtain a sample saw-tooth noise image carrying both the image saw-teeth and the image noise, and taking the sample saw-tooth noise image as a sample interference image.

In actual implementation, the server may add noise to the sample original image to obtain a sample noise image carrying image noise, may also add a sawtooth to the sample original image to obtain a sample noise image carrying image sawtooth, may also add noise and sawtooth to the sample original image to obtain a sample sawtooth noise image carrying image noise and image sawtooth, and may also add other interference information, such as adding a blocking block, to the sample original image to obtain another sample interference image carrying other interference information. Each sample original image may correspond to one or more sample interference images.

Exemplarily, referring to fig. 25, fig. 25 is an alternative schematic diagram of a sample set provided by the embodiment of the present application. Here, the sample set includes a sample original image set and a sample interference image set. Each sample original image corresponds to a sample interference image set formed by a plurality of sample interference images with different interference information. The set of sample interference images includes at least one of a set of sample noise images, a set of sample sawtooth noise images, and a set of other sample interference images. In practical implementation, after determining a sample original image, a server randomly selects a sample interference image from a sample interference image set corresponding to the sample original image, so as to form an image with the sample original image, and the image is input into the image interference removal model of the embodiment of the application to train the image interference removal model.

In practical implementation, the server may perform image storage format conversion on the sample original image to obtain a sample sawtooth image carrying image sawtooth, for example, convert an image in a Portable Network Graphics format (png) into an image in a Joint Photographic Experts Group (JPEG) standard format, so as to add image sawtooth to the sample original image in the conversion process of the image storage format. The server may also perform format conversion on the sample original image in a channel dimension to obtain a sample sawtooth image carrying image sawtooth, for example, convert an RGB image into an RGGB image to add image sawtooth to the sample original image in an image conversion process. In some embodiments, the image jaggies may be added to the sample original image by other means, which is not limited in this application. In practical implementation, the server may add image noise to the sample original image by adding a white gaussian noise with an average value of 0 and a variance of theta to the sample original image to obtain a corresponding sample noise image. In practical implementations, the server may add image noise to the sample jagged image after adding the image jaggy to the sample original image, resulting in a sample jagged noise image, or add image jaggy to the sample noise image after adding the image noise to the sample original image, resulting in a sample jagged noise image, or the like. After the server adds the interference information to the multiple original sample images, a sample set consisting of the multiple original sample images and the corresponding interference sample images is constructed.

In practical implementation, the server may select one or more sample original images from the sample set, select one corresponding sample interference image from a sample interference image set corresponding to the sample original image based on the sample original image, and perform image decomposition on the sample interference image to obtain at least two sample target images.

And 702, performing feature extraction on the sample interference image through a first feature extraction layer of an image interference removal model to obtain sample image features of the sample interference image.

And 703, respectively extracting interference features of each sample target image through a second feature extraction layer of the image interference removal model to obtain corresponding sample interference features.

And 704, performing fusion processing on the sample interference characteristics corresponding to each sample target image through the characteristic fusion layer of the image interference removal model to obtain sample fusion interference characteristics.

Step 705, performing interference elimination processing on the sample interference image through an output layer of the image interference elimination model based on the sample fusion interference feature and the sample image feature to obtain a corresponding prediction interference elimination image.

Step 706, updating model parameters of the image interference elimination model based on the difference between the predicted interference elimination image and the sample original image.

In practical implementation, the server inputs the sample interference image and the at least two sample target images into the image interference removal model to obtain sample image features of the sample interference image and interference features corresponding to the at least two sample target images, the interference features are fused through the feature fusion layer to obtain sample fusion interference features, and then sample difference features of the sample image features and the sample fusion interference features are calculated through the output layer of the image interference removal model. The server also inputs the sample original image into the image interference removal model, extracts the sample original image characteristics of the sample original image through a first characteristic extraction layer of the image interference removal model, and then updates the model parameters of the image interference removal model based on the difference between the sample original image characteristics and the sample difference characteristics, specifically, the server respectively updates the parameters of the first characteristic extraction layer, the second characteristic extraction layer, the characteristic fusion layer and the output layer. Here, the difference between the original image feature of the sample and the difference feature of the sample is obtained by the server by calculating a value of a loss function corresponding to the image interference removal model. And the server iterates the training process until the loss function is converged, and completes the training of the image interference removal model. Here, the loss function may specifically calculate a sum of euclidean distances of the predicted interference-removed image and all pixel points of the sample original image by using a euclidean distance formula, and may be expressed as follows:

Loss(I₀,J₀)＝(1/(2*W*H))*∑_{i\in pred(I₀),j\in J₀}||i-j||2；

wherein, I₀For sample interference images, J₀Pred (I) for the sample original image₀) To predict the de-interference image.

In some embodiments, the sample original image corresponds to at least two sample interference images, each sample interference image is obtained by adding interference information to the sample original image, and accordingly, the image interference removing method further includes: and the server executes the operations of carrying out image decomposition on the sample interference image carrying the interference information and carrying out interference removal processing on the sample interference image respectively aiming at each sample interference image to obtain a prediction interference removal image corresponding to each sample interference image. Correspondingly, the updating the model parameters of the image interference elimination model based on the difference between the prediction interference elimination image and the sample original image comprises: and updating the model parameters of the image interference removal model by the server based on the difference between each predicted interference removal image and the original image of the sample.

In actual implementation, the interference information carried by each sample interference image is different, for example, the interference information may be image noise with different noise intensities, or different types of interference information, for example, image noise and image aliasing. In the embodiment of the application, the server can add various interference information to the original sample image to obtain the corresponding interference images of the multiple samples. In some embodiments, the server may further obtain a sample original image and a sample interference image set corresponding to the sample original image based on the sample set shown in fig. 23. Then, the server executes the image interference removing process for each sample interference image to obtain a sample difference characteristic corresponding to each sample interference image, wherein the image corresponding to the sample difference characteristic is the predicted interference removing image. The server also inputs the sample original image into the image interference removal model for feature extraction to obtain the sample original image features, and then the server updates the model parameters of the image interference removal model based on the difference between the sample difference features corresponding to the sample interference images and the sample original image features. In some embodiments, the server may calculate a center feature of the difference features of each sample, and update the model parameters of the image de-interference model based on a difference between the center feature and the original image features of the samples. Here, the central feature may be a mean value of the difference features of the respective samples.

In the embodiment, the sample interference images containing various interference information corresponding to the sample original images are jointly input into the image interference removing model for training, so that the image interference removing model can learn more interference information with higher intensity or types, and the interference removing accuracy of the image interference removing model is further improved.

In some embodiments, the second feature extraction layer includes N sub-feature extraction layers, and the number of target images is N, where N is a positive integer not less than 2. The method for extracting the interference characteristics of each sample target image to obtain the corresponding sample interference characteristics comprises the following steps: and inputting each target image into a sub-feature extraction layer, and extracting interference features of the target images through the sub-feature extraction layers to obtain corresponding sample interference features. Accordingly, based on fig. 24, step 706 includes: and respectively updating the parameters of the first feature extraction layer, the parameters of each second feature extraction layer, the parameters of the feature fusion layer and the parameters of the output layer based on the difference between the predicted interference-removed image and the sample original image.

In the embodiment of the application, the server decomposes the sample interference image into a plurality of sample target images, each sample target image contains image information of the sample interference image, and the image information of the sample interference image is also decomposed into a plurality of representations along with the decomposition of the sample interference image, so that the image information of the sample interference image can be more clearly and comprehensively represented by the plurality of sample target images, then the interference characteristics of each sample target image are extracted by using an image interference elimination model, the interference characteristics are fused, the obtained fusion interference characteristics correspondingly cover more and more accurate interference information in the sample interference image, therefore, the interference information in the sample interference image can be accurately eliminated based on the fusion interference characteristics, a higher-quality predicted interference image is obtained, and then the image interference elimination model is trained based on the difference between the predicted interference image and the original image of the sample, the image interference removing model obtained by training has high-accuracy interference removing effect.

Continuing with the description of the image interference removal method provided in the embodiment of the present application, fig. 26 is an optional flowchart of the image interference removal method provided in the embodiment of the present application, and referring to fig. 26, the image interference removal method provided in the embodiment of the present application is cooperatively implemented by a client and a server.

In step 801, the client acquires a sample original image in response to an upload operation for the sample original image.

Here, the client may be an image interference-free client disposed in the terminal, the original sample image may be triggered by a user based on a human-computer interaction interface of the client to enable the client to present an image selection interface on the human-computer interaction interface, and the user locally uploads the original sample image from the terminal based on the image selection interface, so that the client obtains the uploaded original sample image.

In some embodiments, the sample original image may also be obtained by shooting with a camera in communication connection with the terminal, and after the sample original image is obtained by shooting with the camera, the sample original image is transmitted to the terminal and automatically uploaded to the client by the terminal.

Step 802, the client sends the sample original image to the server.

Step 803, the server adds interference information to the original sample image to obtain a sample interference image carrying the interference information.

Here, the server may add at least one of image noise and image aliasing to the sample original image, and the server may also add other disturbance information to the sample original image, such as adding occlusion blocks to the sample original image, and so on.

And step 804, the server carries out image decomposition on the sample interference image to obtain at least two sample target images.

Here, the server may construct a gaussian pyramid, a laplacian pyramid, and a downsampled image pyramid corresponding to the gaussian pyramid corresponding to the sample interference image, combine images of the same size in the gaussian pyramid, the laplacian pyramid, and the downsampled image pyramid to obtain a corresponding image combination pyramid, and combine each layer of images in the image combination pyramid as the target image.

Step 805, the server inputs the sample interference image and at least two sample target images into the image interference elimination model, so that the image interference elimination model performs image interference elimination processing on the sample interference image to obtain a corresponding prediction interference elimination image.

The first feature extraction layer of the image interference removal model performs feature extraction on the sample interference image to obtain sample image features, the second feature extraction layer performs interference feature extraction on each target image to obtain sample interference features corresponding to each target image, then the feature fusion layer performs feature fusion on each sample interference feature to obtain sample fusion interference features, and then based on the sample image features and the fusion interference features, the sample interference image is subjected to interference removal processing to obtain a prediction interference image.

In step 806, the server updates the model parameters of the image interference removal model based on the difference between each pixel point of the predicted interference removal image and the corresponding pixel point in the sample original image.

The server calculates and predicts the sum of differences of each pixel point of the interference-removed image and the corresponding pixel point in the original sample image by using the Euclidean distance loss function to obtain the value of the Euclidean distance loss function, and updates the model parameters of the image interference-removed model based on the value. And the server iterates the training process until the Euclidean distance loss function converges, and completes the training of the image interference removal model.

In step 807, the server generates a prompt message indicating that the training of the image interference elimination model is completed.

Step 808, the server sends a prompt message to the client.

In step 809, the client acquires the image to be processed in response to the uploading operation for the image to be processed.

Here, the image to be processed may also be transmitted to the client by another device communicatively connected to the terminal.

In step 810, the client sends the image to be processed to the server in response to the interference elimination instruction for the image to be processed.

Here, the interference elimination instruction may be automatically generated by the client after receiving a message that the training of the image interference elimination model is completed, or may be automatically generated by the client under a certain trigger condition, for example, the interference elimination instruction for a to-be-processed image is automatically generated by the client after the client acquires the to-be-processed image, or may be sent to the client by another device in communication connection with the terminal, or may be generated by the user after triggering a corresponding interference elimination function item based on a human-computer interaction interface of the client.

Step 811, the server performs image decomposition on the image to be processed to obtain at least two target images.

Step 812, the server inputs the image to be processed and the at least two target images into the image interference elimination model, so that the image interference elimination model performs interference elimination processing on the image to be processed based on the at least two target images to obtain corresponding interference elimination images.

In step 813, the server sends the interference free image to the client.

In step 814, the client outputs the interference image.

Here, the client may present the interference-free image in a human-computer interaction interface of the client, may also store the interference-free image to the local terminal, and may also send the interference-free image to other devices in communication connection with the terminal, and the like.

In the embodiment of the application, the client sends the sample interference image to the server, so that the server decomposes the sample interference image into a plurality of sample target images, each sample target image contains image information of the sample interference image, and along with the decomposition of the sample interference image, the image information of the sample interference image is also decomposed into a plurality of representations, so that the image information of the sample interference image can be more clearly and comprehensively represented by the plurality of sample target images, then the interference characteristics of each sample target image are extracted by using an image interference removal model, and the interference characteristics are fused, and the obtained fused interference characteristics correspondingly cover more and more accurate interference information in the sample interference image, so that the interference information in the sample interference image can be accurately removed based on the fused interference characteristics, and a higher-quality predicted interference image is obtained, and then training the image interference removing model based on the difference between the predicted interference image and the original image of the sample, so that the image interference removing model obtained by training has a high-accuracy interference removing effect. Then, the interference removing processing is carried out on the image to be processed based on the image interference removing model, and the high-quality interference removing image can be obtained.

Next, an exemplary application of the embodiment of the present application in a practical application scenario will be described. The image interference removing method provided by the embodiment is implemented by a server for explanation.

The server obtains N RGB images with high image quality and storage format png format, that is, sample original images, here, a set of N sample original images with high image quality is denoted as I _1, I _2, …, I _ N, where N is a positive integer. Then, the server generates a low-quality sawtooth image and a noise image based on the N to-be-processed images. The server may generate the sawtooth image by: the server converts I _ v (v ∈ [1, N ]) into a corresponding RGGB image I _ v ', and the generated RGGB image composition set I ' { I _1 ', I _2 ', …, I _ N ' }; the server also converts the storage format of I _ v from png format to jpg format, and obtains a corresponding jpg image I _ v ", and the generated jpg image constitutes a set I ″ { I _1 ″, I _2 ″, …, I _ N ″ }. It can be understood that the generated RGGB image i _ v' and jpg image i _ v "carry image jaggies, which are sample jaggy images corresponding to the sample original image i _ v. The server may generate the noise image by: the server adds a gaussian noise with mean 0 and variance theta to the image I _ v to generate an image I _ v, and the generated image constitutes a set I _1, I _2, …, I _ N. Here, the server may set theta to 0.25. It can be understood that the generated image i _ v carries image noise, and is a sample noise image corresponding to the sample original image i _ v.

In actual implementation, the server trains the image interference removal model based on the set I, the set I', the set I ″ and the set I ″. The server may randomly select one set from the set I', the set I ", and the set I as a sample interference image set, and then train an image de-interference model based on the set I and the selected sample interference image set. The server also inputs the set I, the set I' and the set I-I into the image interference elimination model together to train the image interference elimination model.

In practical implementation, referring to fig. 17, the server constructs a corresponding gaussian pyramid, a laplacian pyramid, and four downsampled image pyramids as shown in fig. 17 for each sample interference image. Here, a Gaussian pyramid, Laplaca, is constructedFor the process of the pyramid and the four downsampling image pyramids, reference is made to the above embodiments of the present application, which are not described herein again. And then, the server performs image combination on the images with the same size in the Gaussian pyramid, the Laplacian pyramid and the four down-sampling image pyramids to obtain a corresponding image combination pyramid. Illustratively, the number of layers of the constructed gaussian pyramid is 4, in the embodiment of the present application, for one sample interference image, the server performs image decomposition on the sample interference image to obtain four image blocks with different sizes, which are respectively B₀＝{I₀,L₀}，B₁＝{I₁,L₁,ds0(I₀),ds1(I₀),ds2(I₀),ds3(I₀)}，B₂＝{I₂,L₂,ds0(I₁),ds1(I₁),ds2(I₁),ds3(I₁)}，B₃＝{I₃,ds0(I₂),ds1(I₂),ds2(I₂),ds3(I₂)}. The number of channels per image is 3, and the sizes of the four image blocks are (W, H, 2x3), (W/2, H/2, 6x3), (W/4, H/4, 6x3), (W, H, 5x3), respectively.

See the structure of the image de-interference module shown in fig. 19 and the process of image de-interference shown in fig. 23. Here, the image interference elimination model is a residual network model and includes a first feature extraction layer, a second feature extraction layer, a feature fusion layer, and an output layer. And the second feature extraction layer comprises a plurality of sub-feature extraction layers with the same number as the image blocks. The structures or parameters of the sub-feature extraction layers are not completely consistent. Here, the first feature extraction layer and each sub-feature extraction layer are each configured by a plurality of convolutional layers, and each convolutional layer may have a convolutional structure of a resnet, densenet, SEResnet, or acceptance network. Then, the server inputs the sample interference image and the corresponding four image blocks into an image interference removal model, so that the image interference removal model extracts image features F (I) of the image to be processed through a first feature extraction layer₀) Here, the first feature extraction layer performs feature extraction on the image to be processed by using a long-term convolution operation, and obtains image features including image information and interference information of the image to be processed. Then pass throughAnd extracting interference features of each image block by each sub-feature extraction layer of the image interference removal model, wherein each sub-feature extraction layer adopts short-term convolution operation to extract the interference features. Then, the image block B is processed by the feature fusion layer of the image interference elimination model₃Initially, the operation of feature fusion is performed step by step, specifically, for the image block B₃Up-sampling to obtain image block B₃Is converted into image blocks B₂After the images are consistent, the up-sampled image block B is₃And image block B₂Performing characteristic splicing of channel dimension, wherein the size of the spliced image blocks is (W/2, H/2, 11x3), obtaining splicing interference characteristics after characteristic splicing of all the image blocks by gradually performing the above operation, wherein the size of the splicing interference characteristics is (W/2, H/2, 19x3), then performing characteristic fusion of the channel dimension on the splicing interference characteristics with the size of (W/2, H/2, 19x3) by an image interference removing model through a convolution layer in a characteristic fusion layer to obtain a fusion interference characteristic P (I) with the size of (W/2, H/2, 3)₀) Then, the output layer converts the image characteristics F (I) of the image to be processed₀) With fusion of the interference feature P (I)₀) Subtracting to obtain the sample difference characteristic pred (I) of the two₀). In practical implementation, the server further inputs the sample original image into the image interference removal model, performs feature extraction on the sample original image through the first feature extraction layer of the image interference removal model to obtain sample original image features F (I) of the sample original image, and the server obtains the sample original image features pred (I) based on the sample original image features F (I) and the sample difference features pred (I)₀) Determining a loss function value corresponding to the image interference elimination model, updating model parameters of the image interference elimination model based on the loss function value, iterating the model training until the loss function is converged, and finishing the training of the image interference elimination model.

After the training of the image interference elimination model is completed, the image interference elimination model can be used for carrying out interference elimination processing on any image. In practical implementation, the server will input one image to be processed, first calculate its gaussian pyramid, laplacian pyramid and corresponding four downsampled images ds0, ds1, ds2 and ds3, and then construct data blocks B0, B1, B2 and B3. Then the image to be processed and the constructed data blocks B0, B1, B2 and B3 are input into the trained image interference elimination model, and the image output by the image interference elimination model is a high-quality interference elimination image with the same size as the previously input image to be processed.

Continuing with the exemplary structure of the image de-jamming device 555 provided by the embodiment of the present application implemented as a software module, in some embodiments, referring to fig. 27, fig. 27 is an alternative schematic diagram of the image de-jamming device provided by the embodiment of the present application, and the software module stored in the image de-jamming device 555 of the memory 550 may include:

the image decomposition module 5551 is configured to perform image decomposition on the to-be-processed image carrying the interference information to obtain at least two target images;

the image feature extraction module 5552 is configured to perform feature extraction on the image to be processed through a first feature extraction layer of an image interference removal model to obtain an image feature of the image to be processed;

the interference feature extraction module 5553 is configured to perform interference feature extraction on each target image through a second feature extraction layer of the image interference removal model to obtain corresponding interference features;

a feature fusion module 5554, configured to perform fusion processing on the interference features of each target image through a feature fusion layer of the image interference removal model to obtain fusion interference features;

and the interference removing module 5555 is configured to perform interference removing processing on the image to be processed based on the fusion interference feature and the image feature through an output layer of the image interference removing model to obtain a corresponding interference removed image.

In some embodiments, the image decomposition module 5551 is further configured to perform multi-scale image decomposition on the to-be-processed image carrying the interference information to construct a gaussian pyramid corresponding to the to-be-processed image; and taking the image of each layer in the Gaussian pyramid as the target image.

In some embodiments, the image decomposition module 5551 is further configured to perform multi-scale image decomposition on the to-be-processed image carrying the interference information to construct a gaussian pyramid corresponding to the to-be-processed image; constructing a Laplacian pyramid corresponding to the image to be processed based on the Gaussian pyramid corresponding to the image to be processed; combining the images of all layers in the Gaussian pyramid with the images of corresponding layers in the Laplacian pyramid respectively to obtain at least two image combinations; combining the at least two images as the target image.

In some embodiments, the image decomposition module 5551 is further configured to perform multi-scale image decomposition on the to-be-processed image carrying the interference information to construct a gaussian pyramid corresponding to the to-be-processed image; constructing a Laplacian pyramid corresponding to the image to be processed based on the Gaussian pyramid corresponding to the image to be processed; respectively carrying out down-sampling processing on the images of all layers in the Gaussian pyramid to obtain corresponding down-sampled images; combining the images of all layers in the Gaussian pyramid, the images of the corresponding levels in the Laplacian pyramid and the downsampled images with the same size as the images of the corresponding levels in the Laplacian pyramid respectively to obtain at least two image combinations; combining the at least two images as the target image.

In some embodiments, the second feature extraction layer includes N sub-feature extraction layers, the number of the target images is N, and N is a positive integer not less than 2; the interference feature extraction module 5553 is further configured to input each target image into one sub-feature extraction layer, and perform interference feature extraction on the target image through the sub-feature extraction layer to obtain a corresponding interference feature.

In some embodiments, the interference features include size features and channel features, and the feature fusion module 5554 is further configured to perform size transformation on the interference features of each target image respectively to obtain target interference features corresponding to each interference feature; wherein the size characteristics of each target interference characteristic are the same; performing fusion processing on the channel characteristics of the target interference characteristics to obtain fusion channel characteristics; and performing characteristic splicing on the size characteristic of the target interference characteristic and the fusion channel characteristic to obtain the fusion interference characteristic.

In some embodiments, when the number of the target images is m, and correspondingly, the number of the interference features is m, the feature fusion module 5554 is further configured to fuse a jth interference feature and a j +1 th interference feature of the m interference features to obtain a jth fused interference feature; wherein m is a positive integer not less than 3, j is a positive integer, and j belongs to [1, m-1 ]; fusing the jth fusion interference feature with the (j + 2) th interference feature to obtain a (j + 1) th fusion interference feature; and starting to take the value of j as 1, traversing j, and taking the j +1 th fusion interference feature as the fusion interference feature when the value of j +2 is the same as the value of m.

In some embodiments, the interference elimination module 5555 is further configured to determine a difference feature between the image feature and the fused interference feature through an output layer of the image interference elimination model; and decoding the difference characteristic through an output layer of the image interference removal model to obtain the interference removal image.

In some embodiments, the software modules stored in the image de-jamming device 555 in the memory 550 may include: the model training module is used for carrying out image decomposition on the sample interference image carrying the interference information to obtain at least two sample target images; the sample interference image is obtained by adding interference information to the original sample image; performing feature extraction on the sample interference image through a first feature extraction layer of the image interference removal model to obtain sample image features of the sample interference image; respectively extracting interference features of each sample target image through a second feature extraction layer of the image interference removal model to obtain corresponding sample interference features; performing fusion processing on the sample interference features corresponding to the sample target images through the feature fusion layer of the image interference removal model to obtain sample fusion interference features; performing interference elimination processing on the sample interference image based on the sample fusion interference characteristic and the sample image characteristic through an output layer of the image interference elimination model to obtain a corresponding prediction interference elimination image; updating model parameters of the image de-interference model based on a difference between the predicted de-interference image and the sample original image.

In some embodiments, the sample original image corresponds to at least two sample interference images, each of the sample interference images is obtained by adding interference information to the sample original image, and correspondingly, the image decomposition module 5551 is further configured to perform, for each of the sample interference images, the operation from performing image decomposition on the sample interference image carrying the interference information to performing interference removal processing on the sample interference image, so as to obtain a predicted interference removal image corresponding to each of the sample interference images; correspondingly, the model training module is further configured to update the model parameters of the image interference elimination model based on the difference between each predicted interference elimination image and the sample original image.

In some embodiments, the interference information includes at least one of image aliasing and image noise; accordingly, the software modules stored in the image de-noising device 555 in the memory 550 may include: the interference information adding module is used for adding image saw teeth to the sample original image to obtain a sample saw tooth image carrying the image saw teeth, and the sample saw tooth image is used as the sample interference image; or adding image noise to the sample original image to obtain a sample noise image carrying the image noise, and taking the sample noise image as the sample interference image; or adding image saw-teeth and image noise to the sample original image to obtain a sample saw-tooth noise image simultaneously carrying the image saw-teeth and the image noise, and taking the sample saw-tooth noise image as the sample interference image.

It should be noted that the description of the apparatus in the embodiment of the present application is similar to the description of the method embodiment, and has similar beneficial effects to the method embodiment, and therefore, the description is not repeated.

a memory for storing executable instructions;

and the processor is used for realizing the image interference removing method or the training method of the image interference removing model provided by the embodiment of the application when the executable instructions stored in the memory are executed.

The embodiment of the present application provides a computer-readable storage medium, which stores executable instructions and is used for implementing the image interference elimination method or the training method of the image interference elimination model provided in the embodiment of the present application when being executed by a processor.

Embodiments of the present application provide a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to execute the image interference elimination method or the training method of the image interference elimination model according to the embodiment of the present application.

Embodiments of the present application provide a computer-readable storage medium storing executable instructions, which when executed by a processor, will cause the processor to perform the method provided by embodiments of the present application, for example, the image de-interference method as shown in fig. 3, or the training method of the image de-interference model as shown in fig. 24.

In some embodiments, the computer-readable storage medium may be memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

In some embodiments, executable instructions may be written in any form of programming language (including compiled or interpreted languages), in the form of programs, software modules, scripts or code, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

By way of example, executable instructions may correspond, but do not necessarily have to correspond, to files in a file system, and may be stored in a portion of a file that holds other programs or data, such as in one or more scripts in a hypertext Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code).

By way of example, executable instructions may be deployed to be executed on one computing device or on multiple computing devices at one site or distributed across multiple sites and interconnected by a communication network.

In conclusion, the interference information in the image can be accurately removed through the embodiment of the application.

The above description is only an example of the present application, and is not intended to limit the scope of the present application. Any modification, equivalent replacement, and improvement made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An image de-interference method, the method comprising:

2. The method according to claim 1, wherein the image decomposition of the to-be-processed image carrying the interference information to obtain at least two target images comprises:

carrying out multi-scale image decomposition on an image to be processed carrying interference information to construct a Gaussian pyramid corresponding to the image to be processed;

and taking the image of each layer in the Gaussian pyramid as the target image.

3. The method according to claim 1, wherein the image decomposition of the to-be-processed image carrying the interference information to obtain at least two target images comprises:

constructing a Laplacian pyramid corresponding to the image to be processed based on the Gaussian pyramid corresponding to the image to be processed;

combining the images of all layers in the Gaussian pyramid with the images of corresponding layers in the Laplacian pyramid respectively to obtain at least two image combinations;

combining the at least two images as the target image.

4. The method according to claim 1, wherein the image decomposition of the to-be-processed image carrying the interference information to obtain at least two target images comprises:

respectively carrying out down-sampling processing on the images of all layers in the Gaussian pyramid to obtain corresponding down-sampled images;

combining the images of all layers in the Gaussian pyramid, the images of the corresponding levels in the Laplacian pyramid and the downsampled images with the same size as the images of the corresponding levels in the Laplacian pyramid respectively to obtain at least two image combinations;

combining the at least two images as the target image.

5. The method according to claim 1, wherein the second feature extraction layer comprises N sub-feature extraction layers, the number of the target images is N, and N is a positive integer not less than 2;

the step of extracting interference features of each target image carrying interference information respectively through a second feature extraction layer of the image interference removal model to obtain corresponding interference features comprises the following steps:

and inputting each target image into one sub-feature extraction layer, and extracting interference features of the target images through the sub-feature extraction layers to obtain corresponding interference features.

6. The method according to claim 1, wherein the interference features include size features and channel features, and the fusing the interference features of each target image to obtain fused interference features includes:

respectively carrying out size transformation on the interference features of each target image to obtain target interference features corresponding to each interference feature; wherein the size characteristics of each target interference characteristic are the same;

performing fusion processing on the channel characteristics of the target interference characteristics to obtain fusion channel characteristics;

and performing characteristic splicing on the size characteristic of the target interference characteristic and the fusion channel characteristic to obtain the fusion interference characteristic.

7. The method according to claim 1, wherein when the number of the target images is m, and correspondingly, the number of the interference features is m, and performing fusion processing on the interference features of each target image to obtain a fused interference feature comprises:

fusing the jth interference feature and the (j + 1) th interference feature in the m interference features to obtain a jth fused interference feature; wherein m is a positive integer not less than 3, j is a positive integer, and j belongs to [1, m-1 ];

fusing the jth fusion interference feature with the (j + 2) th interference feature to obtain a (j + 1) th fusion interference feature;

and starting to take the value of j as 1, traversing j, and taking the j +1 th fusion interference feature as the fusion interference feature when the value of j +2 is the same as the value of m.

8. The method according to claim 1, wherein the performing, by an output layer of the image interference elimination model, interference elimination processing on the image to be processed based on the fusion interference feature and the image feature to obtain a corresponding interference elimination image comprises:

determining a difference feature between the image feature and the fusion interference feature through an output layer of the image interference removal model;

and decoding the difference characteristic through an output layer of the image interference removal model to obtain the interference removal image.

9. The method according to claim 1, wherein before performing feature extraction on the image to be processed by the first feature extraction layer of the image interference elimination model, the method further comprises:

carrying out image decomposition on the sample interference image carrying the interference information to obtain at least two sample target images;

the sample interference image is obtained by adding interference information to the original sample image;

performing feature extraction on the sample interference image through a first feature extraction layer of the image interference removal model to obtain sample image features of the sample interference image;

updating model parameters of the image de-interference model based on a difference between the predicted de-interference image and the sample original image.

10. The method of claim 9, wherein the sample original image corresponds to at least two sample interference images, each of the sample interference images is obtained by adding interference information to the sample original image, and the method further comprises:

respectively executing the operations from image decomposition to interference removal processing of the sample interference image carrying interference information to the prediction interference removal image corresponding to each sample interference image aiming at each sample interference image;

correspondingly, the updating the model parameters of the image interference elimination model based on the difference between the predicted interference elimination image and the sample original image comprises:

and updating the model parameters of the image interference elimination model based on the difference between each predicted interference elimination image and the sample original image.

11. The method of claim 9, wherein the interference information comprises at least one of image aliasing and image noise;

before the image decomposition of the sample interference image carrying the interference information, the method further comprises:

adding image saw teeth to the sample original image to obtain a sample saw tooth image carrying the image saw teeth, and taking the sample saw tooth image as the sample interference image; or

Adding image noise to the sample original image to obtain a sample noise image carrying the image noise, and taking the sample noise image as the sample interference image; or

Adding image saw-teeth and image noise to the sample original image to obtain a sample saw-tooth noise image simultaneously carrying the image saw-teeth and the image noise, and taking the sample saw-tooth noise image as the sample interference image.

12. A training method of an image interference elimination model is characterized by comprising the following steps:

13. An image de-interference apparatus, comprising:

14. An electronic device, comprising:

a memory for storing executable instructions;

a processor for implementing the method of any one of claims 1 to 12 when executing executable instructions stored in the memory.

15. A computer-readable storage medium having stored thereon executable instructions for, when executed by a processor, implementing the method of any one of claims 1 to 12.