WO2021179764A1 - 图像处理模型生成方法、处理方法、存储介质及终端 - Google Patents

图像处理模型生成方法、处理方法、存储介质及终端 Download PDF

Info

Publication number
WO2021179764A1
WO2021179764A1 PCT/CN2020/141932 CN2020141932W WO2021179764A1 WO 2021179764 A1 WO2021179764 A1 WO 2021179764A1 CN 2020141932 W CN2020141932 W CN 2020141932W WO 2021179764 A1 WO2021179764 A1 WO 2021179764A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
processed
training
pixel
model
Prior art date
Application number
PCT/CN2020/141932
Other languages
English (en)
French (fr)
Inventor
李松南
张瑜
俞大海
Original Assignee
Tcl科技集团股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from CN202010163472.4A external-priority patent/CN113379611B/zh
Priority claimed from CN202010162684.0A external-priority patent/CN113379608A/zh
Priority claimed from CN202010162709.7A external-priority patent/CN113379610B/zh
Application filed by Tcl科技集团股份有限公司 filed Critical Tcl科技集团股份有限公司
Publication of WO2021179764A1 publication Critical patent/WO2021179764A1/zh
Priority to US17/865,340 priority Critical patent/US20220398698A1/en

Links

Images

Classifications

    • G06T5/77—
    • G06T5/73—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
    • G06T5/60—
    • G06T5/70—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/14—Picture signal circuitry for video frequency region
    • H04N5/21—Circuitry for suppressing or minimising disturbance, e.g. moiré or halo
    • H04N5/211—Ghost signal cancellation

Definitions

  • the present disclosure relates to the field of image processing technology, and in particular to an image processing model generation method, processing method, storage medium and terminal.
  • Existing full-screen terminals generally include a display panel area and a camera area.
  • the camera area is located on the top of the display panel area. Although the screen-to-body ratio can be increased, the camera area still occupies part of the display area and cannot truly achieve the full screen. Therefore, in order to realize a full-screen terminal, an imaging system needs to be installed under the display panel.
  • Existing display panels generally include substrates and polarizers. When light passes through the display panel, the display panel refracts the light on the one hand to make the light transmittance low, and on the other hand, One side of the display panel will absorb light, which will affect the quality of the captured image, for example, the color of the captured image does not match the shooting scene and the increase in image noise.
  • the technical problem to be solved by the present disclosure is to provide an image processing model generation method, processing method, storage medium and terminal in view of the deficiencies of the prior art.
  • the first aspect of the present disclosure provides a method for generating an image processing model, wherein the method for generating the image processing model specifically includes:
  • the preset network model generates a generated image corresponding to the first image according to the first image in the training image set, wherein the training image set includes multiple training image groups, and each training image group includes the first image and the first image. Two images, the first image is a color cast image corresponding to the second image;
  • the preset network model corrects the model parameters according to the second image corresponding to the first image and the generated image corresponding to the first image, and continues to execute according to the next training image group in the training image set.
  • the step of generating an image corresponding to the first image is generated until the training condition of the preset network model meets the preset condition, so as to obtain the image processing model.
  • a second aspect of the present disclosure provides an image processing method, wherein the image processing model generated by the method for generating the image processing model described above is applied, and the image processing method includes:
  • a third aspect of the present disclosure provides a method for generating an image processing model, wherein the method for generating the image processing model specifically includes:
  • the preset network model generates the generated image corresponding to the first image according to the first image in the training image set; wherein, the training image set includes multiple training image groups, and each training image group includes the first image and the second image.
  • Image the first image is an image with ghosting corresponding to the second image;
  • the preset network model corrects the model parameters of the preset network model according to the second image corresponding to the first image and the generated image corresponding to the first image, and continues to execute the download according to the training image set
  • a first image in a training image group is generated in the step of generating an image corresponding to the first image until the training condition of the preset network model meets a preset condition, so as to obtain a trained image processing model.
  • a fourth aspect of the present disclosure provides an image processing method, wherein the image processing model generated by applying the image processing model generation method as described above, the image processing method includes:
  • a fifth aspect of the present disclosure provides an image processing method, wherein the method includes:
  • the processed image is input to a trained second image processing model, and the processed image is de-ghosted by the second image processing model to obtain an output image.
  • one of the multiple images included in the image set to be processed is a base image, and the remaining images are adjacent images of the base image, and the image to be processed is generated according to the set of images to be processed
  • the denoising images corresponding to the set specifically include:
  • the weight parameter set corresponding to the basic image block includes a first weight parameter and a second weight parameter
  • the first weight parameter is the weight parameter of the basic image block
  • the second weight parameter is The weight parameter of the adjacent image block corresponding to the basic image block in the adjacent image
  • the denoising image is determined according to the image set to be processed and the weight parameter set corresponding to each basic image block.
  • the number of images in the image set to be processed is determined according to shooting parameters corresponding to the image set to be processed.
  • the image definition of the base image is greater than or equal to the image definition of the adjacent image.
  • the determining the weight parameter set corresponding to each basic image block specifically includes:
  • For each basic image block determine the second weight parameter of each adjacent image block corresponding to the basic image block, and obtain the first weight parameter corresponding to the basic image block to obtain the weight parameter set corresponding to the basic image block.
  • the determining the second weight parameter of each adjacent image block corresponding to the basic image block specifically includes:
  • the calculating the second weight parameter of the adjacent image block according to the similarity value specifically includes:
  • the preset second preset parameter is used as the second weight parameter of the adjacent image block.
  • the first threshold and the second threshold are both determined according to the similarity values of adjacent image blocks corresponding to the basic image block.
  • the method further includes:
  • the first image processing model is obtained by training based on a first training image set.
  • the first training image set includes a plurality of training image groups, and each training image group includes a first image and a second image.
  • Image the first image is a color cast image corresponding to the second image.
  • the first image is an image captured by an off-screen imaging system.
  • the method before training the first image processing model based on the first training image set, the method includes:
  • the first image in the training image group and the second image corresponding to the first image are aligned to obtain an alignment aligned with the second image Image, and use the aligned image as the first image.
  • the first training image set includes several training sub-image sets, each training sub-image set includes several training sample image groups, and any two training sample image groups in the several training image groups
  • the exposure of the first image is the same, the exposure of the second image in each training sample image group in the several training image groups is within the preset range, and the exposure of the first image in any two training sub-image sets The degrees are not the same.
  • the first image processing model corresponds to a number of model parameters
  • each model parameter is obtained by training according to a training sub-image set in the first training image set
  • any two model parameters respectively correspond to The training sub-image sets of are different from each other.
  • the inputting the denoised image into the trained first image processing model specifically includes:
  • the denoised image is input to the updated first image processing model.
  • the extracting the exposure of the denoised image specifically includes:
  • a third pixel that meets a preset condition is determined, where the preset condition is at least one of the R value, the G value, and the B value Greater than the preset threshold;
  • the highlight area of the denoised image is determined according to all the third pixel points that meet the preset condition, and the exposure of the denoised image is determined according to the highlight area.
  • the determining the highlight area of the denoising image according to all third pixel points that meet a preset condition specifically includes:
  • the R value, G value and B value of the three pixels are of the same R value, G value and/or B value which is greater than the preset threshold;
  • the determining the exposure of the denoised image according to the highlight area specifically includes:
  • the exposure degree corresponding to the denoising image is determined according to the ratio of the first area and the second area.
  • the first image processing model includes a down-sampling module and a transformation module; the denoising image is de-pigmented by the first image processing model to obtain the denoised image
  • the corresponding processed image specifically includes:
  • the denoised image is input to the downsampling module, and the bilateral grid corresponding to the denoised image and the guide image corresponding to the denoised image are obtained through the downsampling module, wherein the resolution of the guide image The same as the resolution of the denoising image;
  • the guide image, the bilateral grid, and the denoising image are input to the transformation module, and the processed image corresponding to the denoising image is generated by the transformation module.
  • the down-sampling module includes a down-sampling unit and a convolution unit; the denoising image is input to the down-sampling module, and the two sides corresponding to the denoised image are obtained through the down-sampling module.
  • the grid and the guide image corresponding to the denoising image specifically include:
  • the two-sided grid corresponding to the denoising image is obtained by the down-sampling unit, and the guide image corresponding to the denoising image is obtained by the convolution unit.
  • the transformation module includes a segmentation unit and a transformation unit.
  • the guidance image, the bilateral grid, and the denoising image are input to the transformation module, and the transformation module is used to generate the denoising image.
  • the processed image corresponding to the noisy image specifically includes:
  • the denoising image and the color transformation matrix of each pixel in the denoising image are input to the transformation unit, and the processed image corresponding to the denoising image is generated by the transformation unit.
  • the second image processing model is obtained by training based on a second training image set.
  • the second training image set includes a plurality of training image groups, and each training image group includes a third image and a fourth image.
  • the third image is an image with ghosting corresponding to the fourth image.
  • the third image is generated based on the fourth image and a point spread function, wherein the point spread function is generated based on a grayscale image generated by a light shielding structure in an under-screen imaging system.
  • the third image is an image captured by an off-screen imaging system.
  • the under-screen imaging system is an under-screen camera.
  • the method before training the second image processing model based on the second training image set, the method further includes:
  • the third image in the training image group and the fourth image corresponding to the third image are aligned to obtain an alignment aligned with the fourth image Image, and use the aligned image as the third image.
  • the second image processing model includes an encoder and a decoder; the second image processing model is used to de-ghost the processed image to obtain the corresponding image of the processed image.
  • the output image specifically includes:
  • the characteristic image is input to the decoder, and an output image corresponding to the processed image is output through the decoder, wherein the image size of the output image is equal to the image size of the processed image.
  • the process of the alignment processing specifically includes:
  • the reference image and the reference image in the training image group and calculate the pixel deviation between the reference image and the reference image; wherein, when the reference image is the second image, the reference image is the first image; when the reference image is the second image, the reference image is the first image; When the image is the fourth image, the reference image is the third image;
  • the alignment method corresponding to the reference image is determined according to the pixel deviation amount, and the alignment method is used to align the reference image with the reference image.
  • the determining the alignment method corresponding to the reference image according to the pixel deviation amount, and using the alignment method to align the reference image with the reference image specifically includes:
  • the reference pixel point set of the reference image and the reference pixel point set of the reference image are extracted, and the reference pixel point set includes the reference pixel point set in the reference image.
  • a number of reference pixels the reference pixel point set includes a number of reference pixels in the base image, the reference pixel points in the reference pixel point set correspond to the reference pixels in the reference pixel point set in a one-to-one correspondence;
  • the coordinate difference between the reference pixel point and its corresponding reference pixel point is calculated, and the position of the reference pixel point is adjusted according to the coordinate difference value corresponding to the reference pixel point, so as to The reference pixel point is aligned with the reference pixel point corresponding to the reference pixel point.
  • the processed image is input to a second image processing model that has been trained, and the processed image is de-ghosted through the second image processing model to obtain an output image It also includes:
  • a sixth aspect of the present disclosure provides a device for generating an image processing model, wherein the device for generating an image processing model includes:
  • the first generation module is used to generate a generated image corresponding to the first image according to the first image in the training image set by using a preset network model, wherein the training image set includes a plurality of training image groups, each of which is trained The image group includes a first image and a second image, and the first image is a color cast image corresponding to the second image;
  • the first correction module is configured to use the preset network model to correct the model parameters according to the second image corresponding to the first image and the generated image corresponding to the first image, and to continue to execute according to the training image set For the first image in the next training image group, the step of generating the image corresponding to the first image is generated until the training condition of the preset network model meets the preset condition, so as to obtain the image processing model.
  • the number of first target pixels in the first image that meets the preset color cast condition meets the preset number condition;
  • the preset color cast condition is the value of the first target pixel in the first image
  • the error between the display parameter and the display parameter of the second target pixel in the second image satisfies a preset error condition, wherein there is a one-to-one correspondence between the first target pixel and the second target pixel.
  • the first target pixel is any pixel in the first image or any pixel in the target area of the first image.
  • the training image set includes several training sub-image sets, each training sub-image set includes several training sample image groups, and any two training sample image groups in the training image groups
  • the exposure of an image is the same, the exposure of the second image in each training sample image group in the several training image groups is within the preset range, and the exposure of the first image in any two training sub-image sets is different. same.
  • the image processing model corresponds to a number of model parameters, and each model parameter is obtained by training according to a training sub-image set in the training image set, and any two model parameters respectively correspond to the training sub-images.
  • the image sets are different from each other.
  • the preset network model includes a down-sampling module and a transformation module; the first generation module is specifically configured to:
  • Input the first image in the training image set to the down-sampling module obtain the bilateral grid corresponding to the first image and the guidance image corresponding to the first image through the down-sampling module; and transfer the guidance
  • the image, the bilateral grid, and the first image are input to the transformation module, and a generated image corresponding to the first image is generated by the transformation module, wherein the resolution of the guide image is the same as the resolution of the first image
  • the rate is the same.
  • the down-sampling module includes a down-sampling unit and a convolution unit; the first generation module is specifically configured to:
  • the transformation module includes a segmentation unit and a transformation unit, and the first generation module is specifically configured to:
  • the first image and the color transformation matrix of each pixel in the first image are input to the transformation unit, and a generated image corresponding to the first image is generated by the transformation unit.
  • the first image is an image captured by an off-screen imaging system.
  • the under-screen imaging system is an under-screen camera.
  • the device for generating the image processing model further includes:
  • the first alignment module is used to align the first image in the training image group with the second image corresponding to the first image for each training image group in the training image set, to obtain the first image corresponding to the first image
  • An alignment image that is aligned with the two images, and the alignment image is used as the first image.
  • the first alignment module is specifically configured to:
  • the pixel deviation amount between the first image in the training image group and the second image corresponding to the first image is obtained; the pixel deviation amount is determined according to the pixel deviation amount.
  • the first alignment module is specifically configured to:
  • the first pixel point set of the first image and the second pixel point set of the second image are extracted, and the first pixel point set includes all A number of first pixels in the first image
  • the second pixel point set includes a number of second pixels in the second image, the second pixel point in the second pixel point set and the first pixel point
  • One-to-one correspondence between the first pixel in a pixel set for each first pixel in the first pixel set, the coordinate difference between the first pixel and its corresponding second pixel is calculated, and according to the first pixel
  • the coordinate difference value corresponding to the point performs position transformation on the first pixel to align the first pixel with the second pixel corresponding to the first pixel.
  • a seventh aspect of the present disclosure provides an image processing device, wherein the image processing model generated by the image processing model generation method as described in the first method or the image processing model generated by the image processing model generation device as described in the sixth aspect is applied, and
  • the image processing device includes:
  • the first acquisition module is configured to acquire an image to be processed and input the image to be processed into the image processing model
  • the first processing module is configured to perform color cast processing on the image to be processed through the image processing model to obtain a processed image corresponding to the image to be processed.
  • the image processing model corresponds to a number of model parameters, and each model parameter is obtained by training based on a training sub-image set, and the training sub-image sets corresponding to each of any two model parameters are different from each other.
  • the first acquisition module is specifically configured to:
  • the image to be processed is input to the updated image processing model.
  • the first acquisition module is specifically configured to:
  • a third pixel that meets a preset condition is determined, where the preset condition is at least one of R value, G value, and B value Greater than the preset threshold;
  • the highlight area of the image to be processed is determined according to all the third pixel points that meet the preset condition, and the exposure degree of the image to be processed is determined according to the highlight area.
  • the first acquisition module is specifically configured to:
  • the R value, G value and B value of the three pixels are of the same R value, G value and/or B value which is greater than the preset threshold;
  • the first acquisition module is specifically configured to:
  • the exposure degree corresponding to the image to be processed is determined according to the ratio of the first area and the second area.
  • the image processing model includes a down-sampling module and a transformation module; the first processing module is specifically configured to:
  • the image to be processed is input to the down-sampling module, and the bilateral grid corresponding to the image to be processed and the guide image corresponding to the image to be processed are obtained through the down-sampling module, wherein the resolution of the guide image
  • the resolution is the same as the resolution of the image to be processed; and the guide image, the bilateral grid, and the image to be processed are input to the transformation module, and the processed image corresponding to the first image is generated by the transformation module .
  • the down-sampling module includes a down-sampling unit and a convolution unit; the first processing module is specifically configured to:
  • the transformation module includes a segmentation unit and a transformation unit
  • the first processing module is specifically configured to:
  • the guide image is input to the segmentation unit, and the bilateral grid is segmented by the segmentation unit to obtain the color transformation matrix of each pixel in the image to be processed; and
  • the image and the color transformation matrix of each pixel in the image to be processed are input to the transformation unit, and the processed image corresponding to the image to be processed is generated by the transformation unit.
  • the image processing device further includes:
  • the noise reduction processing unit is configured to perform sharpening and noise reduction processing on the processed image, and use the sharpened and noise reduction processed image as a processed image corresponding to the to-be-processed image.
  • An eighth aspect of the present disclosure provides a device for generating an image processing model, wherein the device for generating an image processing model includes:
  • the second generation module is configured to use a preset network model to generate a generated image corresponding to the first image according to the first image in the training image set; wherein, the training image set includes multiple sets of training image groups, each of which The group includes a first image and a second image, and the first image is an image with ghosting corresponding to the second image;
  • the second correction module is configured to use the preset network model to correct the model parameters of the preset network model according to the second image corresponding to the first image and the generated image corresponding to the first image, and Continue to execute the step of generating the generated image corresponding to the first image according to the first image in the next training image group in the training image set, until the training condition of the preset network model meets the preset condition to obtain the trained Image processing model.
  • the preset network model includes an encoder and a decoder; the second generation module is specifically configured to:
  • the first image in the training image set is input to the encoder, and the characteristic image of the first image is obtained through the encoder; and the characteristic image is input to the decoder, and the decoder outputs the An image is generated, wherein the image size of the characteristic image is smaller than the image size of the first image; the image size of the generated image is equal to the image size of the first image.
  • the second correction module is specifically configured to:
  • the first image is generated based on the second image and a point spread function, wherein the point spread function is generated based on a grayscale image generated by a light shielding structure in an under-screen imaging system.
  • the first image is an image captured by an off-screen imaging system.
  • the under-screen imaging system is an under-screen camera.
  • the device for generating the image processing model further includes:
  • the second alignment module is configured to align the first image in the training image group with the second image corresponding to the first image for each training image group in the training image set, to obtain the first image corresponding to the first image in the training image group.
  • An alignment image that is aligned with the two images, and the alignment image is used as the first image.
  • the second alignment module is specifically configured to:
  • the pixel deviation amount between the first image in the training image group and the second image corresponding to the first image is obtained; the pixel deviation amount is determined according to the pixel deviation amount.
  • the second alignment module is specifically configured to:
  • the first pixel point set of the first image and the second pixel point set of the second image are extracted, and the first pixel point set includes all A number of first pixels in the first image
  • the second pixel point set includes a number of second pixels in the second image, the second pixel point in the second pixel point set and the first pixel point
  • One-to-one correspondence between the first pixel in the pixel set for each first pixel in the first pixel set, the coordinate difference between the first pixel and its corresponding second pixel is calculated, and according to the first pixel
  • the coordinate difference value corresponding to the pixel point adjusts the position of the first pixel point to align the first pixel point with the second pixel point corresponding to the first pixel point.
  • a ninth aspect of the present disclosure provides an image processing device that uses the image processing model generation method described in the third aspect or the image processing model generated by the image processing model generation device described in the eighth aspect, and
  • the image processing device includes:
  • the second acquisition module is used to acquire the image to be processed and input the image to be processed into the image processing model
  • the second processing module is configured to perform de-ghosting processing on the image to be processed through the image processing model to obtain an output image corresponding to the image to be processed.
  • the image processing model includes an encoder and a decoder; the second processing module specifically includes:
  • Input the image to be processed into the encoder obtain a characteristic image of the image to be processed through the encoder; and input the characteristic image into the decoder, and output the image to be processed through the decoder
  • the corresponding output image wherein the image size of the characteristic image is smaller than the image size of the image to be processed; the image size of the output image is equal to the image size of the image to be processed.
  • the image processing device further includes:
  • the sharpening module is used to perform sharpening and noise reduction processing on the output image, and use the output image after the sharpening and noise reduction processing as the output image corresponding to the image to be processed.
  • a tenth aspect of the present disclosure provides an image processing device, wherein the image processing device includes:
  • the third acquisition module is configured to acquire a set of images to be processed, wherein the set of images to be processed includes multiple images;
  • a third generation module configured to generate a denoising image corresponding to the to-be-processed image set according to the to-be-processed image set;
  • the third processing module is used to input the denoised image to the trained first image processing model, and perform the de-pigmentation processing on the denoised image through the image processing model to obtain the corresponding denoised image The processed image;
  • the fourth processing module is used to input the processed image to a second trained image processing model, and perform de-ghosting processing on the processed image through the second image processing model to obtain an output image.
  • one of the multiple images included in the image set to be processed is a base image, and the remaining images are neighboring images of the base image, and the third generation module is specifically configured to:
  • the weight parameter set corresponding to the basic image block includes a first weight parameter and a second weight parameter
  • the first weight parameter is the weight parameter of the basic image block
  • the second weight parameter is The weight parameter of the adjacent image block corresponding to the basic image block in the adjacent image
  • the denoising image is determined according to the image set to be processed and the weight parameter set corresponding to each basic image block.
  • the number of images in the image set to be processed is determined according to shooting parameters corresponding to the image set to be processed.
  • the image definition of the base image is greater than or equal to the image definition of the adjacent image.
  • the third generation module is specifically configured to:
  • For each basic image block determine the second weight parameter of each adjacent image block corresponding to the basic image block, and obtain the first weight parameter corresponding to the basic image block to obtain the weight parameter set corresponding to the basic image block.
  • the third generation module is specifically configured to:
  • the third generation module is specifically configured to:
  • the preset second preset parameter is used as the second weight parameter of the adjacent image block.
  • the first threshold and the second threshold are both determined according to the similarity values of adjacent image blocks corresponding to the basic image block.
  • the image processing device further includes:
  • the spatial denoising module is used to perform spatial denoising on the denoised image, and use the image obtained after spatial denoising as the denoised image.
  • the first image processing model is obtained by training based on a first training image set.
  • the first training image set includes a plurality of training image groups, and each training image group includes a first image and a second image.
  • Image the first image is a color cast image corresponding to the second image.
  • the first image is an image captured by an off-screen imaging system.
  • the image processing device further includes:
  • the third alignment module is used for aligning the first image in the training image group with the second image corresponding to the first image for each training image group in the first training sample set to obtain the The second image is aligned with the alignment image, and the alignment image is used as the first image.
  • the first training image set includes several training sub-image sets, each training sub-image set includes several training sample image groups, and any two training sample image groups in the several training image groups
  • the exposure of the first image is the same, the exposure of the second image in each training sample image group in the several training image groups is within the preset range, and the exposure of the first image in any two training sub-image sets The degrees are not the same.
  • the first image processing model corresponds to a number of model parameters
  • each model parameter is obtained by training according to a training sub-image set in the first training image set
  • any two model parameters respectively correspond to The training sub-image sets of are different from each other.
  • the third processing module is specifically configured to:
  • the third processing module is specifically configured to:
  • a third pixel that meets a preset condition is determined, where the preset condition is at least one of the R value, the G value, and the B value Greater than a preset threshold; determining the highlight area of the denoised image according to all third pixel points that meet the preset condition, and determining the exposure of the denoised image according to the highlight area.
  • the third processing module is specifically configured to:
  • the connected areas formed by all the third pixel points that meet the preset conditions select target areas that meet the preset rules from all the obtained connected areas, calculate the respective areas corresponding to each target area obtained by screening, and The target area with the largest area is selected as the highlight area, where the preset rule is that the R value, G value and/or B value of the third pixel in the target area are greater than the preset threshold.
  • the values are of the same type.
  • the third processing module is specifically configured to:
  • the first image processing model includes a down-sampling module and a transformation module; the third processing module is specifically configured to:
  • the denoised image is input to the downsampling module, and the bilateral grid corresponding to the denoised image and the guide image corresponding to the denoised image are obtained through the downsampling module, wherein the resolution of the guide image The same as the resolution of the denoising image;
  • the guide image, the bilateral grid, and the denoising image are input to the transformation module, and the processed image corresponding to the denoising image is generated by the transformation module.
  • the down-sampling module includes a down-sampling unit and a convolution unit; the third processing module is specifically configured to:
  • the two-sided grid corresponding to the denoising image is obtained by the down-sampling unit, and the guide image corresponding to the denoising image is obtained by the convolution unit.
  • the transformation module includes a segmentation unit and a transformation unit
  • the third processing module is specifically configured to:
  • the denoising image and the color transformation matrix of each pixel in the denoising image are input to the transformation unit, and the processed image corresponding to the denoising image is generated by the transformation unit.
  • the second image processing model is obtained by training based on a second training image set.
  • the second training image set includes a plurality of training image groups, and each training image group includes a third image and a fourth image.
  • the third image is an image with ghosting corresponding to the fourth image.
  • the third image is generated based on the fourth image and a point spread function, wherein the point spread function is generated based on a grayscale image generated by a light shielding structure in an under-screen imaging system.
  • the third image is an image captured by an off-screen imaging system.
  • the under-screen imaging system is an under-screen camera.
  • the image processing device further includes:
  • the fourth alignment module is used for aligning the third image in the training image group with the fourth image corresponding to the third image for each training image group in the second training image set to obtain the The fourth image is aligned with the alignment image, and the alignment image is used as the third image.
  • the second image processing model includes an encoder and a decoder; the fourth processing module is specifically configured to:
  • the processed image is input to the encoder, and the characteristic image of the processed image is obtained through the encoder; and the characteristic image is input to the decoder, and the processed image is output through the decoder
  • the corresponding output image wherein the image size of the characteristic image is smaller than the image size of the processed image; the image size of the output image is equal to the image size of the processed image.
  • both the third alignment module and/or the fourth alignment module are specifically configured to:
  • the alignment method aligns the reference image with the reference image, wherein when the reference image is the second image, the reference image is the first image; when the reference image is the fourth image, the reference image is the third image .
  • both the third alignment module and/or the fourth alignment module are specifically configured to:
  • the reference pixel point set of the reference image and the reference pixel point set of the reference image are extracted, and the reference pixel point set includes the reference pixel point set in the reference image.
  • a number of reference pixels the reference pixel point set includes a number of reference pixels in the base image, the reference pixel points in the reference pixel point set correspond to the reference pixels in the reference pixel point set in a one-to-one correspondence;
  • the coordinate difference between the reference pixel point and its corresponding reference pixel point is calculated, and the position of the reference pixel point is adjusted according to the coordinate difference value corresponding to the reference pixel point.
  • the reference pixel point is aligned with the reference pixel point corresponding to the reference pixel point.
  • the image processing device further includes:
  • the sharpening and noise reduction module is used to perform sharpening and noise reduction processing on the processed image, and use the processed image after the sharpening and noise reduction processing as the output image.
  • An eleventh aspect of the present disclosure provides a computer-readable storage medium that stores one or more programs, and the one or more programs can be executed by one or more processors to implement The method for generating an image processing model and/or a step in the image processing method as described above.
  • a twelfth aspect of the present disclosure provides a terminal device, which includes: a processor, a memory, and a communication bus; the memory stores a computer readable program that can be executed by the processor;
  • the communication bus realizes connection and communication between the processor and the memory
  • the present disclosure provides an image processing model generation method, processing method, storage medium, and terminal.
  • the generation method integrates preset training images into a preset network.
  • the model is used to sequence the preset model through the generated image generated by the preset network model and the second image corresponding to the first image to obtain an image processing model.
  • the image processing model is obtained by in-depth learning of the color cast process of the training image set with multiple training image groups.
  • Each training image group includes a first image and a second image.
  • the first image corresponds to the second image.
  • the color cast of the image It can be seen from this that the present disclosure uses a trained image processing model obtained by deep learning based on a training image set to perform color cast processing, which can quickly adjust the color cast of the image, improve the color quality of the image, and thereby improve the image quality.
  • FIG. 1 is a schematic diagram of an application scenario of a method for generating an image processing model provided by the public.
  • Fig. 2 is a flow chart of a publicly provided method for generating an image processing model.
  • FIG. 3 is a schematic diagram of a preset network model of a method for generating an image processing model provided by the public.
  • FIG. 4 is a schematic flowchart of a method for generating an image processing model provided by the public.
  • FIG. 5 is an example diagram of the first image provided by the public.
  • FIG. 6 is an example diagram of the second image provided by the public.
  • FIG. 7 is a flowchart of the process of determining the alignment provided by the public.
  • FIG. 8 is a flowchart of step S10 in a method for generating an image processing model provided by the public.
  • FIG. 9 is a flowchart of step S11 in a method for generating an image processing model provided by the public.
  • FIG. 10 is a flowchart of step S12 in a method for generating an image processing model provided by the public.
  • Fig. 11 is a flowchart of an image processing method provided by the public.
  • FIG. 12 is a flowchart of step A100 in an image processing method provided by the public.
  • Fig. 13 is an example diagram of an image to be processed provided by the public.
  • FIG. 14 is an example diagram of a processed image corresponding to an image to be processed provided by the public.
  • FIG. 15 is a flowchart of a method for generating an image processing model provided by the public.
  • FIG. 16 is a schematic flowchart of a method for generating an image processing model provided by the public.
  • Fig. 17 is a diagram showing an example of the first image publicly provided.
  • FIG. 18 is an example diagram of the second image provided by the public.
  • FIG. 19 is a diagram showing an example of a signal circuit of a method for generating an image processing model provided by the public.
  • FIG. 20 is an example diagram of a grayscale image of a method for generating an image processing model provided by the public.
  • FIG. 21 is a flowchart of step N10 in a method for generating an image processing model provided by the public.
  • FIG. 22 is a schematic structural diagram of a preset network model in a method for generating an image processing model provided by the public.
  • FIG. 23 is a flowchart of step N20 in a method for generating an image processing model provided by the public.
  • FIG. 24 is a flowchart of an image processing method provided by the public.
  • FIG. 25 is an example diagram of an image to be processed provided by the public.
  • Fig. 26 is an example diagram of an output image corresponding to an image to be processed provided publicly.
  • FIG. 27 is a flowchart of an embodiment of an image processing method provided by the present disclosure.
  • FIG. 28 is a flowchart of step H20 in an embodiment of the image processing method provided by the present disclosure.
  • FIG. 29 is a flowchart of an acquisition process of adjacent image blocks in an embodiment of the image processing method provided by the present disclosure.
  • FIG. 30 is an example diagram of a designated area in an embodiment of the image processing method provided by the present disclosure.
  • FIG. 31 is a flowchart of the calculation process of the second weight parameter in an embodiment of the image processing method provided by the present disclosure.
  • Fig. 32 is a flowchart of the training process of the second image processing model in an embodiment of the image processing method provided by the present disclosure.
  • FIG. 33 is a flowchart of step L200 in an embodiment of the image processing method provided by the present disclosure.
  • FIG. 34 is a schematic structural diagram of an embodiment of an apparatus for generating an image processing model provided by the present disclosure.
  • FIG. 35 is a schematic structural diagram of an embodiment of an image processing apparatus provided by the present disclosure.
  • FIG. 36 is a schematic structural diagram of an embodiment of an image processing model generation apparatus provided by the present disclosure.
  • FIG. 37 is a schematic structural diagram of an embodiment of an image processing apparatus provided by the present disclosure.
  • FIG. 38 is a schematic structural diagram of an embodiment of an image processing apparatus provided by the present disclosure.
  • FIG. 39 is a schematic structural diagram of a terminal device provided by the present disclosure.
  • the present disclosure provides an image processing model generation method, processing method, storage medium, and terminal.
  • image processing model generation method processing method, storage medium, and terminal.
  • Existing display panels generally include substrates and polarizers. When light passes through the display panel, the display panel refracts the light to make the light transmittance low, and the other display panel absorbs the light, which will affect the quality of the image captured. For example, the color of the captured image does not match the shooting scene, the increase in image noise, and the blur of the image.
  • the second image is used as the target image
  • the color cast image (referring to the first image) of the second image is used as the training sample image
  • the first image is input to the preset network Model, output the generated image corresponding to the first image through the preset network model, and then train the preset network model according to the second image corresponding to the first image and the generated image corresponding to the first image to obtain the trained Image processing model.
  • the image processing model is obtained by deep learning the preset network model, so that the image processing model obtained by training can remove the color cast in the image, and the image processing model obtained by the training can be used to image the image under the screen.
  • the image captured by the system is processed to remove the color cast carried by the image and improve the image quality of the image captured by the imaging system under the screen.
  • the embodiment of the present invention can be applied to the scenario shown in FIG. 1.
  • the terminal device 1 may collect a training image set, and input the training image set into the server 2, so that the server 2 trains a preset network model according to the training image set.
  • the server 2 may pre-store a preset network model, and respond to the input training image set of the terminal device 1, input the first image in the training image set as an input item into the preset network model, and then obtain the preset network
  • the generated image output by the model is used to correct the preset network model through the second image corresponding to the first image and the generated image corresponding to the first image, and continue to perform the input of the first image in the training image collection into the preset network model And continue to perform the step of generating an image corresponding to the first image according to the first image in the training image set, until the training condition of the preset network model meets the preset conditions, so as to obtain the image processing Model.
  • the trained image processing model can be used to process photos taken by a terminal device having an under-screen imaging system (for example, an under-screen camera).
  • a photo taken by a terminal device with an under-screen imaging system for example, an under-screen camera
  • the photo is processed through the trained image processing model.
  • the photos can be quickly processed to remove color cast, so as to improve the image quality of the photos taken by the under-screen camera.
  • the trained image processing model can be used as a color-removing functional module to be configured in a terminal device with an under-screen imaging system (e.g., under-screen camera), when it has an under-screen imaging system (e.g., When the terminal device of the under-screen camera captures a photo, it activates the color-removing function module, and performs color-removal processing on the photo through the color-removing function module, so that it has an off-screen imaging system (for example, an off-screen imaging system).
  • the terminal device of the camera outputs the photo after the color cast is removed, so that the terminal device with an under-screen imaging system (for example, an under-screen camera) can directly output the image that has undergone the color cast processing.
  • This embodiment provides a method for generating an image processing model. As shown in FIGS. 2 and 4, the method includes:
  • the preset network model generates a generated image corresponding to the first image according to the first image in the training image set.
  • the preset network model is a deep learning network model, and the preset network model is trained based on a preset training image set.
  • the training image set includes a plurality of training image groups with different image content, each training image group includes a first image and a second image, and the first image is a color cast image corresponding to the second image.
  • that the first image is a color cast image corresponding to the second image means that the first image corresponds to the second image, the first image and the second image present the same image scene, and the first image meets the requirements It is assumed that the number of the first target pixel of the color cast condition satisfies the preset number condition.
  • the second image is a normal display image, and there are a number of first target pixels that meet a preset color cast condition in the first image, and the number of the first target pixels meets the preset condition.
  • the second image is the image shown in FIG. 6, and the first image is the image shown in FIG. 5.
  • the image content of the first image is the same as the image content of the second image, but the apple The corresponding color is different from the color of the apple in the second image.
  • the color of the apple in the first image is green to blue; in Figure 6, the second image The color of the middle apple in the second image is dark green.
  • the preset color cast condition is that the error between the display parameter of the first target pixel in the first image and the display parameter of the second target pixel in the second image satisfies a preset error condition, and the first target There is a one-to-one correspondence between the pixel points and the second target pixel point.
  • the display parameter is a parameter for reflecting the color corresponding to the pixel.
  • the display parameter may be the RGB value of the pixel, where R is the red channel value, G is the green channel value, and the B value is It is the blue channel value; it can also be the hsl value of the pixel, where h is the hue value, l is the brightness value, and s is the saturation value.
  • the display parameter is the RGB value of the pixel
  • the display parameter of any pixel in the first image and the second image includes three display parameters of R value, G value and B value;
  • the display is displayed as a pixel hls value
  • the display parameters of any pixel in the first image and the second image include three display parameters of h value, l value and s value.
  • the preset error condition is used to measure whether the first target pixel is a pixel that meets a preset color cast condition, wherein the preset error condition is a preset error threshold, and an error that meets the preset error condition is an error greater than or Equal to the preset error threshold.
  • the display parameter includes several display parameters, for example, the display parameter is the RGB value of the pixel.
  • the display parameter includes three display parameters of R value, G value and B value. When the display parameter is the hsl value of the pixel, the display parameter Including three display parameters of h value, l value and s value.
  • the error may be the maximum value of the error of each display parameter in the display parameter, or the minimum value of the error of each display parameter in the display parameter, or the average value of the errors of all the display parameters.
  • the display parameter is the RGB value of the pixel.
  • the display parameter of the first target pixel is (55, 86, 108), and the display parameter of the second target pixel is (58, 95, 120).
  • the display parameters of each display parameter The error value is divided into 3, 9 and 12; therefore, when the error between the first target pixel and the second target pixel is the maximum error of each display parameter, the error is 12; when the first target pixel and the second target pixel When the error of the second target pixel is the minimum error of each display parameter, the error is 3; when the error of the first target pixel and the second target pixel is the average of the errors of all the display parameters, the error is 8; It should be noted that in a possible implementation manner, it is also possible to refer to only one parameter (such as R, G, or B) in RGB or the error of any two parameters. When the display parameter is the hsl value of the pixel, the same reason.
  • the second target pixel used to calculate the error with the first target pixel and the first target display point there is a one-to-one correspondence between the second target pixel used to calculate the error with the first target pixel and the first target display point. It is understandable that for the first target pixel, there is a unique second target pixel in the second image corresponding to the first target pixel, where the first target pixel and the second target pixel correspond to the first target pixel.
  • the pixel position of a target pixel in the first image corresponds to the pixel position of the second target pixel in the second image. For example, the pixel position of the first target pixel in the first image is (5, 6), and the pixel position of the second target pixel in the second image is (5, 6).
  • the first target pixel point may be any pixel point in the first image, or any pixel point in the target area in the first image, where the target area may be the item in the first image.
  • Area, where the area where the object is located may be a corresponding area of the person or object in the image.
  • the target area is the area where the apple is located in the first image. That is to say, all pixels in the first image can be compared with the second image to have a color cast, that is, all pixels in the first image are the first target pixels, or only a part of the pixels can be compared with the second image Color cast appears, that is, some of the pixels in the first image are the first target pixels.
  • the image can also be understood as a color cast image corresponding to the second image, that is, the first image.
  • first image and the second image correspond to each other means that the image size of the first image is equal to the image size of the second image, and the first image and the second image correspond to the same image scene.
  • the first image and the second image corresponding to the same image scene means that the similarity between the image content carried by the first image and the image content carried by the second image reaches a preset threshold, and the image size of the first image is equal to that of the first image.
  • the image sizes of the two images are the same, so that when the first image and the second image overlap, the coverage rate of the object carried by the first image to the corresponding object in the second image reaches the preset condition.
  • the preset threshold may be 99%
  • the preset condition may be 99.5% and so on.
  • the first image may be captured by an off-screen imaging system; the second image may be captured by a normal on-screen imaging system (e.g., on-screen camera), or through a network (e.g., , Baidu), it can also be sent through other external devices (such as smart phones).
  • a normal on-screen imaging system e.g., on-screen camera
  • a network e.g., , Baidu
  • the second image is obtained by shooting through a normal on-screen imaging system, and shooting parameters of the second image and the first image are the same.
  • the shooting parameters may include exposure parameters of the imaging system, and the exposure parameters may include aperture, shutter speed, sensitivity, focus, white balance, and the like.
  • the shooting parameters may also include ambient light, shooting angle, and shooting range.
  • the first image is an image obtained by shooting a scene through an on-screen camera as shown in FIG. 5
  • the second image is an image obtained by shooting the scene through an on-screen camera as shown in FIG.
  • the image content of the first image and the image content of the second image may be completely same. That is, the first image and the second image having the same image content means that the object content of the first image is the same as the object content of the second image, and the image size of the first image is the same as the image size of the second image. And when the first image and the second image overlap, the objects in the first image can cover the corresponding objects in the second image.
  • the image size of the first image is 400*400
  • the image content of the first image is a circle
  • the position of the center of the circle in the first image in the first image is (200, 200), the radius length It is 50 pixels.
  • the image size of the second image is 400*400
  • the image content of the second image is also a circle
  • the position of the center of the circle in the second image in the second image is (200, 200), and the radius is 50 Pixels
  • the on-screen imaging system may be caused when the imaging system is replaced.
  • the change in the shooting angle and/or shooting position of the imaging system under the screen causes the first image and the second image to be spatially misaligned. Therefore, in a possible implementation of this embodiment, when the second image is captured by the on-screen imaging system and the first image is captured by the off-screen imaging system, the on-screen imaging system and the off-screen imaging system can be set to the same On the fixed frame, the on-screen imaging system and the under-screen imaging system are arranged side by side on the fixed frame, and the on-screen imaging system and the under-screen imaging system are kept in contact.
  • the on-screen imaging system and the under-screen imaging system connect the on-screen imaging system and the under-screen imaging system to wireless settings (such as Bluetooth watches, etc.), and trigger the shutter of the on-screen imaging system and the under-screen imaging system through the wireless settings, which can reduce the on-screen imaging during shooting.
  • the position change of the imaging system and the imaging system under the screen improves the spatial alignment of the first image and the second image.
  • the shooting time and shooting range of the on-screen imaging system and the under-screen imaging system are the same.
  • the shooting position, shooting angle, shooting time, and exposure coefficient of the off-screen imaging system and the on-screen imaging system can be fixed.
  • the first image captured by the off-screen imaging system and the second image captured by the on-screen imaging system may still be spatially misaligned. . Therefore, before the first image in the training image set is input into the preset network model, the first image and the second image in each training image group in the training image set can be aligned, so that in an implementation of this embodiment ,
  • the preset network model according to the first image in the training image set, before generating the generated image corresponding to the first image further includes
  • each training image group in the training image set refers to performing alignment processing on each training image group in the training image set.
  • Each training image group is aligned to obtain an aligned training image group, and after all the training image groups are aligned, the step of inputting the first image in each training image group into the preset network model is performed; of course; It can also be that before the first image in each training image group is input into the preset network model, the training image group is aligned to obtain the aligned training image group corresponding to the training image, and then the The first image in the aligned training image group is input to the preset network model.
  • the alignment process is performed on each training image group after the training image set is obtained, and after all the training image groups are aligned, the first image input in the training image set is executed. Preset the operation of the network model.
  • the alignment processing of the first image in the training image group with the second image corresponding to the first image means that the second image is used as a reference, and the pixels in the first image are compared with the second image.
  • the pixels in the image are aligned with the corresponding pixels, so that the alignment rate of the pixels in the first image and the pixels in the second image can reach a preset value, for example, 99%.
  • the pixel point in the first image aligned with the pixel point corresponding to it in the second image refers to: for the first pixel point in the first image and the second pixel point corresponding to the first pixel point in the second image Pixel, if the pixel coordinate corresponding to the first pixel is the same as the pixel coordinate corresponding to the second pixel, then the first pixel is aligned with the second pixel; if the pixel coordinate corresponding to the first pixel corresponds to the second pixel If the pixel coordinates are not the same, the first pixel point is aligned with the second pixel point.
  • the aligned image refers to an image obtained by performing alignment processing on the first image, and each pixel in the aligned image has the same pixel coordinates as its corresponding pixel in the second image.
  • the first image corresponding to the alignment image is replaced with the alignment image to update the training image group, so that the first image and the second image in the updated training image group are spatially aligned.
  • the alignment processing of the first image in the training image group with the second image corresponding to the first image specifically includes:
  • M12. Determine an alignment mode corresponding to the first image according to the pixel deviation amount, and perform alignment processing on the first image and the second image by using the alignment mode.
  • the pixel deviation amount refers to the total number of first pixels in the first image that are not aligned with the second pixels corresponding to the first pixel in the second image.
  • the pixel deviation amount can be obtained by obtaining the first coordinate of each first pixel in the first image, and the second coordinate of each second pixel in the second image, and then comparing the first coordinate of the first pixel to its corresponding The second coordinate of the second pixel is compared.
  • first coordinate is the same as the second coordinate, it is determined that the first pixel is aligned with its corresponding second pixel; if the first coordinate is not the same as the second coordinate, the first coordinate is determined One pixel is not aligned with its corresponding second pixel, and finally the total number of all the first pixels that are not aligned is obtained to obtain the pixel deviation amount.
  • the first coordinate of the first pixel in the first image is (200, 200)
  • the second coordinate of the second pixel in the second image corresponding to the first pixel is (201, 200)
  • the total number of the first pixel that is not aligned is increased by one; when the first coordinate of the first pixel in the first image is (200, 200), the second When the second coordinate of the second pixel corresponding to the first pixel in the image is (200, 200), the first pixel is aligned with the second pixel, and the total number of misaligned first pixels remains unchanged .
  • a deviation amount threshold may need to be set.
  • the pixel deviation amount of the first image is obtained, the obtained pixel deviation amount can be compared with a preset deviation threshold value.
  • the alignment method corresponding to the first image is determined according to the pixel deviation amount, and the alignment method is used to combine the first image with the second image.
  • the image alignment processing includes:
  • the first pixel point set includes a plurality of first pixels in the first image
  • the second pixel set includes a plurality of second pixels in the second image
  • the second pixel in the second pixel set and the The first pixel points in the first pixel point set correspond to each other; for each first pixel point in the first pixel point set, the coordinate difference between the first pixel point and its corresponding second pixel point is calculated, and according to the The position of the first pixel is adjusted by the coordinate difference corresponding to the first pixel to align the first pixel with the second pixel corresponding to the first pixel.
  • the preset deviation threshold is set in advance, for example, the preset deviation threshold is 20.
  • the pixel deviation amount is less than or equal to the preset deviation amount threshold value, it means that when the pixel deviation amount is less than or equal to the preset deviation amount threshold value, the pixel deviation amount is less than or equal to the preset deviation amount threshold value.
  • the pixel deviation is less than or equal to the preset deviation threshold, it means that the spatial deviation between the first image and the second image is small.
  • the information aligns the first image and the second image.
  • the process of aligning the first image and the second image with the mutual information between the first image and its corresponding second image may adopt an image registration method, in which Mutual information is used as the metric criterion.
  • the metric criterion is iteratively optimized by the optimizer to obtain the alignment parameters.
  • the first image and the second image are aligned by the register that registers the alignment parameters, which ensures that the first image and the second image are aligned.
  • the basis of the alignment effect of the second image reduces the complexity of aligning the first image with the second image, thereby improving the alignment efficiency.
  • the optimizer mainly uses translation and rotation transformations to optimize the metric criteria through the translation and rotation transformations.
  • the pixel deviation is greater than the preset deviation threshold, indicating that the first image and the second image are spatially misaligned to a high degree.
  • the first image and the second image can be aligned by selecting the first pixel point set in the first image and the second pixel point set in the second image.
  • the first pixel point of the first pixel point set corresponds to the second pixel point of the second pixel point set in a one-to-one correspondence, so that any first pixel point in the first pixel point set can be in the second pixel point set.
  • a second pixel point is found, and the position of the second pixel point in the second image corresponds to the position of the first pixel point in the first image.
  • first pixel point set and the second pixel point set may be determined by determining the second pixel according to the corresponding relationship between the first pixel point and the second pixel point after the first pixel point set/the second pixel point set are obtained.
  • Point set/first pixel point set for example, the first pixel point set is generated by randomly selecting a plurality of first pixels in the first image, and the second pixel point is based on the first pixel point set. Each first pixel is determined.
  • both the first pixel point set and the second pixel point set are obtained by means of scale-invariant feature transform (sift), that is, the first pixel point set
  • the first pixel point is the first sift feature point in the first image
  • the second pixel point in the second pixel point set is the second sift feature point of the second image.
  • the calculated coordinate difference between the first pixel point and its corresponding second pixel point is the point-to-point matching of the first sift feature point in the first pixel point and the second sift feature point in the second pixel point set to Obtain the coordinate difference between each first sift feature point and its corresponding second sift feature point, and perform a position transformation on the first sift feature point according to the coordinate difference corresponding to the first sift feature point to transform the first sift feature point
  • the pixel points are aligned with the second sift feature points corresponding to the first sift feature point, so that the first sift feature point in the first image is at the same position as the second sift feature point in the second image, thereby realizing the first image and the second sift feature point.
  • the alignment of the two images is the point-to-point matching of the first sift feature point in the first pixel point and the second sift feature point in the second pixel point set to
  • the preset network model includes a down-sampling module 100 and a transformation module 200.
  • the preset network model is based on training In the first image in the image set, generating the generated image corresponding to the first image may specifically include:
  • the two-sided grid 10 is a three-dimensional two-sided grid obtained by adding a one-dimensional dimension representing pixel intensity to the pixel coordinates of a two-dimensional image, wherein the three dimensions of the three-dimensional bilateral network are pixels of the two-dimensional image.
  • the guide image is obtained by performing pixel-level operations on the first image, and the resolution of the guide image 50 is the same as the resolution of the first image.
  • the guide image 50 corresponds to the first image Grayscale image.
  • the down-sampling module 100 since the down-sampling module 100 is used to output the bilateral grid 10 and the guide image 50 corresponding to the first image, the down-sampling module 100 includes a down-sampling unit 70 and a convolution unit 30, and the down-sampling unit 70 The bilateral grid 10 corresponding to the first image is output, and the convolution unit 30 is used to output a guide image 50 corresponding to the first image.
  • the first image in the training image set is input to the down-sampling module, and the bilateral grid parameters corresponding to the first image are obtained through the down-sampling module
  • the guide image corresponding to the first image specifically includes:
  • the down-sampling unit 70 is used to down-sample the first image to obtain a feature image corresponding to the first image, and generate a bilateral grid corresponding to the first image according to the feature image,
  • the number of spatial channels is greater than the number of spatial channels of the first image.
  • the bilateral grid is generated based on the local features and global features of the feature image, where the local features are features extracted from local regions of the image, such as edges, corners, lines, curves, and attribute regions, etc.
  • the local feature may be a regional color feature.
  • the global features refer to features that represent attributes of the entire image, for example, color features, texture features, and shape features. In this embodiment, the global feature may be the color feature of the entire image.
  • the down-sampling unit 70 includes a down-sampling layer, a local feature extraction layer, a global feature extraction layer, and a fully connected layer, and the local feature extraction layer is connected to the down-sampling layer.
  • the global feature extraction layer is connected between the down-sampling layer and the fully connected layer, and the global feature extraction layer is connected in parallel with the local feature extraction layer.
  • the first image is input to the down-sampling layer as an input item, and the feature image is output through the down-sampling layer;
  • the feature image of the down-sampling layer is input to the local feature extraction layer and the global feature extraction layer, and the local feature extraction layer extracts the feature image Local features, the global feature extraction layer extracts the global features of the feature image;
  • the local features output by the local feature extraction layer and the global features output by the global feature extraction layer are respectively input to the fully connected layer to output the bilateral network corresponding to the first image through the fully connected layer grid.
  • the downsampling layer includes a downsampling convolutional layer and four first convolutional layers.
  • the convolution kernel of the first convolutional layer is 1*1, and the step size is 1;
  • the local feature extraction layer may include two second convolutional layers, the convolution kernels of the two second convolutional layers are both 3*3, and the step size is 1;
  • the global feature extraction layer may include two There are three third convolutional layers and three fully connected layers. The convolution kernels of the two third convolutional layers are all 3*3, and the step size is both 2.
  • the convolution unit 30 includes a fourth convolution layer, the first image is input to the fourth convolution layer, and the guidance image is input through the fourth convolution layer, wherein the resolution of the guidance image is the same as that of the first image.
  • the first image is a color image
  • the fourth convolution layer performs pixel-level operations on the first image, so that the guide image is a grayscale image of the first image.
  • the first image I is input to the down-sampling convolutional layer, and the down-sampling convolutional layer outputs a three-channel low-resolution image of 256x256 size.
  • the three-channel low-resolution image of 256x256 size passes through the four first convolutional layers in turn.
  • the first image is input to the convolution unit, and the guide image corresponding to the first image is input through the convolution unit.
  • the transformation module 200 includes a segmentation unit 40 and a transformation unit 60.
  • the guide image, Inputting the bilateral grid and the first image to the transformation module, and generating the generated image corresponding to the first image through the transformation module specifically includes:
  • the segmentation unit 40 includes an up-sampling layer, and the input items of the up-sampling layer are a guide image and a bilateral grid, and the bilateral grid is up-sampled through the guide image to obtain each of the first images.
  • the color transformation matrix of the pixel may be up-sampling the bilateral grid with reference to the guide map to obtain the color transformation matrix of each pixel in the first image.
  • the input items of the transformation unit 60 are the color transformation matrix of each pixel and the first image, and the color of the corresponding pixel in the first image is transformed by the color transformation matrix of each pixel to obtain the The generated image corresponding to the first image.
  • the preset network model corrects model parameters according to the second image corresponding to the first image and the generated image corresponding to the first image, and continues to execute the generation of the first image in the training image set according to the second image corresponding to the first image.
  • the step of generating an image corresponding to the first image until the training condition of the preset network model satisfies a preset condition, so as to obtain the image processing model.
  • the preset condition includes that the loss function value meets a preset requirement or the number of training times reaches a preset number.
  • the preset requirement may be determined according to the accuracy of the image processing model, which will not be described in detail here.
  • the preset number of times may be the maximum number of training times of the preset network model, for example, 5000 times.
  • the generated image is output in the preset network model
  • the loss function value of the preset network model is calculated according to the generated image and the second image, and after the loss function value is calculated, it is determined whether the loss function value satisfies Preset requirements; if the loss function value meets the preset requirements, the training ends; if the loss function value does not meet the preset requirements, it is judged whether the training times of the preset network model reaches the predicted times, and if the preset times are not reached, The network parameters of the preset network model are corrected according to the loss function value; if the preset number of times is reached, the training ends. In this way, it is judged whether the training of the preset network model is completed through the loss function value and the number of training times, which can avoid the infinite loop of the training of the preset network model caused by the loss function value not meeting the preset requirements.
  • the network parameters of the preset network model are modified when the training condition of the preset network model does not meet the preset conditions (that is, the loss function value does not meet the preset requirements and the number of training times does not reach the preset number)
  • the loss function value does not meet the preset requirements and the number of training times does not reach the preset number
  • the network model After correcting the network parameters of the preset network model according to the loss function value, it is necessary to continue training the network model, that is, continue to perform the step of inputting the first image in the training image set into the preset network model.
  • the first image in the training image set that continues to be input into the preset network model is the first image that has not been input to the preset network model as an input item.
  • all first images in the training image set have unique image identifiers (for example, image numbers), and the image identifier of the first image input for the first training is different from the image identifier of the first image input for the second training, for example,
  • the image number of the first image input in one training session is 1, the image number of the first image input in the second training session is 2, and the image number of the first image input in the Nth training session is N.
  • the first images in the training image set can be sequentially input to the preset network model to compare the preset network model.
  • the operation of sequentially inputting the first images in the training image set to the preset network model can be performed to make the training images in the training image set
  • the group is cyclically input to the preset network model.
  • the degree of diffusion of the highlight part of the image taken at different exposures is different, so that the degree of diffusion of the highlight part of the image captured by the under-screen imaging system under different light intensities is different, so that the image captured by the under-screen imaging system The quality is different. Therefore, when training the image processing model, multiple training image sets can be obtained, and each training image set corresponds to a different exposure, and each training image set is used to train the preset network model to obtain each The model parameters corresponding to the training image set. In this way, using the first image with the same exposure as the training sample image can increase the training speed of the network model, and at the same time make different exposures correspond to different model parameters.
  • the corresponding model parameters can be selected according to the exposure degree of the image to be processed, and the diffusion of the highlight part of the image at each exposure degree can be suppressed, so as to improve the image quality of the processed image corresponding to the image to be processed.
  • the training image set includes several training sub-image sets, each training sub-image set includes multiple sets of training sample image groups, and each training sub-image set includes several sets of training sample images.
  • the exposure of the first image in any two training sample image groups in several training image groups is the same (that is, for each training image group, the first image in each training sample image group in the group is the same).
  • the exposure of one image is the same
  • the exposure of the second image in each training sample image group in the several training image groups is within the preset range
  • the exposure of the first image in any two training sub-image sets The degrees are not the same.
  • the preset range of the exposure of the second image may be determined according to the exposure time and ISO (the aperture of the existing mobile phone is a fixed value), and the preset range of the exposure represents the range of the image taken without exposure compensation.
  • Exposure the second image captured by the on-screen camera at the first exposure within the preset range of exposure is a normal exposure image, which can be obtained by training based on the training image set by using the normal exposure image as the second image
  • the image output by the image processing model has normal exposure, so that the image processing model has the function of brightening. For example, when the image A input to the image processing model is a low-exposure image, then after the image A is processed by the image processing model, the exposure of the processed image A can be made the normal exposure, thereby improving the image A's exposure. Image brightness.
  • the training image set may include 5 training sub-image sets, which are respectively denoted as the first training sub-image set, the second training sub-image set, the third training sub-image set, the fourth training sub-image set, and the fifth training sub-image set.
  • the exposure of the first image in each training image group included in the first training sub-image set corresponds to 0 level, and the second image is the image with the exposure within a preset range;
  • the second training sub-image set The exposure of the first image in each training image group included corresponds to the -1 level, and the second image is the image with the exposure within the preset range;
  • the third training sub-image set contains the first image in each training image group.
  • the exposure of one image corresponds to -2 level, and the second image is an image whose exposure is within a preset range; the exposure of the first image in each training image group included in the fourth training sub-image set corresponds to -3 level , The second image is an image with an exposure within a preset range; the exposure of the first image in each training image group included in the fifth training sub-image set corresponds to -4 level, and the second image is an exposure in the preset range.
  • the number of training image groups contained in the first training sub-image set, the second training sub-image set, the third training sub-image set, the fourth training sub-image set, and the fifth training sub-image set may be The same can be different.
  • the first training sub-image set, the second training sub-image set, the third training sub-image set, the fourth training sub-image set, and the fifth training sub-image set all include 5000 training image groups.
  • the training sub-image set is a training image set of the preset network model
  • the preset network model is trained through the training sub-image set to obtain the corresponding training sub-image set Model parameters.
  • the process of using the training sub-image set as a training image set to train the preset network model includes: the preset network model generates a generated image corresponding to the first image according to the first image in the training sub-image set; the preset The network model corrects the model parameters according to the second image corresponding to the first image and the generated image corresponding to the first image, and the preset network model continues to execute the first image in the training sub-image set to generate the first image Corresponding steps of generating an image until the training condition of the preset network model meets the preset conditions to obtain the model parameters corresponding to the training sub-image.
  • step S10 and step S20 can be parameterized, which will not be repeated here.
  • each training sub-image set for the preset network model is independent of each other, that is, each training sub-image set is used to train the preset network model.
  • training the preset network model with a set of training sub-images can obtain several model parameters.
  • Each model parameter is trained based on a training sub-image set, and any two model parameters are corresponding to the training sub-models.
  • the image sets are different from each other. It can be seen that the image processing model corresponds to several model parameters, and several model parameters correspond to several training sub-image sets one-to-one.
  • the image processing model It includes 5 model parameters, which are respectively marked as the first model parameter, the second model parameter, the third model parameter, the fourth model parameter, and the fifth model parameter.
  • the first model parameter corresponds to the first training sub-image set
  • the second The model parameter corresponds to the second training sub-image set
  • the third model parameter corresponds to the third training sub-image set
  • the fourth model parameter corresponds to the fourth training sub-image set
  • the fifth model parameter corresponds to the fifth training sub-image set.
  • the preset network model is trained according to each training sub-image set.
  • the training image set includes 5 training sub-image sets as an example.
  • the process of using the first training sub-image set, the second training sub-image set, the third training sub-image set, the fourth training sub-image set, and the fifth training sub-image set to separately train the preset network model can be: first adopt The first training sub-image set trains the preset network model to obtain the first model parameters corresponding to the first training sub-image set, and then the second training sub-image set is used to train the preset network model to obtain the second training sub-image set.
  • the second model parameter corresponding to the image set is sequentially deduced to obtain the fifth model parameter corresponding to the fifth training sub-image set.
  • each training sub-image set has an impact on the model parameters of the preset network model.
  • training sub-image Set A includes 1000 training image groups
  • training sub-image set B includes 200 training image groups. Then, first use training sub-image set A to train the preset network model, and then use training sub-image set B to pre-set
  • the model parameters corresponding to the training sub-image set B obtained by the network model training, and the model parameters corresponding to the training sub-image set B obtained by training the preset network model only with the training sub-image set B, are different.
  • the preset network model may be initialized first, and then the initialized preset network model can be used to download A training sub-image set for training.
  • the preset network model is trained according to the first training sub-image set, and after the first model parameters corresponding to the first training sub-image set are obtained, the preset network model can be initialized so as to be used for training the second
  • the initial model parameters and model structure of the preset network model of model parameters are the same as the preset network model used to train the first model parameters.
  • the preset network model can be initialized, so that the initial model parameters and model structure of the preset network model corresponding to each training sub-image set are the same.
  • the preset network model is trained according to the first training sub-image set, and after obtaining the first model parameters corresponding to the first training sub-image set, it can also be directly used after training based on the first training sub-image set.
  • the preset network model trains the second training sub-image set to obtain the second model parameters corresponding to the second training sub-image set, and continue to execute the preset network model (configure the second model parameters) according to The third training sub-image set is trained until the fifth training sub-image set is completed, and the fifth model parameter corresponding to the fifth training sub-image set is obtained.
  • the first training sub-image set, the second training sub-image set, the third training sub-image set, the fourth training sub-image set, and the fifth training sub-image set all include a certain number of training image groups, so that each group of training images All sub-images can meet the training requirements of the preset network model.
  • the training image group in the training sub-image set can be input to the preset network model in a loop, so as to compare the preset network The model is trained so that the preset network model meets the preset requirements.
  • the acquisition process of acquiring training samples containing each training sub-image set may be as follows: firstly, the under-screen imaging system is set to the first exposure, and the first exposure is acquired through the under-screen imaging system. The first image in the training sub-image set, and the second image corresponding to the first image in the first training sub-image set obtained through the on-screen imaging system; after the acquisition of the first training sub-image set is completed, set the off-screen imaging system to For the second exposure, the first image in the second training sub-image set and the second image corresponding to the first image are acquired through the off-screen imaging system and the on-screen imaging system; after the acquisition of the second training sub-image set is completed; continue to execute the setting screen Lower the exposure of the imaging system and the steps of obtaining the training sub-image set until all the training sub-image sets contained in the training image set are obtained.
  • the number of training image groups contained in each training sub-image set contained in the training image set may be the same or different. In an implementation of this embodiment, the number of training image groups contained in each training sub-image set contained in the training image set may be the same. For example, the number of training image groups contained in each training sub-image set is 5000. .
  • each training sub-image set corresponds to a different exposure
  • the model corresponding to the training sub-image set can be The parameter is associated with the exposure corresponding to the training sub-image set to establish the corresponding relationship between the exposure and the model parameter.
  • the exposure of the image to be processed can be obtained first, and then the model parameters corresponding to the image to be processed are determined according to the exposure, and then the model parameters corresponding to the image to be processed are configured in the preset network Model to obtain an image processing model corresponding to the image to be processed, so that the image processing model can be used to process the image to be processed.
  • image processing models with different network parameters can be determined for images to be processed with different exposures, and the image processing models corresponding to the images to be processed can be used to process the images to be processed to avoid the impact of exposure on color cast, thereby improving the removal of the to-be-processed images.
  • the second image may adopt a normal exposure, so that the processed image output by the image processing model has a normal exposure, and the image to be processed has a brightening effect.
  • this embodiment also provides an image processing method. As shown in FIG. 10, the image processing method includes:
  • A100 Obtain an image to be processed, and input the image to be processed into the image processing model.
  • the image to be processed may be an image taken by an image device used to process the image to be processed, or an image stored in an image processing device by another external device, and an image sent through the cloud.
  • the image to be processed is an image captured by an under-screen imaging system (for example, an under-screen camera), where the under-screen imaging system may be configured by the imaging device itself, or may be other devices. Configured.
  • the image to be processed is an image of a person captured by a mobile phone equipped with an under-screen imaging system.
  • the image processing model may be pre-trained by the image device that processes the image to be processed (for example, a mobile phone equipped with an under-screen camera), or it may be transplanted to the image device by other trained files corresponding to the image processing model middle.
  • the image device can use the image processing model as a color-removing functional module, and when the image device acquires the image to be processed, the color-removing functional module is activated, and the image to be processed is output to the image processing model.
  • the image processing model includes several model parameters, and each model parameter corresponds to an exposure degree. Therefore, in this implementation, after the image to be processed is acquired, the number of model parameters included in the image processing model can be detected first, and when the number of model parameters is one, the image to be processed is directly input to the image processing model.
  • the image to be processed is processed by the image processing; when there are multiple model parameters, the exposure of the image to be processed can be obtained first, and then the model parameter corresponding to the image to be processed is determined according to the exposure,
  • the model parameters corresponding to the image to be processed are configured in the image processing model to update the model parameters of the image processing parameter configuration, and the image to be processed is input to the updated image processing model.
  • the image processing model corresponds to a number of model parameters
  • each model parameter is obtained by training according to a training sub-image set
  • any two model parameters respectively correspond to the training sub-sets.
  • the image sets are different from each other (for example, the training sub-image set corresponding to model parameter A and the training sub-image set corresponding to model parameter B are different).
  • the acquiring the image to be processed and inputting the image to be processed into the image processing model specifically includes:
  • A101 Acquire an image to be processed, and extract the exposure of the image to be processed.
  • the exposure degree is the degree to which the photosensitive element of the image acquisition device is irradiated by light, and is used to reflect the degree of exposure during imaging.
  • the image to be processed may be an RGB three-channel image, the exposure of the image to be processed is determined according to the highlight area of the image to be processed, the R (that is, the red channel) value, G of each pixel contained in the highlight area At least one of the value of (ie, the green channel) and the value of B (ie, the blue channel) is greater than the preset threshold.
  • the image to be processed may also be a Y channel image or a Bell format image, and when the image to be processed is a Y channel image or a Bell format image (Raw format), the image to be processed is extracted Before the image, the Y-channel image or the Bell format image needs to be converted into an RGB three-channel image, so as to determine the highlight area of the image to be processed according to the red channel R value, the green channel G value and the blue channel B value of the image to be processed .
  • the extracting the exposure of the image to be processed specifically includes:
  • B10 Determine a third pixel that meets a preset condition according to the red channel R value, the green channel G value, and the blue channel B value of each pixel in the image to be processed, where the preset condition is the R value, At least one of the G value and the B value is greater than a preset threshold;
  • the image to be processed is an RGB three-channel image, so for each pixel in the image to be processed, the pixel includes the red channel R value, the green channel G value, and the blue channel B value.
  • the R value of the red channel, the G value of the green channel, and the B value of the blue channel of the pixel can be obtained. Therefore, in the process of extracting the exposure of the image to be processed, first, for each pixel of each image to be processed, the red channel R value, the green channel G value, and the blue channel B value of the pixel are acquired.
  • the preset condition is that the preset condition is that at least one of the R value, the G value, and the B value is greater than a preset threshold.
  • the third pixel meeting the preset condition means that the R value of the third pixel is greater than the preset threshold.
  • the G value of the third pixel is greater than the preset threshold
  • the B value of the third pixel is greater than the preset threshold
  • the R and G values of the third pixel are both greater than the preset threshold
  • the R and B values of the third pixel Are greater than the preset threshold
  • the G value and the B value of the third pixel are both greater than the preset threshold
  • the R value, B value, and G value of the third pixel are all greater than the preset threshold.
  • all the acquired third pixels are recorded as the third pixel point set.
  • the third pixel point set there are adjacent pixels, and there are also non-adjacent pixels. Pixels, where adjacent pixels refer to the positions of the pixels in the image to be processed are adjacent, and non-adjacent means that the positions of the pixels in the image to be processed are not adjacent, and the positions are similar to each other. In the pixel coordinates to be processed, one of the abscissa and ordinate of two adjacent pixels is the same.
  • the third pixel point set includes the pixel point (100, 101), the pixel point (100, 100), the pixel point (101, 101) and the pixel point (200, 200), then the pixel point (100, 101) and the pixel point (100, 100) are adjacent pixels, And the pixel point (100, 101), the pixel point (101, 101) are adjacent pixels, and the pixel point (100, 101), the pixel point (100, 100), the pixel point (101, 101) and the pixel point (200, 200) are all non-adjacent pixels.
  • the highlight area is a connected area formed by adjacent pixels in the third pixel point set, that is, the pixel value of each third pixel point included in the highlight area meets a preset condition. Therefore, in an implementation manner of this embodiment, the determining the highlight area of the image to be processed according to all the third pixel points that meet the preset condition specifically includes:
  • R value, G value and B value of the third pixel point are of the same R value, G value and/or B value which is greater than the preset threshold;
  • the connected area is a closed area formed by all adjacent third pixel points in the third pixel point set, each pixel point contained in the connected area is a third pixel point, and for each pixel point in the connected area For the third pixel point A, at least one third pixel point B in the connected area is adjacent to the third pixel point A. At the same time, for the third pixel point, each third pixel point C except the third pixel point included in the connected area is concentratedly removed, and the third pixel point C is not corresponding to any third pixel point A in the connected area. adjacent.
  • the third pixel point set includes pixel point (100,101), pixel point (100,100), pixel point (101,100), pixel point (101,101), pixel point (100,102) and pixel point (200,200), then pixel point (100,101) ), pixel (100, 100), pixel (101, 100), pixel (101, 101), pixel (100, 102) form a connected area.
  • the connected area of the image to be processed is formed by a light source, and the light source will produce the same color of light. Therefore, after all the connected areas contained in the image to be processed are acquired, the connected areas can be selected according to the area color corresponding to each connected area. Therefore, after the connected area of the image to be processed is obtained, it is determined that the R value, G value, and B value of each third pixel in the connected area are greater than the preset value among the R value, G value, and B value of the third pixel. Whether the types of the R value, G value and/or B value of the threshold are the same, to determine whether the connected area meets the preset rule. The same type refers to the two third pixels, which are respectively marked as pixel A and pixel B.
  • the R value of pixel A is greater than the preset threshold, then only the R value of pixel B is greater than the preset threshold. ; If the R and G values of pixel A are greater than the preset threshold, then only the R and G values of pixel B are greater than the preset threshold; if the R, G, and B values of pixel A are greater than the preset Threshold, then the R value, G value, and B value of pixel B are all greater than the preset threshold.
  • the different types refer to the two third pixel points, which are respectively marked as pixel point C and pixel point D.
  • V value can be one of R value, G value, and B value
  • M value is R value, G value and B value One of the two values excluding the V value
  • the preset rule is that the R value, G value, and B value of the third pixel in each connected area are of the same R value, G value, and/or B value that are greater than the preset threshold.
  • the target area can be filtered according to the area of the target area to obtain the highlight area.
  • the area of the target area refers to the area of the area where the target area is located in the image to be processed, and the area is calculated in the pixel coordinate system of the image to be processed.
  • the area of each target area can be compared, and the target area with the largest area is selected, and the target area is regarded as the highlight area, so that the target area with the largest area is regarded as the highlight area.
  • the exposure is determined according to the area with the largest brightness area, which can improve the accuracy of the exposure.
  • the determining the exposure of the image to be processed according to the highlight area specifically includes:
  • the corresponding relationship between the ratio interval and the exposure degree is preset. After obtaining the ratio, first obtain the ratio area where the ratio is located, and then determine the The ratio interval corresponds to the exposure to obtain the exposure of the image to be processed.
  • the corresponding relationship between the ratio interval and the exposure is: when the interval is [0,1/100), the exposure corresponds to 0 level; when the interval is [1/100,1/50), the exposure corresponds to ⁇ 1 level; when the interval is [1/50,1/20), the exposure corresponds to -2 level; when the interval is [1/20,1/50), the exposure corresponds to -3 level; when the interval is [1 /20,1], the exposure corresponds to -4 level. Then when the ratio of the first area to the second area is 1/10, the ratio is in the interval [1/20, 1], so that the exposure degree corresponding to the image to be processed is -4 level.
  • A102 Determine model parameters corresponding to the image to be processed according to the exposure, and use the model parameters to update the model parameters of the image processing model.
  • the corresponding relationship between the exposure and the model parameters is established during the training of the image processing model, so that after the exposure of the image to be processed is obtained, the model parameter corresponding to the exposure can be determined according to the corresponding relationship between the exposure and the model parameters
  • the exposure degree refers to the exposure degree level, that is, the corresponding relationship between the exposure degree and the model parameter is the corresponding relationship between the exposure degree level and the model parameter.
  • each exposure level corresponds to a ratio interval. After the image to be processed is obtained, the ratio of the area of the highlight area to the image image can be obtained, and the ratio is determined.
  • the exposure level corresponding to the image to be processed is determined according to the ratio area, and finally the model parameter corresponding to the image to be processed is determined according to the exposure level, so as to obtain the model parameter corresponding to the image to be processed.
  • the acquired model parameters are used to update the model parameters of the image processing model configuration to update the image processing model, that is, the image processing model corresponding to the acquired model parameters.
  • A103 Input the image to be processed into the updated image processing model.
  • the image to be processed is used as an input item of the updated image processing model, and the image to be processed is output to the updated image processing model to process the image to be processed.
  • the model parameters of the image processing model corresponding to the image to be processed are model parameters determined according to the exposure of the image to be processed, and the model parameters are model parameters obtained by training a preset network model, In this way, the accuracy of the image processing to be processed by the updated image processing model can be ensured. So far, the introduction of step A100 (that is, obtaining the image to be processed and inputting the image to be processed into the image processing model) is completed, and the subsequent steps of step A100 are described below.
  • A200 Perform color cast processing on the image to be processed through the image processing model to obtain a processed image corresponding to the image to be processed.
  • the depigmentation of the image to be processed by the image processing model refers to inputting the image to be processed into the image processing model as an input item of the image processing model, and passing through the image processing model.
  • the image processing model removes the color cast of the image to be processed, that is, removes the first target pixel of the image to be processed to obtain a processed image, wherein the processed image corresponds to the image to be processed.
  • the image after color cast processing is performed on the image to be processed through the processing model that is, the image to be processed is a color cast image corresponding to the processed image. For example, after the image to be processed as shown in FIG. 12 is processed through the image, the processed image as shown in FIG. 13 is obtained.
  • the image processing model includes a down-sampling module and a transformation module, so that when processing the image to be processed through the image processing model, it needs to pass through the down-sampling module and the transformation module in sequence To process.
  • the image processing model includes; the performing color cast processing on the image to be processed by the image processing model to obtain the processed image corresponding to the image to be processed specifically includes:
  • A201 Input the to-be-processed image into the down-sampling module, and obtain a bilateral grid corresponding to the to-be-processed image and a guide image corresponding to the to-be-processed image through the down-sampling module, wherein The resolution is the same as the resolution of the image to be processed;
  • A202 Input the guidance image, the bilateral grid, and the image to be processed into the transformation module, and generate a processed image corresponding to the first image through the transformation module.
  • the input items of the down-sampling module are the image to be processed
  • the output items are the bilateral grid and the guidance image corresponding to the image to be processed
  • the input items of the transformation module are the guidance image, the bilateral grid, and the image to be processed
  • the output item is the processed image.
  • the structure of the down-sampling module is the same as the structure of the down-sampling module in the preset network model. For details, reference may be made to the description of the structure of the down-sampling module in the preset network model.
  • the processing of the image to be processed by the down-sampling module of the image processing model is the same as the processing of the first image by the down-sampling module in the preset network model.
  • step S11 the specific execution process of step A201 can refer to step S11.
  • the structure of the transformation module is the same as the structure of the transformation module in the preset network model.
  • the processing of the image to be processed by the transformation module of the image processing model is the same as the processing of the first image by the transformation module in the preset network model, so the specific execution process of step A202 can refer to step S12.
  • the down-sampling module includes a down-sampling unit and a convolution unit.
  • the inputting the to-be-processed image into the down-sampling module, and obtaining the bilateral grid corresponding to the to-be-processed image and the guidance image corresponding to the to-be-processed image through the down-sampling module specifically includes:
  • A2011 Input the to-be-processed image into the down-sampling unit and the convolution unit respectively;
  • A2012. Obtain a bilateral grid corresponding to the image to be processed through the down-sampling unit, and obtain a guide image corresponding to the image to be processed through the convolution unit.
  • the input item of the down-sampling unit is the image to be processed
  • the output item is a bilateral grid
  • the input item of the convolution unit is the image to be processed
  • the output item is the guide image.
  • the structure of the down-sampling unit is the same as the structure of the down-sampling unit in the preset network model.
  • the processing of the image to be processed by the down-sampling unit of the image processing model is the same as the processing of the first image by the down-sampling unit in the preset network model, so the specific execution process of step A2011 can refer to step S111.
  • the structure of the convolution unit is the same as the structure of the convolution unit in the preset network model.
  • the processing of the image to be processed by the convolution unit of the image processing model is the same as the processing of the first image by the convolution unit in the preset network model, so the specific execution process of step A2012 can refer to step S112.
  • the transformation module includes a segmentation unit and a transformation unit.
  • the inputting the guide image, the bilateral grid, and the image to be processed into the transformation module, and generating the processed image corresponding to the image to be processed through the transformation module specifically includes:
  • A2021. Input the guidance image to the segmentation unit, and segment the bilateral grid by the segmentation unit to obtain a color transformation matrix of each pixel in the image to be processed;
  • A2022 input the image to be processed and the color transformation matrix of each pixel in the image to be processed into the transformation unit, and generate a processed image corresponding to the image to be processed through the transformation unit.
  • the input items of the segmentation unit are the guidance image and the bilateral grid
  • the output items are the color transformation matrix of each pixel in the image to be processed
  • the input items of the transformation unit are the image to be processed and the image to be processed.
  • the color transformation matrix of each pixel, and the output item is the processed image.
  • the structure of the segmentation unit is the same as the structure of the segmentation unit in the preset network model.
  • the segmentation unit of the image processing model processes the bilateral grid corresponding to the image to be processed and the guidance image, and the processing process of the downsampling unit in the preset network model on the bilateral grid corresponding to the first image and the guidance image is the same.
  • step S121 the specific execution process of step A2021 can refer to step S121.
  • the structure of the transformation unit is the same as the structure of the transformation unit in the preset network model.
  • the transformation unit of the image processing model is based on the color transformation matrix of each pixel in the image to be processed, and the transformation unit in the preset network model is based on the color transformation matrix of each pixel in the first image.
  • the processing procedure is the same, so the specific execution procedure of step A2022 can refer to step S122.
  • the network structure corresponding to the image processing model during the training process is the same as the network structure corresponding to the application process (de-paste processing of the image to be processed).
  • the image processing model includes a down-sampling module and a transformation module. Accordingly, when the image to be processed is de-pigmented through the image processing model, the image processing model also includes a down-sampling module and a transformation module.
  • the down-sampling module of the image processing model includes a down-sampling unit and a convolution unit, and the transformation module includes a segmentation unit and a transformation unit; accordingly, when the image to be processed is de-casted by the image processing model ,
  • the down-sampling module can also include a down-sampling unit and a convolution unit, and the transformation module includes a segmentation unit and a transformation unit; and in the application process, the working principle of each layer is the same as the working principle of each layer in the training process. Therefore, the input and output conditions of each layer of neural network in the application process of the image processing model can be referred to the relevant introduction in the training process of the image processing model, which will not be repeated here.
  • the present invention provides a method for generating and processing an image processing model.
  • the generating method includes: the method inputs a preset training image into a preset network model by collecting a first image into a preset network model, and a generated image generated by the preset network model and a second image corresponding to the first image pair the preset model Perform the sequence to get the image processing model.
  • the image processing model is obtained by in-depth learning of the color cast process of the training image set with multiple training image groups. Each training image group includes a first image and a second image. The first image corresponds to the second image.
  • the color cast of the image It can be seen from this that the present invention uses a trained image processing model based on training image sets for deep learning to perform color cast processing. This can quickly adjust the color cast of the image, that is, correct the color cast, improve the color quality of the image, and thereby improve Image Quality.
  • post-processing may be performed on the processed image, where the post-processing may include sharpening processing. And noise reduction processing.
  • the method further includes:
  • the sharpening processing refers to compensating the contour of the processed image, enhancing the edge of the processed image, and the portion of the grayscale jump, so as to improve the image quality of the processed image.
  • the sharpening processing may adopt an existing sharpening processing method, for example, a high-pass filtering method.
  • the noise reduction processing refers to removing noise in the image and improving the signal-to-noise ratio of the image.
  • the noise reduction processing may adopt an existing noise reduction algorithm or a trained noise reduction network model, etc., for example, the noise reduction processing adopts a Gaussian low-pass filtering method or the like.
  • this embodiment provides an image processing model generation device, and the image processing model generation device includes:
  • the first generation module 101 is configured to use a preset network model to generate a generated image corresponding to the first image according to the first image in the training image set, wherein the training image set includes multiple training image groups, each The training image group includes a first image and a second image, and the first image is a color cast image corresponding to the second image;
  • the first correction module 102 is configured to use a preset network model to correct the model parameters according to the second image corresponding to the first image and the generated image corresponding to the first image, and to continue to perform the collection based on the training image
  • the step of generating an image corresponding to the first image in the next training image group is generated until the training condition of the preset network model meets the preset condition, so as to obtain the image processing model.
  • the number of first target pixels in the first image that meets the preset color cast condition meets the preset number condition;
  • the preset color cast condition is the value of the first target pixel in the first image
  • the error between the display parameter and the display parameter of the second target pixel in the second image satisfies a preset error condition, wherein there is a one-to-one correspondence between the first target pixel and the second target pixel.
  • the first target pixel is any pixel in the first image or any pixel in the target area of the first image.
  • the training image set includes several training sub-image sets, each training sub-image set includes several training sample image groups, and any two training sample image groups in the training image groups
  • the exposure of an image is the same, the exposure of the second image in each training sample image group in the several training image groups is within the preset range, and the exposure of the first image in any two training sub-image sets is different. same.
  • the image processing model corresponds to a number of model parameters, and each model parameter is obtained by training according to a training sub-image set in the training image set, and any two model parameters respectively correspond to the training sub-images.
  • the image sets are different from each other.
  • the preset network model includes a down-sampling module and a transformation module; the first generation module is specifically configured to:
  • Input the first image in the training image set to the down-sampling module obtain the bilateral grid corresponding to the first image and the guidance image corresponding to the first image through the down-sampling module; and transfer the guidance
  • the image, the bilateral grid, and the first image are input to the transformation module, and a generated image corresponding to the first image is generated by the transformation module, wherein the resolution of the guide image is the same as the resolution of the first image
  • the rate is the same.
  • the down-sampling module includes a down-sampling unit and a convolution unit; the first generation module is specifically configured to:
  • the transformation module includes a segmentation unit and a transformation unit, and the first generation module is specifically configured to:
  • the first image and the color transformation matrix of each pixel in the first image are input to the transformation unit, and a generated image corresponding to the first image is generated by the transformation unit.
  • the first image is an image captured by an off-screen imaging system.
  • the under-screen imaging system is an under-screen camera.
  • the device for generating the image processing model further includes:
  • the first alignment module is used to align the first image in the training image group with the second image corresponding to the first image for each training image group in the training image set, to obtain the first image corresponding to the first image
  • An alignment image that is aligned with the two images, and the alignment image is used as the first image.
  • the first alignment module is specifically configured to:
  • the pixel deviation amount between the first image in the training image group and the second image corresponding to the first image is obtained; the pixel deviation amount is determined according to the pixel deviation amount.
  • the first alignment module is specifically configured to:
  • the first pixel point set of the first image and the second pixel point set of the second image are extracted, and the first pixel point set includes all A number of first pixels in the first image
  • the second pixel point set includes a number of second pixels in the second image, the second pixel point in the second pixel point set and the first pixel point
  • One-to-one correspondence between the first pixel in a pixel set for each first pixel in the first pixel set, the coordinate difference between the first pixel and its corresponding second pixel is calculated, and according to the first pixel
  • the coordinate difference value corresponding to the point performs position transformation on the first pixel to align the first pixel with the second pixel corresponding to the first pixel.
  • this embodiment provides an image processing device that applies the foregoing image processing model generation method or the image processing model generated by the generation device.
  • the image processing device includes:
  • the first acquisition module 201 is configured to acquire an image to be processed and input the image to be processed into the image processing model;
  • the first processing module 202 is configured to perform color cast processing on the image to be processed through the image processing model to obtain a processed image corresponding to the image to be processed.
  • the image processing model corresponds to a number of model parameters, and each model parameter is obtained by training based on a training sub-image set, and the training sub-image sets corresponding to each of any two model parameters are different from each other.
  • the first acquisition module is specifically configured to:
  • the image to be processed is input to the updated image processing model.
  • the first acquisition module is specifically configured to:
  • a third pixel that meets a preset condition is determined, where the preset condition is at least one of R value, G value, and B value Greater than the preset threshold;
  • the highlight area of the image to be processed is determined according to all the third pixel points that meet the preset condition, and the exposure degree of the image to be processed is determined according to the highlight area.
  • the first acquisition module is specifically configured to:
  • the R value, G value and B value of the three pixels are of the same R value, G value and/or B value which is greater than the preset threshold;
  • the first acquisition module is specifically configured to:
  • the exposure degree corresponding to the image to be processed is determined according to the ratio of the first area and the second area.
  • the image processing model includes a down-sampling module and a transformation module; the first processing module is specifically configured to:
  • the image to be processed is input to the down-sampling module, and the bilateral grid corresponding to the image to be processed and the guide image corresponding to the image to be processed are obtained through the down-sampling module, wherein the resolution of the guide image
  • the resolution is the same as the resolution of the image to be processed; and the guide image, the bilateral grid, and the image to be processed are input to the transformation module, and the processed image corresponding to the first image is generated by the transformation module .
  • the down-sampling module includes a down-sampling unit and a convolution unit; the first processing module is specifically configured to:
  • the transformation module includes a segmentation unit and a transformation unit
  • the first processing module is specifically configured to:
  • the guide image is input to the segmentation unit, and the bilateral grid is segmented by the segmentation unit to obtain the color transformation matrix of each pixel in the image to be processed; and
  • the image and the color transformation matrix of each pixel in the image to be processed are input to the transformation unit, and the processed image corresponding to the image to be processed is generated by the transformation unit.
  • the image processing device further includes:
  • the noise reduction processing unit is configured to perform sharpening and noise reduction processing on the processed image, and use the sharpened and noise reduction processed image as a processed image corresponding to the image to be processed.
  • This embodiment provides a method for generating an image processing model. As shown in FIGS. 15 and 16, the method includes:
  • the preset network model generates a generated image corresponding to the first image according to the first image in the training image set.
  • the preset network model is a deep learning network model
  • the training image set includes multiple sets of training image groups with different image content
  • each training image group includes a first image and a second image.
  • the image corresponds to the second image, and they present the same image scene.
  • the second image is a normally displayed image (or original image).
  • the image content of the first image corresponds to the second image, but the objects in the image content have ghosting Or a blur effect similar to ghosting.
  • the ghosting refers to the formation of a virtual image around the object in the image, for example, it may include the situation where one or multiple contours or virtual images appear on the edge of the object in the image, for example, when the object in the image has a double image (That is, when a double contour or virtual image appears at the edge of the object), the image column with a smaller pixel value can be understood as the real image of the object, and the other column image with a larger pixel value can be understood as the contour or virtual image of the object.
  • the first image and the second image correspond to the same image scene.
  • the first image and the second image corresponding to the same image scene means that the similarity between the image content carried by the first image and the image content carried by the second image reaches a preset threshold, and the image size of the first image is equal to that of the first image.
  • the image sizes of the two images are the same, so that when the first image and the second image overlap, the coverage rate of the object carried by the first image to the corresponding object in the second image reaches the preset condition.
  • the preset threshold may be 99%, and the preset condition may be 99.5% and so on.
  • the image content of the first image and the image content of the second image may be completely same.
  • the first image is an image with a ghost image with an image size of 600*800
  • the image content of the first image is a square
  • the positions of the four vertices of the square in the first image in the first image are respectively (200,300), (200,400), (300,400) and (300,300).
  • the image size of the second image is an image of 600*800
  • the image content of the second image is a square
  • the positions of the four vertices of the square in the second image in the second image are (200, 300) , (200,400), (300,400) and (300,300)
  • the first image covers the second image
  • the square in the first image It overlaps the square of the second image up and down.
  • the second image may be an image obtained by normal shooting, for example, an image taken by an under-screen camera after removing the display panel in the under-screen imaging system, or by making a light-shielding structure without data lines and scan lines.
  • the experimental display panel replaces the actual display panel, and then uses it as the display panel of the under-screen imaging system. Images sent by other external devices (such as smart phones).
  • the first image may be captured by an under-screen imaging system (for example, an under-screen camera), or may be obtained by processing the second image.
  • the processing of the second image refers to forming a ghost on the second image.
  • the image size and image content of the second image can be kept unchanged during the processing.
  • the first image is captured by an off-screen imaging system
  • the shooting parameters of the first image and the second image are the same
  • the shooting scene corresponding to the first image is the same as that of the first image.
  • the shooting scenes of the two images are the same.
  • the first image is the image shown in FIG. 17, the content of the image is relatively blurred due to the influence of the light-shielding structure in the display panel
  • the second image is the normally displayed image as shown in FIG. 18.
  • the shooting parameters may include the exposure parameters of the imaging system, where the exposure parameters may include aperture, door opening speed, sensitivity, focus, white balance, and the like.
  • the shooting parameters may also include ambient light, shooting angle, and shooting range.
  • the method further includes:
  • the first image in the training image group and the second image corresponding to the first image are aligned to obtain an alignment aligned with the second image Image, and use the aligned image as the first image.
  • each training image group in the training image set refers to performing alignment processing on each training image group in the training image set.
  • Each training image group is aligned to obtain an aligned training image group, and after all the training image groups are aligned, the step of inputting the first image in each training image group into the preset network model is performed; of course; It can also be that before the first image in each training image group is input into the preset network model, the training image group is aligned to obtain the aligned training image group corresponding to the training image, and then the The first image in the aligned training image group is input to the preset network model.
  • the alignment process is performed on each training image group after the training image set is obtained, and after all the training image groups are aligned, the first image input in the training image set is executed. Preset the operation of the network model.
  • the alignment processing of the first image in the training image group with the second image corresponding to the first image means that the second image is used as a reference, and the pixels in the first image are compared with the second image.
  • the pixels in the image are aligned with the corresponding pixels, so that the alignment rate of the pixels in the first image and the pixels in the second image can reach a preset value, for example, 99%.
  • the pixel point in the first image aligned with the pixel point corresponding to it in the second image refers to: for the first pixel point in the first image and the second pixel point corresponding to the first pixel point in the second image Pixel, if the pixel coordinate corresponding to the first pixel is the same as the pixel coordinate corresponding to the second pixel, then the first pixel is aligned with the second pixel; if the pixel coordinate corresponding to the first pixel corresponds to the second pixel If the pixel coordinates are not the same, the first pixel point is aligned with the second pixel point.
  • the aligned image refers to an image obtained by performing alignment processing on the first image, and each pixel in the aligned image has the same pixel coordinates as its corresponding pixel in the second image.
  • the first image corresponding to the alignment image is replaced with the alignment image to update the training image group, so that the first image and the second image in the updated training image group are spatially aligned.
  • the aligning the first image in the training image group with the second image corresponding to the first image specifically includes:
  • P12. Determine the alignment mode corresponding to the first image according to the pixel deviation amount, and use the alignment mode to align the first image with the second image.
  • the pixel deviation amount refers to the number of the first pixel in the first image that is not aligned with the second pixel in the second image corresponding to the first pixel.
  • the pixel deviation amount can be obtained by obtaining the first coordinate of each first pixel in the first image, and the second coordinate of each second pixel in the second image, and then comparing the first coordinate of the first pixel to its corresponding The second coordinate of the second pixel is compared.
  • first coordinate is the same as the second coordinate, it is determined that the first pixel is aligned with its corresponding second pixel; if the first coordinate is not the same as the second coordinate, the first coordinate is determined One pixel is not aligned with its corresponding second pixel, and finally the number of all the first pixels that are not aligned is obtained to obtain the pixel deviation amount.
  • the first coordinate of the first pixel in the first image is (100, 100)
  • the second coordinate of the second pixel in the second image corresponding to the first pixel is (101, 100)
  • the first pixel is not aligned with the second pixel, and the number of unaligned first pixels is increased by one;
  • the first coordinate of the first pixel in the first image is (100, 100)
  • the second coordinate of the second pixel point corresponding to the first pixel point is (100, 100)
  • the first pixel point is aligned with the second pixel point, and the first pixel point is not aligned.
  • a deviation amount threshold may need to be set.
  • the pixel deviation amount of the first image is obtained, the obtained pixel deviation amount can be compared with a preset deviation threshold value.
  • the alignment method corresponding to the first image is determined according to the pixel deviation amount, and the alignment method is used to combine the first image with the second image.
  • the image alignment processing includes:
  • the first pixel point set includes a plurality of first pixels in the first image
  • the second pixel set includes a plurality of second pixels in the second image
  • the second pixel in the second pixel set and the The first pixel points in the first pixel point set correspond to each other; for each first pixel point in the first pixel point set, the coordinate difference between the first pixel point and its corresponding second pixel point is calculated, and according to the The position of the first pixel is adjusted by the coordinate difference corresponding to the first pixel to align the first pixel with the second pixel corresponding to the first pixel.
  • the preset deviation threshold is set in advance, for example, the preset deviation threshold is 20.
  • the pixel deviation amount is less than or equal to the preset deviation amount threshold value, it means that when the pixel deviation amount is less than or equal to the preset deviation amount threshold value, the pixel deviation amount is less than or equal to the preset deviation amount threshold value.
  • the pixel deviation is less than or equal to the preset deviation threshold, it means that the spatial deviation between the first image and the second image is small.
  • the information aligns the first image and the second image.
  • the process of aligning the first image and the second image with the mutual information between the first image and its corresponding second image may adopt an image registration method, in which Mutual information is used as the metric criterion.
  • the metric criterion is iteratively optimized by the optimizer to obtain the alignment parameters.
  • the first image and the second image are aligned by the register that registers the alignment parameters, which ensures that the first image and the second image are aligned.
  • the basis of the alignment effect of the second image reduces the complexity of aligning the first image with the second image, thereby improving the alignment efficiency.
  • the optimizer mainly uses translation and rotation transformations to optimize the metric criteria through the translation and rotation transformations.
  • the pixel deviation is greater than the preset deviation threshold, indicating that the first image and the second image are spatially misaligned to a high degree.
  • the first image and the second image can be aligned by selecting the first pixel point set in the first image and the second pixel point set in the second image.
  • the first pixel point of the first pixel point set corresponds to the second pixel point of the second pixel point set in a one-to-one correspondence, so that any first pixel point in the first pixel point set can be in the second pixel point set.
  • a second pixel point is found, and the position of the second pixel point in the second image corresponds to the position of the first pixel point in the first image.
  • first pixel point set and the second pixel point set may be determined by determining the second pixel according to the corresponding relationship between the first pixel point and the second pixel point after the first pixel point set/the second pixel point set are obtained.
  • Point set/first pixel point set for example, the first pixel point set is generated by randomly selecting a plurality of first pixels in the first image, and the second pixel point is based on the first pixel point set. Each first pixel is determined.
  • both the first pixel point set and the second pixel point set are obtained by means of scale-invariant feature transform (sift), that is, the first pixel point set
  • the first pixel point is the first sift feature point in the first image
  • the second pixel point in the second pixel point set is the second sift feature point of the second image.
  • the calculated coordinate difference between the first pixel point and its corresponding second pixel point is the point-to-point matching of the first sift feature point in the first pixel point and the second sift feature point in the second pixel point set to Obtain the coordinate difference between each first sift feature point and its corresponding second sift feature point, and perform a position transformation on the first sift feature point according to the coordinate difference corresponding to the first sift feature point to transform the first sift feature point
  • the pixel points are aligned with the second sift feature points corresponding to the first sift feature point, so that the first sift feature point in the first image is at the same position as the second sift feature point in the second image, thereby realizing the first image and the second sift feature point.
  • the alignment of the two images is the point-to-point matching of the first sift feature point in the first pixel point and the second sift feature point in the second pixel point set to
  • the first image is an image with ghosting obtained by preprocessing the second image, and the image size and image content of the first image and the second image are the same,
  • This can improve the degree of similarity between the scene and imaging parameters corresponding to the first image and the scene and imaging parameters corresponding to the second image, and training the preset network model through the training image group with a high degree of scene similarity can improve the preset network model The training speed and the processing effect of the trained image processing model.
  • the specific process of preprocessing the second image may be: firstly generating a grayscale image based on the shading structure, secondly generating a point spread function based on the grayscale image, and finally generating based on the point spread function and the second image
  • the first image That is, the first image is generated according to the second image and the point spread function, and the point spread function is generated according to the grayscale image generated according to the light shielding structure.
  • the point spread function (PSF) is used to describe the response of the imaging system to a point light source or a point object, and the point spread function is formed by the spatial domain of the light transfer function of the imaging system.
  • the light-shielding structure may include signal lines, capacitor lines, and power lines of the display panel of the terminal device.
  • the signal line may include several data lines (e.g., S1, S2,..., Sn, where n is a positive integer) and several scan lines (e.g., G1, G2,..., Gm, where m is Positive integer), the data lines and the scan lines are alternately arranged horizontally and vertically to form a plurality of grids.
  • the multiple grids formed by the signal lines correspond to the multiple pixels configured on the display panel.
  • the light When light passes through the display panel, the light can transmit through each pixel, but cannot transmit through the light-shielding structure, so that the light is irradiated on the light-shielding structure The rays of light are diffracted. Then when the imaging system is set under the display panel, a number of shading structures and pixels in the display panel above the imaging system will appear in the shooting area. When the imaging system is shooting, the shading structure above the imaging system will cause shooting The image has blurring problems such as ghosting. Therefore, when processing the second image, the second image is processed according to the point spread function corresponding to the grayscale image generated by the shading structure to generate the first image with ghosting corresponding to the second image.
  • the image sizes of the first image and the second image may also have a certain error range, that is, the image sizes of the first image and the second image may also be different.
  • the same image content of the two images can be understood as meaning that the objects contained in each of the two images (such as people, objects, and backgrounds) are the same, but it does not mean that each object is in the two images.
  • the image quality is the same, that is, the same image content can indicate that the objects contained in the two images (such as people, objects, and backgrounds in the image) are the same, but it does not mean that the image quality of each object in the two images is the same. identical.
  • the on-screen imaging system is a terminal device with an on-screen camera, and the light-shielding structure (eg, signal line) of the terminal device is shown in FIG. 19, assuming that the under-screen imaging system is configured under the display panel corresponding to the light-shielding structure , Then, according to the grayscale image corresponding to the light-shielding structure (for example, including the signal line), the grayscale image in FIG. 71 may correspond to the data line in the signal line, and the second black line 72 may correspond to the scan line in the signal line.
  • the grayscale image in FIG. 71 may correspond to the data line in the signal line
  • the second black line 72 may correspond to the scan line in the signal line.
  • all the light-shielding structures corresponding to the display panel can be directly obtained, or a partial area of the light-shielding structure corresponding to the display panel can be obtained, and then all the light-shielding structures of the display panel can be simulated through the partial area, as long as it can be obtained Light-proof structure is sufficient.
  • the acquisition process of the training image group may be: first acquire the signal line of the terminal display panel, and select among the signal lines The signal line area, secondly, determine the grayscale image corresponding to the signal line area, generate a point spread function according to the grayscale image and the Fraunhofer diffraction formula; then take a second image through the on-screen imaging system, and The second image is convolved with the point spread function to obtain the first image corresponding to the second image; finally, the second image is associated with the first image generated from the second image to obtain the training image group.
  • each second image is convolved with the point spread function to obtain each second image in turn.
  • the first image corresponding to the image so as to obtain multiple training image groups, so that after all the second images needed by the training image group are obtained, the first image corresponding to each second image can be obtained by calculation to improve the training image The speed of group acquisition.
  • the signal lines corresponding to different display panels can contain different grid sizes
  • the signal lines of multiple display panels can be obtained when the grayscale image is generated, and the signal lines of multiple display panels can be obtained according to each display panel.
  • One signal line generates a grayscale image.
  • a grayscale image can be randomly selected from the generated multiple grayscale images, and the second image The processing is performed to obtain the first image corresponding to the second image, which can improve the de-ghosting effect of the image processing model training.
  • the preset network model includes an encoder and a decoder; the preset network model generates the first image in the training image set according to the The generated image corresponding to the first image specifically includes:
  • N11 Input the first image in the training image set to the encoder, and obtain the characteristic image of the first image through the encoder, wherein the image size of the characteristic image is smaller than the image size of the first image ;
  • N12. Input the characteristic image to the decoder, and output the generated image through the decoder, wherein the image size of the generated image is equal to the image size of the first image.
  • the preset network model adopts a decoding-encoding structure
  • the decoding-encoding structure is a convolutional neural network CNN structure
  • the encoder 100 is used to convert the input image into an image whose spatial size is smaller than the input image and The number of channels is more than the feature image of the input image
  • the decoder 200 is used to convert the feature image into a generated image with the same image size as the input image.
  • the encoder includes a first redundant learning layer 101 and a down-sampling layer 102 arranged in sequence. The first image in the training image group is input to the first redundant learning layer 101, and the first redundant learning layer 101 is passed through the first redundant learning layer.
  • the layer 101 outputs a first feature map with the same image size as the first image; the first feature image is input to the downsampling layer 102 as an input item of the downsampling layer 102, and the first feature image is downsampled through the downsampling layer 102 to output
  • the second feature image corresponding to the first image (the second feature image is the feature image of the first image generated by the encoder), wherein the image size of the second feature image is smaller than the image size of the first image.
  • the decoder 200 includes an up-sampling layer 201 and a second redundant learning layer 202 arranged in sequence.
  • the feature image output by the encoder 100 is input to the up-sampling layer 201, and the third feature is output after being up-sampled by the up-sampling layer 201.
  • the image, the third characteristic image is input to the second redundant learning layer 202, and the generated image is output after passing through the second redundant learning layer 202, wherein the image size of the generated image is the same as the image size of the first image.
  • multi-scale training can be performed on the preset network model, so that the de-ghosting effect of the image processing model obtained by training can be improved.
  • the first redundant learning layer 101 includes a first convolutional layer 11 and a first redundant learning module 12
  • the down-sampling layer 102 includes a first coding redundancy learning module 110 and The second coding redundancy learning module 120
  • the first coding redundancy learning module 110 includes a first down-sampling convolution layer 13 and a second redundancy learning module 14
  • the second coding redundancy learning module 120 includes a second down-sampling convolution Layer 15 and the third redundant learning module 16.
  • the input item of the first convolutional layer 11 is the first image
  • the first image is sampled to obtain the first feature image
  • the first feature image is input to the first redundant learning module 12 Perform feature extraction
  • the first feature image passing through the first redundant learning module 12 sequentially passes through the first down-sampling convolutional layer, the second redundant learning module 14, the second down-sampling convolutional layer 15, and the third redundant learning module 16 performs down-sampling to obtain the second feature image.
  • the first convolutional layer 11 samples the first image
  • the first down-sampling convolutional layer 13 and the second down-sampling convolutional layer 15 are both used to down-sample the input feature image.
  • the first redundant learning module 12, the second redundant learning module 14 and the third redundant learning module 16 are used to extract image features.
  • the first down-sampling convolutional layer 13 and the second down-sampling convolutional layer 15 may both use convolutional layers with a step size of 2.
  • a redundant learning module 12, a second redundant learning module 14 and a third redundant learning module 16 each include three redundant learning blocks arranged in sequence, and the three redundant learning blocks sequentially extract image features of the input image.
  • the first image is a 256*256 image
  • the first image is input to the first redundant learning layer 101 through the input layer, and after the first redundant learning layer 101, the first feature image of 256*256 is output
  • the feature image is input to the first down-sampling convolution layer 13 of the first coding redundancy learning module 110, and the fourth feature image with an image size of 128*128 is output through the first down-sampling convolution layer 13, and the fourth feature image passes through the first
  • the first redundancy learning module 12 of the coding redundancy learning module 110 performs feature extraction;
  • the fourth feature image of the first redundancy learning module 12 is input into the second down-sampling convolution layer 15 of the second coding redundancy learning module 120,
  • the second feature image with an image size of 64*64 is output through the second down-sampling convolutional layer 15, and the second feature image is subjected to feature extraction through the second redundancy learning module 16 of the second coding redundancy learning module 120.
  • the up-sampling layer 201 includes a first decoding redundancy learning module 210 and a second decoding redundancy learning module 220, and the first decoding redundancy learning module 210 includes a fourth redundancy learning module 21 and The first up-sampling convolutional layer 22, the second decoding redundancy learning module 220 includes a fifth redundancy learning module 23 and a second up-sampling convolutional layer 24, the second redundancy learning layer 202 includes a sixth redundancy learning Module 25 and second convolutional layer 26.
  • the input item of the first up-sampling convolutional layer 22 is the first feature image, and the input first feature image sequentially passes through the fourth redundant learning module 21, the first up-sampling convolutional layer 22, and the fifth redundant learning module.
  • the module 23 and the second up-sampling convolutional layer 24 perform up-sampling to obtain a third feature image, and input the third feature image to the sixth redundant learning module 25, and perform feature extraction through the sixth redundant learning module 25
  • the latter third characteristic image is input to the second convolution layer 26, and the generated image is obtained through the second convolution layer 26.
  • the first up-sampling convolutional layer 22 and the second up-sampling convolutional layer 24 are used for up-sampling the input feature image
  • the module 23 and the sixth redundant learning module 25 are both used to extract image features
  • the second convolutional layer 26 is used to sample the input feature images.
  • the first up-sampling convolutional layer 22 and the second up-sampling convolutional layer 24 are both deconvolutional layers with a step length of 2
  • the module 21, the fifth redundant learning module 23, and the sixth redundant learning module 25 each include three redundant learning blocks, and the three redundant learning blocks sequentially extract image features of the input image.
  • the third redundant learning block of the redundant learning module in the first redundant learning layer 101 is hop-connected to the first redundant learning block of the redundant learning module in the second redundant learning layer 202
  • the third redundant learning block of the redundant learning module in the first encoding redundancy learning module 110 is hop-connected to the first redundant learning block of the redundant learning module in the second decoding redundancy learning module 220.
  • the first image is a 256*256 image through the encoder 100 to obtain a 64*64 second feature image
  • the 64*64 second feature image is input through the fourth redundancy of the first decoding redundancy learning module 210.
  • the co-learning module 21 performs feature extraction.
  • the second feature image of 64*64 after feature extraction is input to the first up-sampling convolutional layer 22 of the first decoding redundant learning module 210, and the first up-sampling convolutional layer 22 is output.
  • a fifth feature image with an image size of 128*128.
  • the fifth feature image is subjected to feature extraction through the fifth redundant learning module 23 of the second decoding redundant learning module 220; the fifth feature image that passes through the fifth redundant learning module 23 Output the second up-sampling convolutional layer 24 of the second decoding redundancy learning module 220.
  • the third feature image with an image size of 256*256 is output, and the third feature image is input to the second redundancy
  • the learning layer 202 outputs a 256*256 generated image after passing through the second redundant learning layer 202.
  • the encoder and the decoder include a first convolutional layer, a second convolutional layer, a first up-sampling convolutional layer, a second up-sampling convolutional layer, a first down-convolution layer, and a second down-convolution layer.
  • the convolutional layer and the convolutional layer in all redundant learning modules use linear rectification functions as the activation function and the convolution kernels are all 5*5, which can improve the gradient transfer efficiency of each layer, and after many times of reverse Propagation, the gradient amplitude changes small, which improves the accuracy of the trained generator and at the same time increases the receptive field of the network.
  • the preset network model corrects the model parameters of the preset network model according to the second image corresponding to the first image and the generated image corresponding to the first image, and continues to perform the collection based on the training image
  • the step of generating an image corresponding to the first image in the next training image group is generated until the training condition of the preset network model meets the preset condition, so as to obtain a trained image processing model.
  • the correcting the preset network model refers to correcting the model parameters of the preset network model until the model parameters meet a preset condition.
  • the preset condition includes that the loss function value meets a preset requirement or the number of training times reaches a preset number.
  • the preset requirement may be determined according to the accuracy of the image processing model, which will not be described in detail here.
  • the preset number of times may be the maximum number of training times of the preset network model, for example, 4000 times.
  • the generated image is output in the preset network model
  • the loss function value of the preset network model is calculated according to the generated image and the second image, and after the loss function value is calculated, it is determined whether the loss function value satisfies Preset requirements; if the loss function value meets the preset requirements, the training ends; if the loss function value does not meet the preset requirements, it is judged whether the training times of the preset network model reaches the predicted times, and if the preset times are not reached, The network parameters of the preset network model are corrected according to the loss function value; if the preset number of times is reached, the training ends. In this way, it is judged whether the training of the preset network model is completed through the loss function value and the number of training times, which can avoid the infinite loop of the training of the preset network model caused by the loss function value not meeting the preset requirements.
  • the network parameters of the preset network model are modified when the training situation of the preset network model does not meet the preset conditions (for example, the loss function value does not meet the preset requirements and the number of training times does not reach the preset number)
  • the loss function value does not meet the preset requirements and the number of training times does not reach the preset number
  • the first image that continues to perform the input of the first image in the training image set into the preset network model may be the first image that has not been input to the preset network model as an input item.
  • all first images in the training image set have a unique image identifier (for example, image number), the image identifier of the first image input to the preset network model for the first training and the first image input for the preset network model for the second training
  • the image ID is different, for example, the image number of the first image input to the preset network model for the first training is 1, the image number of the first image input to the preset network model for the second training is 2, and the image number for the Nth training input
  • the image number of the first image of the preset network model is N.
  • the first images in the training image set can be sequentially input to the preset network model to compare the preset network model.
  • the operation of sequentially inputting the first images in the training image set to the preset network model can be performed to make the training images in the training image set
  • the group is cyclically input to the preset network model. It should be noted that in the process of inputting the first image into the preset network model training, it can be input in the order of the image number of each first image or not in the order of the image number of each first image. Of course, it can be repeated.
  • the same first image is used to train the preset network model, or the same first image is not used repeatedly to train the preset network model.
  • “continue to execute the first image in the training image set The specific implementation method of "Steps to Enter the Preset Network Model" is defined.
  • the loss function value is calculated from a structural similarity loss function and a content bidirectional loss function.
  • the preset network model corrects the model parameters of the preset network model according to the second image corresponding to the first image and the generated image corresponding to the first image, And continue to execute the step of generating an image corresponding to the first image according to the first image in the next training image group in the training image set, until the training condition of the preset network model meets the preset conditions, so as to obtain
  • the trained image processing model specifically includes:
  • N21 Calculate the structural similarity loss function value and the content bidirectional loss function value corresponding to the preset network model according to the second image corresponding to the first image and the generated image corresponding to the first image;
  • the preset network model adopts a structural similarity index (Structural similarity index, SSIM) loss function and a content bidirectional (Contextual bilateral loss, CoBi) loss function based on VGG (Visual Geometry Group Network, VGG network) extraction features Combine as a loss function. Then, when calculating the loss function value of the preset network model, the structural similarity loss function value and the content two-way loss function value can be calculated separately, and then the prediction is calculated based on the structure similarity loss function value and the content two-way loss function value. Set the value of the loss function in the network model.
  • the total loss function value of the preset network model a*structure similarity loss function value+b*content bidirectional loss function value, where a and b are weight coefficients.
  • a and b are weight coefficients.
  • the total loss function value of the preset network model the structural similarity loss function value+the content two-way loss function value.
  • the stochastic gradient descent method is used to train the preset network model, where the initial network parameters for training are set to 0.0001, and the network parameters are Use exponential decay to make corrections.
  • the structural similarity loss function value is used to measure the structural similarity between the generated image and the second image.
  • the expression of the structural similarity loss function corresponding to the structural similarity loss function value may be:
  • ⁇ x is the average value of the pixel values of all pixels in the generated image
  • ⁇ y is the average value of the pixel values of all pixels in the second image
  • ⁇ x is the variance of the pixel values of all pixels in the generated image
  • ⁇ y is the variance of the pixel values of all pixels in the second image
  • ⁇ xy is the covariance between the generated image and the second image.
  • the value of the content bidirectional loss function is calculated by the CoBi loss function based on the VGG feature, and the CoBi loss function based on the VGG feature extracts several sets of VGG features of the generated image and the second image respectively, and is specific to the generated image
  • search for a second VGG feature match close to the first VGG feature in the second VGG feature of the second image, and finally calculate the distance sum between each first VGG feature and its matching second VGG feature .
  • the content two-way loss function value so that the two-sided distance is searched through the content two-way loss function, considering the spatial loss of the first VGG feature and its matching second VGG feature, so that the first image and the second image can be avoided
  • the impact of incomplete alignment improves the speed and accuracy of the preset network model training.
  • the content two-way loss function value is determined according to the distance and position relationship between the first VGG feature and the second VGG feature, which improves the accuracy of matching. Thereby, the influence of the misalignment of the first image and the second image on the training of the preset network model is further reduced.
  • the expression of the content bidirectional loss function may be:
  • D is the cosine distance between the VVG feature of the generated image and the VVG feature of the second image
  • D′ is the spatial position distance between the VVG feature of the generated image and the VVG feature of the second image
  • N is the VVG of the generated image
  • the present invention also provides an image processing method that applies the image processing model generation method described in the foregoing embodiment to train to obtain an image processing model, as shown in FIG. 24,
  • the image processing methods include:
  • the image to be processed may be an image captured by an off-screen imaging system, or a preset image, or an image determined according to a received selection operation.
  • the image to be processed is preferably an image captured by an under-screen imaging system, for example, the image to be processed is an image of a person captured by a mobile phone equipped with an under-screen imaging system.
  • E200 Perform de-ghosting processing on the image to be processed through the image processing model to obtain an output image corresponding to the image to be processed.
  • the de-ghosting of the image to be processed by the image processing model refers to inputting the image to be processed into the image processing model as an input item of the image processing model, and passing through the image processing model.
  • the image processing model removes ghosting of the image to be processed to obtain an output image, where the output image is an image obtained by de-ghosting the image to be processed.
  • the image to be processed is an image with ghosting corresponding to the output image, that is, the output image corresponds to the image to be processed, and they present the same image scene, the output image is the image that is normally displayed, the image of the image to be processed
  • the content corresponds to the output image, but the object in the image content to be processed has ghosting or a blur effect similar to ghosting.
  • the image to be processed as shown in FIG. 25 is subjected to the de-ghosting process to obtain the output image as shown in FIG. 26.
  • the image processing model includes an encoder and a decoder, so that when processing the image to be processed through the image processing model, the encoder and the decoder need to be processed respectively.
  • the de-ghosting the image to be processed by the image processing model to obtain the output image corresponding to the image to be processed specifically includes:
  • E201 Input the image to be processed into the encoder, and obtain a characteristic image of the image to be processed through the encoder, wherein the image size of the characteristic image is smaller than the image size of the image to be processed;
  • E202 Input the characteristic image to the decoder, and output an output image corresponding to the image to be processed through the decoder, wherein the image size of the output image is equal to the image size of the image to be processed.
  • the encoder converts the input image to be processed into a feature image with an image space size smaller than the input image and more channels than the input image, and the feature image is input to the decoder, and the decoder converts the input image to a feature image.
  • the feature image is converted into a generated image with the same image size as the image to be processed.
  • the structure of the encoder is the same as the structure of the encoder in the preset network model. For details, reference may be made to the description of the structure of the encoder in the preset network model.
  • the processing of the image to be processed by the encoder of the image processing model is the same as the processing of the first image by the encoder in the preset network model, so the specific execution process of step E201 can refer to step N11.
  • the structure of the decoder is the same as the structure of the decoder in the preset network model.
  • the processing of the characteristic image corresponding to the image to be processed by the decoder of the image processing model is the same as the processing of the characteristic image corresponding to the first image by the decoder in the preset network model, so that the specific execution process of step E202 can be Refer to step N12.
  • the network structure corresponding to the image processing model in the training process is the same as the corresponding network structure in the application process (removal of ghost images carried by the output image).
  • the image processing model includes an encoder and an encoder, and accordingly, when the ghost image carried by the output image is removed by the image processing model, the image processing model also includes an encoder and an encoder.
  • the encoder of the image processing model includes the encoder including a first redundant learning layer and a down-sampling layer, and the decoder includes an up-sampling layer and a second redundant learning layer; accordingly, When removing ghost images carried by the output image through the image processing model, the encoder may also include a first redundant learning layer and a down-sampling layer, and the decoder may include an up-sampling layer and a second redundant learning layer; and in the application process, The working principle of each layer is the same as the working principle of each layer in the training process. Therefore, the input and output of each layer of neural network in the image processing model application process can be referred to the relevant introduction in the image processing model training process. I won't repeat it here.
  • post-processing may be performed on the output image, where the post-processing may include sharpening processing and noise reduction processing, etc. .
  • the method further includes:
  • the sharpening process refers to compensating the contour of the output image, enhancing the edge of the output image and the part where the gray level jumps, so as to improve the image quality of the output image.
  • the sharpening processing may adopt an existing sharpening processing method, for example, a high-pass filtering method.
  • the noise reduction processing refers to removing noise in the image and improving the signal-to-noise ratio of the image.
  • the noise reduction processing may adopt an existing noise reduction algorithm or a trained noise reduction network model, etc., for example, the noise reduction processing adopts a Gaussian low-pass filtering method or the like.
  • this embodiment provides an image processing model generation device, wherein the image processing model generation device includes:
  • the second generation module 301 is configured to use a preset network model to generate a generated image corresponding to the first image according to the first image in the training image set; wherein, the training image set includes multiple sets of training image groups, each of which is trained The image group includes a first image and a second image, and the first image is an image with ghosting corresponding to the second image;
  • the second correction module 302 is configured to use the preset network model to correct the model parameters of the preset network model according to the second image corresponding to the first image and the generated image corresponding to the first image, And continue to execute the step of generating the generated image corresponding to the first image according to the first image in the next training image group in the training image set, until the training condition of the preset network model meets the preset conditions, so as to obtain Trained image processing model.
  • the preset network model includes an encoder and a decoder; the second generation module is specifically configured to:
  • the first image in the training image set is input to the encoder, and the characteristic image of the first image is obtained through the encoder; and the characteristic image is input to the decoder, and the decoder outputs the An image is generated, wherein the image size of the characteristic image is smaller than the image size of the first image; the image size of the generated image is equal to the image size of the first image.
  • the second correction module is specifically configured to:
  • the first image is generated based on the second image and a point spread function, wherein the point spread function is generated based on a grayscale image generated by a light shielding structure in an under-screen imaging system.
  • the first image is an image captured by an off-screen imaging system.
  • the under-screen imaging system is an under-screen camera.
  • the device for generating the image processing model further includes:
  • the second alignment module is configured to align the first image in the training image group with the second image corresponding to the first image for each training image group in the training image set, to obtain the first image corresponding to the first image in the training image group.
  • An alignment image that is aligned with the two images, and the alignment image is used as the first image.
  • the second alignment module is specifically configured to:
  • the pixel deviation amount between the first image in the training image group and the second image corresponding to the first image is obtained; the pixel deviation amount is determined according to the pixel deviation amount.
  • the second alignment module is specifically configured to:
  • the first pixel point set of the first image and the second pixel point set of the second image are extracted, and the first pixel point set includes all A number of first pixels in the first image
  • the second pixel point set includes a number of second pixels in the second image, the second pixel point in the second pixel point set and the first pixel point
  • One-to-one correspondence between the first pixel in the pixel set for each first pixel in the first pixel set, the coordinate difference between the first pixel and its corresponding second pixel is calculated, and according to the first pixel
  • the coordinate difference value corresponding to the pixel point adjusts the position of the first pixel point to align the first pixel point with the second pixel point corresponding to the first pixel point.
  • this embodiment provides an image processing device, the image processing device includes:
  • the second acquisition module 401 is configured to acquire an image to be processed, and input the image to be processed into the image processing model;
  • the second processing module 402 is configured to perform de-ghosting processing on the image to be processed through the image processing model to obtain an output image corresponding to the image to be processed.
  • the image processing model includes an encoder and a decoder; the second processing module specifically includes:
  • Input the image to be processed into the encoder obtain a characteristic image of the image to be processed through the encoder; and input the characteristic image into the decoder, and output the image to be processed through the decoder
  • the corresponding output image wherein the image size of the characteristic image is smaller than the image size of the image to be processed; the image size of the output image is equal to the image size of the image to be processed.
  • the image processing device further includes:
  • the sharpening module is used to perform sharpening and noise reduction processing on the output image, and use the output image after the sharpening and noise reduction processing as the output image corresponding to the image to be processed.
  • This implementation provides an image processing method. As shown in FIG. 27, the method includes:
  • the to-be-processed image set includes at least two images, and the manner of acquiring each image in the to-be-processed image set may include: captured by an imaging system (e.g., under-screen camera, etc.), an external device (e.g., , Smart phones, etc.), and through the Internet (such as Baidu, etc.).
  • each image included in the image set to be processed is a low-exposure image
  • each denoising image in the image set to be processed is passed through an imaging system (eg, camera, video camera, under-screen camera).
  • Etc. obtained by shooting
  • each denoising image belongs to the same color space (for example, RGB color space and YUV color space, etc.).
  • the denoising images are all captured by an under-screen camera, and the basic image and each adjacent image belong to the RGB color space.
  • the shooting scenes corresponding to the images to be processed in the image set to be processed are all the same, and the shooting parameters of the images to be processed may also be the same, where the shooting parameters may include environmental illumination and exposure parameters, where the Exposure parameters can include aperture, door opening speed, sensitivity, focus, and white balance.
  • the shooting parameters may also be shooting angles and shooting ranges.
  • the image captured by the imaging system has different levels of noise. For example, when the environmental illuminance is low, the image captured by the imaging system carries more noise. When the environmental illuminance is high, the imaging system captures more noise. The image carries less noise.
  • the display panel has different absorption intensity for different light intensities, and the absorption degree of the display panel and the light intensity are nonlinear light (for example, when the ambient illuminance is low, the light intensity is low, the display panel The ratio of light absorption is high.
  • the number of denoising images included in the image set to be processed may be determined according to shooting parameters corresponding to the image set to be processed, wherein the shooting parameters at least include environmental illuminance.
  • the corresponding relationship between the environmental illuminance interval and the number of images of the image to be processed may be preset. After acquiring the environmental illuminance, first determine the environmental illuminance interval in which the environmental illuminance is located, and determine the image quantity of the image to be processed corresponding to the environmental illuminance interval according to the corresponding relationship to obtain the image quantity of the image to be processed.
  • the corresponding relationship between the environmental illuminance interval and the number of images of the image to be processed is: when the environmental illuminance interval is [0.5, 1), the number of images of the image to be processed corresponds to 8; when the environmental illuminance is [1, 3) , The number of images to be processed corresponds to 7; when the environmental illuminance is [3,10), the number of images to be processed corresponds to 6; when the environmental illuminance interval is [10,75), the number of images to be processed corresponds to 5 ; When the environmental illuminance interval is [75,300), the number of images to be processed corresponds to 4, when the environmental illuminance interval is [300,1000), the number of images to be processed corresponds to 3; when the environmental illuminance is [1000,5000) When the number of images to be processed corresponds to 2.
  • the image set to be processed is captured by an under-screen imaging system, and the number of images to be processed included in the image set to be processed is captured by the under-screen imaging system.
  • the ambient illuminance of the image is determined.
  • the environmental illuminance can be obtained when the off-screen imaging system is started, or it can be obtained according to the first frame image obtained by shooting, or it can be a preset number of images obtained by shooting in advance, and then obtained according to the shooting. Any one of the preset number of images is determined.
  • the environmental illuminance is acquired when the off-screen imaging system is started.
  • the process of acquiring the image set to be processed may be: when the off-screen imaging system is started, the environmental illuminance is acquired, and the first image number of the images contained in the image set to be processed is determined according to the acquired environmental illuminance, and the number of images contained in the image set to be processed is determined through the screen
  • the lower imaging system continuously acquires images of the first number of images to obtain the to-be-processed image set.
  • the number of the first images may be determined according to the corresponding relationship between the preset environmental illuminance and the number of images of the image to be processed.
  • the environmental illuminance is obtained according to the first frame of image obtained by shooting.
  • the acquisition process of the image set to be processed may be: first acquire the first frame of image through the off-screen imaging system, then acquire the ISO value of the first frame of image, and determine the corresponding image of the first frame of image according to the ISO value.
  • the environmental illuminance is determined by a preset number of images obtained in advance, and then determined according to any one of the preset number of images obtained by shooting.
  • the acquisition process of the image set to be processed may be: first acquire a preset number of images through the off-screen imaging system in advance, and randomly select a third preset image from the acquired images, and acquire the ISO of the third preset image. According to the ISO value, the environmental illuminance corresponding to the third preset image is determined, and finally the number of images contained in the image set to be processed (ie, the third image number) is determined according to the acquired environmental illuminance. In addition, since the preset number of images has been acquired, the preset number can be compared with the third image number.
  • the fourth image is continuously acquired through the under-screen imaging system.
  • the third preset image may be added to the to-be-processed image set. Process the image set, and then select the third image number minus one image from the acquired images. At the same time, in order to make the images in the to-be-processed image set continuous with the third preset image, the images in the to-be-processed image set may be selected according to the order of taking pictures.
  • the preset number is 5, the 5 images are recorded as image A, image B, image C, image D, and image E in the order of shooting.
  • the third image number is 3, and the third preset image is based on the shooting time.
  • Image C in the order of 3 then the selected images according to the shooting order are image B and image D, respectively, so that the image set to be processed includes image B, image C, and image D.
  • the number of images before the third preset image is not enough, follow the shooting order from the third preset image to the back Selection; it can also be selected backward first, and when the number of backward images is not enough, forward selection; it can also be other selection methods, there are no specific restrictions here, as long as it is possible to select images up to the fourth number of images .
  • H20 Generate a denoising image corresponding to the image set to be processed according to the image set to be processed.
  • the set of images to be processed includes a basic image and at least one adjacent image, wherein the basic image is an image reference of each image to be processed in the image set to be processed, and each adjacent image may use the basic image as a reference.
  • the reference is synthesized with the basic image. Therefore, before generating a denoising image based on the image set to be processed, an image in the image set to be processed needs to be selected as the base image, and all images in the image set to be processed except the base image are used as neighboring images of the base image .
  • the basic image needs to be selected from the acquired images.
  • the basic image may be the first image in the acquisition order, or any one of the images in the set of images to be processed, or the image with the highest definition in the set of images to be processed.
  • the basic image is a picture with the highest definition in the processed image set, that is, the definition of the basic image is greater than or equal to the definition of any adjacent image.
  • the process of determining the basic image may be: after acquiring all the images included in the image set to be processed, acquiring the definition of each image, and comparing the acquired definitions , To select the image with the highest definition, and use the selected image as the base image.
  • the clarity of the image can be understood as the difference between the pixel value of the pixel on the feature boundary (or object boundary) in the image and the pixel value of the pixel adjacent to the feature boundary (or object boundary). Difference; it is understandable that if the pixel value of the pixel on the object boundary (or object boundary) in the image is the difference between the pixel value of the pixel adjacent to the object boundary (or object boundary) The larger the value, the higher the definition of the image.
  • the definition of the basic image is higher than the definition of each adjacent image. It can be understood that for each adjacent image, the sum of the pixel values of the pixel points on the object boundary (or object boundary) in the basic image The difference between the pixel values of pixels adjacent to the feature boundary (or object boundary) is greater than the pixel value of the pixel on the feature boundary (or object boundary) in the adjacent image and the pixel value of the feature boundary ( (Or object boundary) the difference between the pixel values of adjacent pixels.
  • the image set to be processed includes image A and a piece of image B, and the image content in image A and image B are exactly the same, where both image A and image B include pixel point a and pixel point b , Pixel a is the pixel on the feature boundary (or object boundary) in the image, and pixel b is the pixel adjacent to the feature boundary (or object boundary); if the pixel of pixel a in image A
  • the difference between the value and the pixel value of pixel b is 10, and the difference between the pixel value of pixel a and the pixel value of pixel b in the image B is 30, then the image B can be considered as clear
  • the degree is higher than the sharpness of the training image A. Therefore, the image A can be used as the basic image in the to-be-processed image set, and the image B can be used as the adjacent image in the to-be-processed image set.
  • the basic image when the basic image is selected from the image set to be processed according to the sharpness, there are multiple images with the same sharpness (denoted as image C) in the image set to be processed, and each image C The sharpness of is not less than the sharpness of any image in the image set to be processed, so the multiple images C can all be used as basic images.
  • one of the multiple images C can be randomly selected as the basic image, or according to the shooting order, the first image C among the multiple images C can be selected as the basic image. In order, the last image C among multiple images C is selected as the base image.
  • the generating a denoising image corresponding to the to-be-processed image set according to the to-be-processed image set specifically includes:
  • the weight parameter set corresponding to the basic image block includes a first weight parameter and a second weight parameter, the first weight parameter is the weight parameter of the basic image block, and the second weight parameter
  • the parameter is the weight parameter of the adjacent image block corresponding to the basic image block in the adjacent image;
  • H23 Determine the denoised image according to the image set to be processed and the weight parameter set corresponding to each basic image block.
  • the basic image block is a partial image area of the basic image, and the basic image will be formed after the several basic image blocks are spliced.
  • the dividing a basic image into a number of basic image blocks refers to taking the basic image as an area, and dividing the area into a number of sub-areas, and the image area corresponding to each sub-area is a basic image block.
  • the division of the area into a plurality of sub-areas may be equal to dividing the area into a plurality of areas.
  • the 8*8 basic image can be divided into 4 4*4 basic image blocks.
  • the method of dividing a basic image into several basic image blocks in this embodiment can be flexibly selected according to a specific scene, as long as several basic image blocks can be obtained by dividing the method.
  • the adjacent image block is the image block corresponding to the basic image in the adjacent image
  • the image block size of the adjacent image block is the same as the size of the basic image block corresponding to the adjacent image block
  • the adjacent image block carries the image
  • the content is the same as the image content carried by the basic image block.
  • Said determining that the basic image block corresponds to the adjacent image block in each adjacent image refers to selecting the image block with the highest similarity to the basic image block in the designated area of the adjacent image block, wherein the designated area is based on The basic image block is determined by the area in the basic image.
  • the respectively determining the neighboring image blocks corresponding to each basic image in each neighboring image specifically includes:
  • A10. Determine the area range of the basic image block in the basic image, and determine the designated area in the adjacent image according to the area range;
  • A20 Select adjacent image blocks in a designated area according to the basic image block, where the adjacent image block is the image block with the highest similarity to the basic image block in the designated area, and the image size of the adjacent image block is the same as that of the basic image block.
  • the image size is equal.
  • the area range refers to the set of coordinate points formed by the pixel coordinates of the boundary pixels of the area where the basic image block is located in the basic image.
  • the basic image block is a square area in the basic image
  • the basic image block is a square area in the basic image.
  • the coordinate points of the four vertices of the image block are respectively (10,10), (10,20), (20,10) and (20,20), then the area range corresponding to the basic image block can be ⁇ (10 ,10), (10,20), (20,10) and (20,20) ⁇ .
  • the specified area is an image area in the adjacent image
  • the area range corresponding to the basic image block may correspond to the area range corresponding to the specified area, that is, when the adjacent image is mapped to the basic image, the area range corresponding to the specified area is the same as the basic image
  • the area range corresponding to the block corresponds.
  • the area range corresponding to the basic image block can be ⁇ (10,10), (10,20), (20,10) and (20,20) ⁇ .
  • the specified area corresponds to The area range of can be ⁇ (10,10), (10,20), (20,10) and (20,20) ⁇ , then the area range corresponding to the basic image block can correspond to the area range corresponding to the specified area.
  • the area range corresponding to the basic image block can also correspond to the area range corresponding to the sub-area of the designated area. That is, when the neighboring image is mapped to the base image, there is a sub-area in the area range corresponding to the designated area.
  • the area range of corresponds to the area range corresponding to the basic image block.
  • the area range corresponding to the image area occupied by the basic image block in the basic image can be ⁇ (10,10), (10,20), (20,10) and (20,20) ⁇ , as shown in Figure 30
  • the area range corresponding to the image area 12 occupied by the specified area in the adjacent image can be ⁇ (9,9), (9,21), (21,9) and (21,21) ⁇
  • the specified area contains Sub-region 11
  • the area range of the sub-region 11 is ⁇ (10,10), (10,20), (20,10) and (20,20) ⁇
  • the area range corresponding to the sub-region 11 is the same as the basic image block Corresponding to the corresponding area.
  • the area range corresponding to the basic image block may also correspond to the area range corresponding to the sub-areas of the designated area, and the designated area is a coordinate point corresponding to the area range.
  • Each coordinate point in the set is obtained by shifting a preset value along the horizontal, vertical or vertical axis away from the area range, where the area range is the area range corresponding to the basic image block.
  • the area range corresponding to the basic image block can be ⁇ (10,10), (10,20), (20,10) and (20,20) ⁇ , the preset value is 5, then the area range of the specified area is ⁇ (5,5), (5,25), (25,5) and (25,25) ⁇ .
  • the preset values corresponding to different neighboring images may be different, and the preset values corresponding to each neighboring image may be determined according to the displacement of the neighboring image relative to the base image.
  • the process of determining the preset value may be: for each adjacent image, calculating the basic image and the projection of the adjacent image in the row and column directions respectively, and according to the projection corresponding to the adjacent image and the projection corresponding to the basic image, Determine the displacement of the adjacent image relative to the basic image in the rows and columns, and use the displacement as the preset value corresponding to the adjacent image, where the displacement can be calculated by using the SAD algorithm.
  • the number of the second weight parameter in the weight parameter set is the same as the number of adjacent images in the image set to be processed, and the second weight parameter in the weight parameter set is one-to-one with the adjacent images in the image set to be processed correspond.
  • Each adjacent image includes at least one adjacent image block corresponding to the basic image block, and each adjacent image block corresponding to the basic image block has a second weight parameter.
  • the weight parameter set includes one first weight parameter and at least one second weight parameter, and each second weight parameter corresponds to a neighboring image block corresponding to the basic image block in the neighboring image.
  • the first weight parameter may be preset to indicate the degree of similarity between the basic image block and itself; the second weight parameter is obtained based on the basic image block and its corresponding neighboring image.
  • the determining the weight parameter set corresponding to each basic image block specifically includes:
  • For each basic image block determine the second weight parameter of each adjacent image block corresponding to the basic image block, and obtain the first weight parameter corresponding to the basic image block to obtain the weight parameter set corresponding to the basic image block.
  • each basic image block corresponds to at least one adjacent image block, wherein the number of adjacent image blocks corresponding to the basic image is equal to the number of adjacent images corresponding to the basic image.
  • the adjacent image block corresponds to a second weight parameter, so that the number of second weight parameters corresponding to the basic image block is equal to the number of adjacent images in the image set to be processed.
  • the second weight parameter is calculated according to the similarity between the basic image block and the adjacent image block.
  • the determining the second weight parameter of each adjacent image block corresponding to the basic image block specifically includes:
  • the similarity refers to the similarity between the basic image block and the adjacent image block
  • the adjacent image block is determined in the adjacent image according to the basic image block
  • the image size of the adjacent image block is the same as the size of the basic image block.
  • each pixel point contained in the basic image block has a one-to-one correspondence with each pixel point contained in the adjacent image block.
  • the similarity can be calculated based on the pixel value of each pixel included in the basic image block and the pixel value of each pixel included in the adjacent image block.
  • the specific process of calculating the similarity based on the pixel value of each pixel contained in the basic image block and the pixel value of each pixel contained in the adjacent image block may be: reading the first corresponding pixel of each pixel contained in the basic image block.
  • the similarity value can be the mean value of the absolute value of each difference calculated, for example, the first pixel value A and the second pixel are calculated
  • the difference between the value A, and the difference between the first pixel value B and the second pixel value B can be based on the difference between the first pixel value A and the second pixel value A, and the first pixel value B and The difference of the second pixel value B determines the similarity between the basic image block and the adjacent image block.
  • the similarity value may be the absolute value of the difference between the first pixel value A and the second pixel value A, which is compared with the first pixel value A and the second pixel value A.
  • the similarity value is related to the image noise intensity of the image in the image set to be processed, and the difference between the image content of the basic image block and the image content of adjacent image blocks, specifically .
  • the similarity value is large; on the contrary, when the image noise intensity is low and the image content of the basic image block is similar to the image content of the adjacent image block When the content difference is small, the similarity value is small.
  • the second weight parameter is negatively related to the similarity value, that is, the greater the similarity value, the smaller the second weight parameter; conversely, the smaller the similarity value, the second weight parameter Bigger.
  • the calculating the second weight parameter of the adjacent image block according to the similarity value specifically includes:
  • B20 may only include any one, any two steps, or all of C10, C20, and C30. That is, in this embodiment, B20 may include C10 and/or C20 and/ Or C30.
  • the first threshold and the second threshold are both used to measure the similarity between the basic image block and the adjacent image block.
  • the second threshold is greater than the first threshold.
  • the similarity value is less than the first threshold.
  • the similarity value is greater than the second threshold, according to The relationship between the similarity value and the similarity can be known that the similarity between the basic image block and the adjacent image block is low, so that the second weight parameter value corresponding to the adjacent image block is small.
  • the first preset parameter is greater than the second preset parameter, and the calculation of the adjacent image block is calculated according to the similarity value, the first threshold, and the second threshold, and the third parameter is located in the first preset. Set between the parameter and the second preset parameter.
  • the calculation process of the third parameter may be: first calculate the first difference between the similarity value and the second threshold, and then calculate the second difference between the first threshold and the second threshold. Then, the ratio of the first difference and the second difference is calculated, and the ratio is used as the second weight parameter of the adjacent image block.
  • the preset parameter is set to 1, and the second preset parameter is set to 0. Therefore, the corresponding relationship between the second weight parameter and the similarity value can be expressed as follows:
  • w i is the weight of the second parameter
  • t 1 is a first threshold
  • t 2 is the second threshold value
  • N is the number of basic image blocks.
  • the similarity is positively correlated with the weight coefficient, that is, the higher the similarity between the basic image and the adjacent image, the larger the weight coefficient corresponding to the adjacent image block.
  • the lower the similarity between the basic image and the adjacent image the adjacent image
  • the weight coefficient corresponding to the image block is lower.
  • the comparison object for the basic image block to determine the similarity is the basic image block itself, then the similarity between the basic image block and itself is greater than or equal to the similarity between the adjacent image block and the basic image block.
  • a weight parameter is greater than or equal to the second weight coefficient.
  • the second weight coefficient is a maximum of 1, so in an implementation of this embodiment, the first weight coefficient corresponding to the basic image block may be equal to the second weight parameter
  • the maximum value of, that is, the first weight parameter is 1.
  • the first threshold and the second threshold may be preset, or may be determined according to the similarity value of the neighboring image blocks corresponding to the basic image block.
  • the first threshold and the second threshold are determined according to the similarity value of the basic image block corresponding to each adjacent image block.
  • the process of determining the first threshold and the second threshold may be: obtaining the similarity values of each adjacent image block, respectively calculating the mean value and standard deviation of each similarity value, and then calculating the first threshold and the first threshold according to the mean and standard deviation. Two thresholds.
  • the first threshold and the second threshold are determined by the similarity values of adjacent image blocks, so that the first threshold and the second threshold can be adaptively adjusted according to the similarity values of the adjacent images, so that the first threshold and the The second threshold is adaptively adjusted according to the noise intensity of each neighboring image, avoiding poor image denoising effects caused by excessively large first and second thresholds, and image blurring caused by excessively small first and second thresholds , Thus improving the clarity of the image on the basis of ensuring the denoising effect of the image.
  • the calculation formulas for the first threshold t 1 and the second threshold t 2 are:
  • S min and S max are constants
  • d max is a constant
  • the accuracy of the selection of adjacent image blocks can cause a large change in the similarity value.
  • the similarity value between the block and the basic image block is greater than the preset value d max , the default is that the image content of the basic image block is too different from the image content of the adjacent image block, and the adjacent image block is regarded as an invalid adjacent image block (that is, discarded) For the invalid adjacent image block, the invalid adjacent image block is not used as the adjacent image block of the basic image block).
  • adjacent image blocks d i ⁇ d max it can be considered the basis of the image content differences in image content and the adjacent image blocks of the image block is too large, thereby eliminating the need to determine the proximity of the image block corresponding to a first threshold and the second threshold value, The calculation speed of the weight parameter set corresponding to the basic image block is improved.
  • adjacent image blocks with a large difference in image content from the basic image block can avoid the problem of smear caused by adjacent image blocks with large image content differences during image fusion, resulting in output image distortion.
  • the output image is formed by stitching a plurality of output image blocks, and the output image block is based on the basic image block, the adjacent image block corresponding to the basic image block, and the weight parameter set corresponding to the basic image block.
  • the output image block is based on the basic image block, the adjacent image block corresponding to the basic image block, and the weight parameter set corresponding to the basic image block.
  • the basic image block and its corresponding neighboring image blocks can be weighted for each basic image block to obtain the The output image block corresponding to the basic image block.
  • the weighting coefficient of the basic image block is the first weighting system in the weight parameter set
  • the weighting coefficient of each adjacent image block is the weighting parameter
  • the second weight parameter corresponding to each adjacent image block is collected.
  • the output image is generated according to each output image block obtained by calculation, wherein generating an output image according to the output image block may be to replace the corresponding basic image in the basic image with each output image block Alternatively, the output image blocks may be spliced together to obtain the output image.
  • the basic image block corresponds to four adjacent image blocks, which are respectively recorded as the first adjacent image block and the second adjacent image block.
  • Image block, third adjacent image block, and fourth adjacent image block and in the order of shooting: basic image block, first adjacent image block, second adjacent image block, third adjacent image block, and fourth adjacent image block ;
  • the pixel value of the pixel is A, and the pixel value corresponding to the pixel in the first adjacent image block
  • the pixel value is B
  • the pixel value of the pixel corresponding to the pixel in the second adjacent image block is C
  • the pixel value of the pixel corresponding to the pixel in the third adjacent image block is D
  • the fourth adjacent image block is
  • the pixel value of the pixel point corresponding to the pixel point is E
  • the first weight parameter corresponding to the basic image block is a
  • the second weight parameter is c
  • the second weight parameter corresponding to the third adjacent image block is d
  • the second weight parameter corresponding to the fourth adjacent image block is e
  • the pixel value of the output pixel corresponding to this pixel (A* a+B*b+C*c+D*d+E*e)/5.
  • the denoising image is generated based on the image set to be processed, and the first image processing model may be pre-trained by an image device that processes the denoising image (for example, a mobile phone equipped with an under-screen camera), or it may be After being trained by others, the file corresponding to the first image processing model is transplanted to the image device.
  • the image device may use the first image processing model as an image processing function module. When the image device obtains a denoised image, the image processing function module is activated and the denoised image is input to the first image processing model. .
  • the first image processing model is obtained by training based on a training image set, and the training process of the first image processing model may be:
  • the first preset network model generates a generated image corresponding to the first image according to the first image in the training image set, wherein the training image set includes multiple training image groups, and each training image group includes the first An image and a second image, where the first image is a color cast image corresponding to the second image;
  • the first preset network model corrects the model parameters according to the second image corresponding to the first image and the generated image corresponding to the first image, and continues to execute according to the next image in the training image set.
  • the first image in the training image group is generated, and the step of generating an image corresponding to the first image is generated until the training condition of the first preset network model meets a preset condition, so as to obtain the image processing model.
  • the first preset network model may adopt a deep network learning model
  • the training image set includes multiple sets of training image groups with different image content
  • each training image group includes The first image and the second image
  • the first image is a color cast image corresponding to the second image.
  • that the first image is a color cast image corresponding to the second image means that the first image corresponds to the second image, the first image and the second image present the same image scene, and the first image meets the requirements It is assumed that the number of the first target pixel of the color cast condition satisfies the preset number condition.
  • the second image is a normal display image, and there are a number of first target pixels that meet a preset color cast condition in the first image, and the number of the first target pixels meets the preset condition.
  • the second image is the image shown in FIG. 6, and the first image is the image shown in FIG. 5.
  • the image content of the first image is the same as the image content of the second image, but the apple The corresponding rendered color is different from the color of the apple in the second image.
  • the color of the apple in the first image in the first image is green to blue; in Figure 9, the second image The color of the middle apple in the second image is dark green.
  • the preset color cast condition is that the error between the display parameter of the first target pixel in the first image and the display parameter of the second target pixel in the second image satisfies a preset error condition, and the first target There is a one-to-one correspondence between the pixel points and the second target pixel point.
  • the display parameter is a parameter for reflecting the color corresponding to the pixel.
  • the display parameter may be the RGB value of the pixel, where R is the red channel value, G is the green channel value, and the B value is It is the blue channel value; it can also be the hsl value of the pixel, where h is the hue value, l is the brightness value, and s is the saturation value.
  • the display parameter is the RGB value of the pixel
  • the display parameter of any pixel in the first image and the second image includes three display parameters of R value, G value and B value;
  • the display is displayed as a pixel hls value
  • the display parameters of any pixel in the first image and the second image include three display parameters of h value, l value and s value.
  • the preset error condition is used to measure whether the first target pixel is a pixel that meets a preset color cast condition, wherein the preset error condition is a preset error threshold, and an error that meets the preset error condition is an error greater than or Equal to the preset error threshold.
  • the display parameter includes several display parameters, for example, the display parameter is the RGB value of the pixel.
  • the display parameter includes three display parameters of R value, G value and B value. When the display parameter is the hsl value of the pixel, the display parameter Including three display parameters of h value, l value and s value.
  • the error may be the maximum value of the error of each display parameter in the display parameter, or the minimum value of the error of each display parameter in the display parameter, or the average value of the errors of all the display parameters.
  • the display parameter is the RGB value of the pixel.
  • the display parameter of the first target pixel is (55, 86, 108), and the display parameter of the second target pixel is (58, 95, 120).
  • the display parameters of each display parameter The error value is divided into 3, 9 and 12; therefore, when the error between the first target pixel and the second target pixel is the maximum error of each display parameter, the error is 12; when the first target pixel and the second target pixel When the error of the second target pixel is the minimum error of each display parameter, the error is 3; when the error of the first target pixel and the second target pixel is the average of the errors of all the display parameters, the error is 8; It should be noted that in a possible implementation manner, it is also possible to refer to only one parameter (such as R, G, or B) in RGB or the error of any two parameters. When the display parameter is the hsl value of the pixel, the same reason.
  • the second target pixel used to calculate the error with the first target pixel and the first target display point there is a one-to-one correspondence between the second target pixel used to calculate the error with the first target pixel and the first target display point. It is understandable that for the first target pixel, there is a unique second target pixel in the second image corresponding to the first target pixel, where the first target pixel and the second target pixel correspond to the first target pixel.
  • the pixel position of a target pixel in the first image corresponds to the pixel position of the second target pixel in the second image. For example, the pixel position of the first target pixel in the first image is (5, 6), and the pixel position of the second target pixel in the second image is (5, 6).
  • the first target pixel point may be any pixel point in the first image, or any pixel point in the target area in the first image, where the target area may be the item in the first image.
  • Area, where the area where the object is located may be a corresponding area of the person or object in the image.
  • the target area is the area where the apple is located in the first image. That is to say, all pixels in the first image can be compared with the second image to have a color cast, that is, all pixels in the first image are the first target pixels, or only a part of the pixels can be compared with the second image Color cast appears, that is, some of the pixels in the first image are the first target pixels.
  • the image can also be understood as a color cast image corresponding to the second image, that is, the first image.
  • first image and the second image correspond to each other means that the image size of the first image is equal to the image size of the second image, and the first image and the second image correspond to the same image scene.
  • the first image and the second image corresponding to the same image scene means that the similarity between the image content carried by the first image and the image content carried by the second image reaches a preset threshold, and the image size of the first image is equal to that of the first image.
  • the image sizes of the two images are the same, so that when the first image and the second image overlap, the coverage rate of the object carried by the first image to the corresponding object in the second image reaches the preset condition.
  • the preset threshold may be 99%
  • the preset condition may be 99.5% and so on.
  • the first image may be captured by an off-screen imaging system; the second image may be captured by a normal on-screen imaging system (e.g., on-screen camera), or through a network (e.g., , Baidu), it can also be sent through other external devices (such as smart phones).
  • a normal on-screen imaging system e.g., on-screen camera
  • a network e.g., , Baidu
  • the second image is obtained by shooting through a normal on-screen imaging system, and shooting parameters of the second image and the first image are the same.
  • the shooting parameters may include exposure parameters of the imaging system, and the exposure parameters may include aperture, shutter speed, sensitivity, focus, white balance, and the like.
  • the shooting parameters may also include ambient light, shooting angle, and shooting range.
  • the first image is an image obtained by shooting a scene through an on-screen camera as shown in FIG. 5
  • the second image is an image obtained by shooting the scene through an on-screen camera as shown in FIG.
  • the image content of the first image and the image content of the second image It can be exactly the same. That is, the first image and the second image having the same image content means that the object content of the first image is the same as the object content of the second image, and the image size of the first image is the same as the image size of the second image. And when the first image and the second image overlap, the objects in the first image can cover the corresponding objects in the second image.
  • the image size of the first image is 400*400
  • the image content of the first image is a circle
  • the position of the center of the circle in the first image in the first image is (200, 200), the radius length It is 50 pixels.
  • the image size of the second image is 400*400
  • the image content of the second image is also a circle
  • the position of the center of the circle in the second image in the second image is (200, 200), and the radius is 50 Pixels
  • the on-screen imaging system may be caused when the imaging system is replaced.
  • the change in the shooting angle and/or shooting position of the imaging system under the screen causes the first image and the second image to be spatially misaligned. Therefore, in a possible implementation of this embodiment, when the second image is captured by the on-screen imaging system and the first image is captured by the off-screen imaging system, the on-screen imaging system and the off-screen imaging system can be set to the same On the fixed frame, the on-screen imaging system and the under-screen imaging system are arranged side by side on the fixed frame, and the on-screen imaging system and the under-screen imaging system are kept in contact.
  • the on-screen imaging system and the under-screen imaging system connect the on-screen imaging system and the under-screen imaging system to wireless settings (such as Bluetooth watches, etc.), and trigger the shutter of the on-screen imaging system and the under-screen imaging system through the wireless settings, which can reduce the on-screen imaging during shooting.
  • the position change of the imaging system and the imaging system under the screen improves the spatial alignment of the first image and the second image.
  • the shooting time and shooting range of the on-screen imaging system and the under-screen imaging system are the same.
  • the shooting position, shooting angle, shooting time, and exposure coefficient of the off-screen imaging system and the on-screen imaging system can be fixed.
  • the first image captured by the off-screen imaging system and the second image captured by the on-screen imaging system may still be spatially misaligned. .
  • the method further includes:
  • each training image group in the training image set refers to performing alignment processing on each training image group in the training image set
  • the alignment processing may be performed on each training image group after obtaining the training image set
  • alignment processing is performed on the training image group to obtain the aligned training image corresponding to the training image. Group, and then input the first image in the aligned training image group into the first preset network model.
  • the alignment process is performed on each training image group after the training image set is obtained, and after all the training image groups are aligned, the first image input in the training image set is executed. The operation of the first preset network model.
  • the first image is used as a reference image
  • the second image is used as a reference image
  • the alignment processing is based on the reference image.
  • the alignment processing of the reference image in the training image group with the reference image corresponding to the reference image is the reference image corresponding to the reference image in the training image group.
  • the image is aligned, and the alignment of the reference image in the training image group with the reference image corresponding to the reference image may be based on the base image, and the pixels in the reference image correspond to those in the base image.
  • the pixel points of the reference image can be aligned with the pixel points in the reference image to reach a preset value, for example, 99%.
  • the alignment of pixels in the reference image with their corresponding pixels in the reference image refers to: for reference pixels in the reference image and reference pixels in the reference image corresponding to the reference pixels, if the reference pixel is The corresponding pixel coordinates are the same as the pixel coordinates of the reference pixel, then the reference pixel is aligned with the reference pixel; if the pixel coordinate corresponding to the reference pixel is not the same as the pixel coordinate corresponding to the reference pixel, then the reference pixel and the reference pixel Point alignment.
  • the aligned image refers to an image obtained by performing alignment processing on a reference image, and each pixel in the aligned image has the same pixel coordinates as its corresponding pixel in the reference image.
  • the corresponding reference image is replaced with the aligned image to update the training image group, so that the reference image and the reference image in the updated training image group are spatially aligned.
  • the aligning the reference image in the training image group with the reference image corresponding to the reference image specifically includes:
  • F12 Determine the alignment mode corresponding to the reference image according to the pixel deviation amount, and perform alignment processing on the reference image and the reference image by using the alignment mode.
  • the pixel deviation amount refers to the total number of reference pixels in the reference image that are not aligned with the reference pixels in the reference image corresponding to the reference pixels.
  • the pixel deviation amount can be obtained by obtaining the reference coordinates of each reference pixel in the reference image and the reference coordinates of each reference pixel in the reference image, and then comparing the reference coordinates of the reference pixel with the reference coordinates of the corresponding reference pixel , If the reference coordinate is the same as the reference coordinate, it is determined that the reference pixel is aligned with its corresponding reference pixel; if the reference coordinate is not the same, it is determined that the reference pixel is not aligned with its corresponding reference pixel, and finally all the misaligned points are obtained The total number of reference pixels to obtain the pixel deviation amount.
  • the reference pixel and the reference If the pixels are not aligned, the total number of misaligned reference pixels is increased by one; when the reference coordinate of the reference pixel in the reference image is (200, 200), the reference pixel in the reference image corresponds to the reference pixel When the reference coordinates are (200, 200), the reference pixels are aligned with the reference pixels, and the total number of unaligned reference pixels remains unchanged.
  • a deviation amount threshold may need to be set.
  • the pixel deviation amount of the reference image is obtained, the obtained pixel deviation amount can be compared with a preset deviation threshold value.
  • the alignment method corresponding to the reference image is determined according to the pixel deviation amount, and the alignment method is used to align the reference image with the reference image
  • the treatment specifically includes:
  • the pixel deviation amount is greater than the preset deviation threshold value, extract a reference pixel point set of the reference image and a reference pixel point set of the reference image, where the reference pixel point set includes the reference image
  • the reference pixel point set includes the reference image
  • the reference pixel point set includes a number of reference pixels in the reference image
  • the reference pixel points in the reference pixel point set correspond to the reference pixels in the reference pixel point set in a one-to-one correspondence;
  • For each reference pixel in the reference pixel set calculate the coordinate difference between the reference pixel and its corresponding reference pixel, and adjust the position of the reference pixel according to the coordinate difference corresponding to the reference pixel to The reference pixel point is aligned with the reference pixel point corresponding to the reference pixel point.
  • the preset deviation threshold is set in advance, for example, the preset deviation threshold is 20.
  • the pixel deviation amount is less than or equal to the preset deviation amount threshold value, it means that when the pixel deviation amount is less than or equal to the preset deviation amount threshold value, the pixel deviation amount is less than or equal to the preset deviation amount threshold value.
  • the pixel deviation is less than or equal to the preset deviation threshold, it means that the spatial deviation between the reference image and the reference image is small.
  • the reference image can be compared to the reference image based on the mutual information between the reference image and the reference image. Align with the reference image.
  • the process of aligning the reference image and the reference image with the mutual information between the reference image and its corresponding reference image may adopt an image registration method, in which the mutual information is used as a metric.
  • Criterion the metric criterion is iteratively optimized by the optimizer to obtain the alignment parameters, and the reference image is aligned with the reference image by the register for registering the alignment parameters, which guarantees the basis for the alignment effect of the reference image and the reference image , Which reduces the complexity of aligning the reference image with the reference image, thereby improving the alignment efficiency.
  • the optimizer mainly uses translation and rotation transformations to optimize the metric criteria through the translation and rotation transformations.
  • the pixel deviation amount is greater than the preset deviation amount threshold, indicating that the reference image and the reference image are spatially misaligned to a high degree, and at this time, it is necessary to focus on the alignment effect. Therefore, at this time, the reference image and the reference image can be aligned by selecting the reference pixel point set in the reference image and the reference pixel point set in the reference image.
  • the reference pixel points in the reference pixel point set correspond to the reference pixel points in the reference pixel point set in a one-to-one correspondence, so that for any reference pixel point in the reference pixel point set, a reference pixel point can be found in the reference pixel point set, so
  • the position of the reference pixel in the reference image corresponds to the position of the reference pixel in the reference image.
  • the reference pixel point set and the reference pixel point set may be determined based on the corresponding relationship between the reference pixel point and the reference pixel point after the reference pixel point set/reference pixel point set is obtained.
  • the reference pixel point set is generated by randomly selecting a plurality of reference pixels in the reference image, and the reference pixel point is determined according to each reference pixel point included in the reference pixel point set.
  • both the reference pixel point set and the reference pixel point set are obtained by means of scale-invariant feature transform (sift), that is, the reference pixel point set is the reference pixel point. It is the first sift feature point in the reference image, and the reference pixel point in the reference pixel point set is the second sift feature point of the reference image.
  • the calculated coordinate difference between the reference pixel point and its corresponding reference pixel point is the point-to-point matching of the first sift feature point in the reference pixel point and the second sift feature point in the reference pixel point set to obtain each first sift feature point.
  • the coordinate difference between the sift feature point and its corresponding second sift feature point, and the position transformation of the first sift feature point is performed according to the coordinate difference corresponding to the first sift feature point, so that the reference pixel point and the first sift feature point are transformed.
  • the second sift feature point corresponding to a sift feature point is aligned, so that the position of the first sift feature point in the reference image and the second sift feature point in the reference image are the same, thereby realizing the alignment of the reference image and the reference image.
  • the first preset network model includes a down-sampling module 100 and a transformation module 200.
  • the first preset network model Assuming that the network model generates the generated image corresponding to the first image according to the first image in the training image set, it may specifically include:
  • the two-sided grid 10 is a three-dimensional two-sided grid obtained by adding a one-dimensional dimension representing pixel intensity to the pixel coordinates of a two-dimensional image, wherein the three dimensions of the three-dimensional bilateral network are pixels of the two-dimensional image.
  • the guide image is obtained by performing pixel-level operations on the first image, and the resolution of the guide image 50 is the same as the resolution of the first image.
  • the guide image 50 corresponds to the first image Grayscale image.
  • the down-sampling module 100 since the down-sampling module 100 is used to output the bilateral grid 10 and the guide image 50 corresponding to the first image, the down-sampling module 100 includes a down-sampling unit 70 and a convolution unit 30, and the down-sampling unit 70 The bilateral grid 10 corresponding to the first image is output, and the convolution unit 30 is used to output a guide image 50 corresponding to the first image.
  • the first image in the training image set is input to the down-sampling module, and the bilateral grid corresponding to the first image is obtained through the down-sampling module
  • the parameters and the guidance image corresponding to the first image specifically include:
  • the down-sampling unit 70 is used to down-sample the first image to obtain a feature image corresponding to the first image, and generate a bilateral grid corresponding to the first image according to the feature image,
  • the number of spatial channels is greater than the number of spatial channels of the first image.
  • the bilateral grid is generated based on the local features and global features of the feature image, where the local features are features extracted from local regions of the image, such as edges, corners, lines, curves, and attribute regions, etc.
  • the local feature may be a regional color feature.
  • the global features refer to features that represent attributes of the entire image, for example, color features, texture features, and shape features. In this embodiment, the global feature may be the color feature of the entire image.
  • the down-sampling unit 70 includes a down-sampling layer, a local feature extraction layer, a global feature extraction layer, and a fully connected layer, and the local feature extraction layer is connected to the down-sampling layer.
  • the global feature extraction layer is connected between the down-sampling layer and the fully connected layer, and the global feature extraction layer is connected in parallel with the local feature extraction layer.
  • the first image is input to the down-sampling layer as an input item, and the feature image is output through the down-sampling layer;
  • the feature image of the down-sampling layer is input to the local feature extraction layer and the global feature extraction layer, and the local feature extraction layer extracts the feature image Local features, the global feature extraction layer extracts the global features of the feature image;
  • the local features output by the local feature extraction layer and the global features output by the global feature extraction layer are respectively input to the fully connected layer to output the bilateral network corresponding to the first image through the fully connected layer grid.
  • the downsampling layer includes a downsampling convolutional layer and four first convolutional layers.
  • the convolution kernel of the first convolutional layer is 1*1, and the step size is 1;
  • the local feature extraction layer may include two second convolutional layers, the convolution kernels of the two second convolutional layers are both 3*3, and the step size is 1;
  • the global feature extraction layer may include two There are three third convolutional layers and three fully connected layers. The convolution kernels of the two third convolutional layers are all 3*3, and the step size is both 2.
  • the convolution unit 30 includes a fourth convolution layer, the first image is input to the fourth convolution layer, and the guidance image is input through the fourth convolution layer, wherein the resolution of the guidance image is the same as that of the first image.
  • the first image is a color image
  • the fourth convolution layer performs pixel-level operations on the first image, so that the guide image is a grayscale image of the first image.
  • the first image I is input to the down-sampling convolutional layer, and the down-sampling convolutional layer outputs a three-channel low-resolution image of 256x256 size.
  • the three-channel low-resolution image of 256x256 size passes through the four first convolutional layers in turn.
  • the first image is input to the convolution unit, and the guide image corresponding to the first image is input through the convolution unit.
  • the transformation module 200 includes a segmentation unit 40 and a transformation unit 60.
  • the guide image , The bilateral grid and the first image are input to the transformation module, and the generation of the generated image corresponding to the first image through the transformation module specifically includes:
  • the segmentation unit 40 includes an up-sampling layer, and the input items of the up-sampling layer are a guide image and a bilateral grid, and the bilateral grid is up-sampled through the guide image to obtain each of the first images.
  • the color transformation matrix of the pixel may be up-sampling the bilateral grid with reference to the guide map to obtain the color transformation matrix of each pixel in the first image.
  • the input items of the transformation unit 60 are the color transformation matrix of each pixel and the first image, and the color of the corresponding pixel in the first image is transformed by the color transformation matrix of each pixel to obtain the The generated image corresponding to the first image.
  • the preset condition includes that the loss function value meets a preset requirement or the number of training times reaches a preset number.
  • the preset requirement may be determined according to the accuracy of the first image processing model, which will not be described in detail here.
  • the preset number of times may be the maximum number of training times of the first preset network model, for example, 5000 times.
  • the generated image is output in the first preset network model
  • the loss function value of the first preset network model is calculated based on the generated image and the second image, and after the loss function value is calculated, the loss is determined Whether the function value meets the preset requirements; if the loss function value meets the preset requirements, the training ends; if the loss function value does not meet the preset requirements, it is determined whether the training times of the first preset network model reaches the predicted times, if If the preset number of times is not reached, the network parameters of the first preset network model are corrected according to the loss function value; if the preset number of times is reached, the training is ended. In this way, it is judged whether the training of the first preset network model is completed by the loss function value and the number of training times, which can avoid the infinite loop of the training of the first preset network model caused by the loss function value not meeting the preset requirements.
  • the network parameters of the first preset network model are modified when the training situation of the first preset network model does not meet the preset conditions (that is, the loss function value does not meet the preset requirements and the number of training times does not reach the preset number of times ), so that after correcting the network parameters of the first preset network model according to the loss function value, it is necessary to continue training the network model, that is, continue to execute inputting the first image in the training image set into the first preset network model A step of.
  • the first image in which the first image in the training image set is continued to be input into the first preset network model is the first image that has not been input into the first preset network model as an input item.
  • all first images in the training image set have unique image identifiers (for example, image numbers), and the image identifier of the first image input for the first training is different from the image identifier of the first image input for the second training, for example,
  • the image number of the first image input in one training session is 1, the image number of the first image input in the second training session is 2, and the image number of the first image input in the Nth training session is N.
  • the first images in the training image set can be sequentially input to the first preset network model to correct The first preset network model is trained.
  • the process of sequentially inputting the first images in the training image set to the first preset network model can be continued. Operate so that the training image group in the training image set is cyclically input to the first preset network model.
  • the degree of diffusion of the highlight part of the image taken at different exposures is different, so that the degree of diffusion of the highlight part of the image captured by the under-screen imaging system under different light intensities is different, so that the image quality captured by the under-screen imaging system is different. different. Therefore, when training the first image processing model, multiple training image sets can be obtained, each training image set corresponds to a different exposure, and each training image set is used to train the first preset network model, To obtain the model parameters corresponding to each training image set. In this way, using the first image with the same exposure as the training sample image can increase the training speed of the network model, and at the same time make different exposures correspond to different model parameters.
  • the corresponding model parameters can be selected according to the exposure corresponding to the denoised image to suppress the diffusion of the highlights of the image at each exposure, so as to improve the image quality of the processed image corresponding to the denoised image.
  • the training image set includes a number of training sub-image sets
  • each training sub-image set includes a number of training sample image groups
  • any two training image groups in the training image groups The exposure of the first image in the sample image group is the same (that is, for each training image group, the exposure of the first image in each training sample image group in the group is the same), and several training image groups
  • the exposure of the second image in each training sample image group in is within a preset range, and the exposure of the first image in any two training sub-image sets is different.
  • the preset range of the exposure of the second image may be determined according to the exposure time and ISO (the aperture of the existing mobile phone is a fixed value), and the preset range of the exposure represents the range of the image taken without exposure compensation.
  • Exposure the second image captured by the on-screen camera at the first exposure within the preset range of exposure is a normal exposure image, which can be obtained by training based on the training image set by using the normal exposure image as the second image
  • the image output by the first image processing model has normal exposure, so that the first image processing model has a brightening function. For example, when the image A of the input image processing module is a low exposure image, then after the image A is processed by the first image processing model, the exposure of the output image A can be made the normal exposure, thereby improving the exposure of the image A Image brightness.
  • the training image set may include 5 training sub-image sets, which are respectively denoted as the first training sub-image set, the second training sub-image set, the third training sub-image set, the fourth training sub-image set, and the fifth training sub-image set.
  • the exposure of the first image in each training image group included in the first training sub-image set corresponds to 0 level, and the second image is the image with the exposure within a preset range;
  • the second training sub-image set The exposure of the first image in each training image group included corresponds to the -1 level, and the second image is the image with the exposure within the preset range;
  • the third training sub-image set contains the first image in each training image group.
  • the exposure of one image corresponds to -2 level, and the second image is an image whose exposure is within a preset range; the exposure of the first image in each training image group included in the fourth training sub-image set corresponds to -3 level , The second image is an image with an exposure within a preset range; the exposure of the first image in each training image group included in the fifth training sub-image set corresponds to -4 level, and the second image is an exposure in the preset range.
  • the number of training image groups contained in the first training sub-image set, the second training sub-image set, the third training sub-image set, the fourth training sub-image set, and the fifth training sub-image set may be The same can be different.
  • the first training sub-image set, the second training sub-image set, the third training sub-image set, the fourth training sub-image set, and the fifth training sub-image set all include 5000 training image groups.
  • the training sub-image set is a training image set of the first preset network model
  • the first preset network model is trained through the training sub-image set to obtain the training sub-image set.
  • the model parameters corresponding to the image set are a training image set of the first preset network model
  • the process of using the training sub-image set as a training image set to train the first preset network model includes: the first preset network model generates a generated image corresponding to the first image according to the first image in the training sub-image set; The first preset network model corrects the model parameters according to the second image corresponding to the first image and the generated image corresponding to the first image, and the first preset network model continues to execute the collection according to the training sub-image
  • steps M10 and M10 can be parameterized. Step Q20 is not repeated here.
  • each training sub-image set to the first preset network model is independent of each other, that is, each training sub-image set is used to train the first preset network model.
  • training the first preset network model with two training sub-image sets can obtain several model parameters.
  • Each model parameter is trained based on a training sub-image set, and any two model parameters correspond to each other.
  • the training sub-image sets are different from each other. It can be seen that the first image processing model corresponds to several model parameters, and several model parameters correspond to several training sub-image sets one-to-one.
  • the processing model includes 5 model parameters, which are respectively denoted as the first model parameter, the second model parameter, the third model parameter, the fourth model parameter, and the fifth model parameter.
  • the first model parameter corresponds to the first training sub-image set
  • the second model parameter corresponds to the second training sub-image set
  • the third model parameter corresponds to the third training sub-image set
  • the fourth model parameter corresponds to the fourth training sub-image set
  • the fifth model parameter corresponds to the fifth training sub-image set.
  • the training image set includes several training sub-image sets
  • the first preset network model is trained according to each training sub-image set.
  • the training image set includes 5 training sub-image sets as an example.
  • the process of using the first training sub-image set, the second training sub-image set, the third training sub-image set, the fourth training sub-image set, and the fifth training sub-image set to separately train the first preset network model may be: First use the first training sub-image set to train the first preset network model to obtain the first model parameters corresponding to the first training sub-image set, and then use the second training sub-image set to train the first preset network model , The second model parameter corresponding to the second training sub-image set is obtained, and the fifth model parameter corresponding to the fifth training sub-image set is obtained by analogy.
  • each training sub-image set affects the model parameters of the first preset network model.
  • the training sub-image set A includes 1000 training image groups
  • the training sub-image set B includes 200 training image groups. Then, first use the training sub-image set A to train the first preset network model, and then use the training sub-image set A to train the first preset network model.
  • the model parameters corresponding to the training sub-image set B obtained by training the first preset network model in image set B and the training sub-image set B obtained by training the first preset network model only with training sub-image set B
  • the model parameters corresponding to the image set B are different.
  • the first preset network model may be initialized first, and then the initialized first network model may be used.
  • the preset network model trains the next training sub-image set.
  • the first preset network model is trained according to the first training sub-image set, and after the first model parameters corresponding to the first training sub-image set are obtained, the first preset network model may be initialized to enable the use of
  • the initial model parameters and model structure of the first preset network model used to train the second model parameters are the same as the first preset network model used to train the first model parameters.
  • the first preset network model may be initialized, so that the initial model parameters and model structure of the first preset network model corresponding to each training sub-image set are the same.
  • the first preset network model is trained according to the first training sub-image set, and after obtaining the first model parameters corresponding to the first training sub-image set, it can also be directly trained based on the first training sub-image set.
  • the first preset network model configure the first model parameters
  • train the second training sub-image set to obtain the second model parameters corresponding to the second training sub-image set
  • continue to execute the first preset network model is the step of training according to the third training sub-image set, until the fifth training sub-image set is trained, and the fifth model parameter corresponding to the fifth training sub-image set is obtained.
  • the first training sub-image set, the second training sub-image set, the third training sub-image set, the fourth training sub-image set, and the fifth training sub-image set all include a certain number of training image groups, so that each group of training images
  • the sub-images can all meet the training requirements of the first preset network model.
  • the training image group in the training sub-image set can be input to the first preset network model in a loop to compare all the training images.
  • the first preset network model is trained so that the first preset network model meets the preset requirements.
  • the acquisition process of acquiring training samples containing each training sub-image set may be as follows: firstly, the under-screen imaging system is set to the first exposure, and the first exposure is acquired through the under-screen imaging system. The first image in the training sub-image set, and the second image corresponding to the first image in the first training sub-image set obtained through the on-screen imaging system; after the acquisition of the first training sub-image set is completed, set the off-screen imaging system to For the second exposure, the first image in the second training sub-image set and the second image corresponding to the first image are acquired through the off-screen imaging system and the on-screen imaging system; after the acquisition of the second training sub-image set is completed; continue to execute the setting screen Lower the exposure of the imaging system and the steps of obtaining the training sub-image set until all the training sub-image sets contained in the training image set are obtained.
  • the number of training image groups contained in each training sub-image set contained in the training image set may be the same or different. In an implementation of this embodiment, the number of training image groups contained in each training sub-image set contained in the training image set may be the same. For example, the number of training image groups contained in each training sub-image set is 5000. .
  • each training sub-image set corresponds to a different exposure
  • the model corresponding to the training sub-image set can be The parameter is associated with the exposure corresponding to the training sub-image set to establish the corresponding relationship between the exposure and the model parameter.
  • the exposure of the denoised image can be obtained first, and then the model parameters corresponding to the denoised image can be determined according to the exposure, and then the model parameters corresponding to the denoised image can be configured in
  • the first preset network model is used to obtain a first image processing model corresponding to the denoised image, so that the first image processing model can be used to process the denoised image.
  • the first image processing model with different network parameters can be determined, and the first image processing model corresponding to the denoising image can be used to process the denoising image to avoid the influence of exposure on the color cast.
  • the second image may adopt a normal exposure, so that the output image output by the first image processing model has a normal exposure, which has a brightening effect on the denoising image.
  • the first image processing model includes several model parameters, and each model parameter corresponds to an exposure degree. Therefore, in this implementation, after the denoised image is obtained, the number of model parameters included in the first image processing model can be detected first, and when the number of model parameters is one, the denoised image is directly input Into the first image processing model to process the denoised image through the image processing; when there are multiple model parameters, the exposure of the denoised image can be obtained first, and then the denoised image can be determined according to the exposure
  • the model parameters corresponding to the denoised image are configured in the first image processing model to update the model parameters configured by the image processing parameters, and the denoised image is input to the updated first image processing model.
  • the first image processing model corresponds to a number of model parameters, and each model parameter is obtained by training according to a training sub-image set, and any two model parameters respectively correspond to The training sub-image sets are different from each other (for example, the training sub-image set corresponding to the model parameter A and the training sub-image set corresponding to the model parameter B are different).
  • the inputting the denoised image into the trained first image processing model specifically includes:
  • F101 Extract the exposure of the denoised image.
  • the exposure degree is the degree to which the photosensitive element of the image acquisition device is irradiated by light, and is used to reflect the degree of exposure during imaging.
  • the denoising image may be an RGB three-channel image, the exposure of the denoising image is determined according to the highlight area of the denoising image, the R (that is, the red channel) value, G of each pixel contained in the highlight area At least one of the value of (ie, the green channel) and the value of B (ie, the blue channel) is greater than the preset threshold.
  • the denoising image can also be a Y-channel image or a Bayer format image, and when the denoising image is a Y-channel image or a Bayer format image (Raw format), the denoising image is extracted Before the image, the Y-channel image or the Bell format image needs to be converted into an RGB three-channel image, so as to determine the highlight area of the denoised image according to the red channel R value, the green channel G value and the blue channel B value of the denoised image .
  • the extracting the exposure of the denoised image specifically includes:
  • G10 Determine a first pixel that meets a preset condition according to the red channel R value, the green channel G value, and the blue channel B value of each pixel in the denoising image, where the preset condition is the R value, At least one of the G value and the B value is greater than a preset threshold;
  • G20 Determine the highlight area of the denoised image according to all the first pixel points that meet the preset condition, and determine the exposure of the denoised image according to the highlight area.
  • the denoised image is an RGB three-channel image, so for each pixel in the denoised image, the pixel includes the red channel R value, the green channel G value, and the blue channel B value, that is, for each pixel in the denoised image, For each pixel in the noisy image, the R value of the red channel, the G value of the green channel, and the B value of the blue channel of the pixel can be obtained. Therefore, in the process of extracting the exposure of the denoised image, first for each pixel of each denoised image, the red channel R value, the green channel G value, and the blue channel B value of the pixel are obtained.
  • the preset condition is that the preset condition is that at least one of the R value, the G value, and the B value is greater than a preset threshold.
  • the first pixel point meeting the preset condition means that the R value of the first pixel point is greater than the preset threshold value.
  • the G value of the first pixel is greater than the preset threshold
  • the B value of the first pixel is greater than the preset threshold
  • the R and G values of the first pixel are both greater than the preset threshold
  • the R and B values of the first pixel Are greater than the preset threshold
  • the G value and the B value of the first pixel are both greater than the preset threshold
  • the R value, B value, and G value of the first pixel are all greater than the preset threshold.
  • all the acquired first pixels are recorded as the first pixel point set, and there are adjacent pixels in the first pixel point set, and there are also non-adjacent pixels. Pixels, where adjacent pixels refer to the positions of the pixels in the denoised image are adjacent, and non-adjacent means that the positions of the pixels in the denoised image are not adjacent, and the positions are similar to each other. In the pixel coordinates to be processed, one of the abscissa and ordinate of two adjacent pixels is the same.
  • the first pixel point set includes pixel point (100,101), pixel point (100,100), pixel point (101,101) and pixel point (200,200), then pixel point (100,101) and pixel point (100,100) are adjacent pixels, And the pixel point (100, 101), the pixel point (101, 101) are adjacent pixels, and the pixel point (100, 101), the pixel point (100, 100), the pixel point (101, 101) and the pixel point (200, 200) are all non-adjacent pixels.
  • the highlight area is a connected area formed by adjacent pixels in the first pixel point set, that is, the pixel value of each first pixel point included in the highlight area meets a preset condition. Therefore, in an implementation manner of this embodiment, the determining the highlight area of the denoising image according to all the first pixels that meet the preset condition specifically includes:
  • the connected area is a closed area formed by all adjacent first pixel points in the first pixel point set, each pixel point contained in the connected area is the first pixel point, and for each pixel point in the connected area For the first pixel point A, at least one first pixel point B in the connected area is adjacent to the first pixel point A. At the same time, for the first pixel point, each first pixel point C outside the first pixel point included in the connected area is concentratedly removed, and the first pixel point C is not corresponding to any first pixel point A in the connected area. adjacent.
  • the first pixel point set includes pixel point (100, 101), pixel point (100, 100), pixel point (101, 100), pixel point (101, 101), pixel point (100, 102) and pixel point (200, 200), then pixel point (100, 101) ), pixel (100, 100), pixel (101, 100), pixel (101, 101), pixel (100, 102) form a connected area.
  • the connected area of the denoising image is formed by a light source, and the light source will produce the same color of light. Therefore, after all the connected areas contained in the denoised image are obtained, the connected areas can be selected according to the color of each connected area. Therefore, after the connected area of the denoised image is obtained, it is determined that the R value, G value, and B value of each first pixel in the connected area are greater than the preset value among the R value, G value, and B value of the first pixel. Whether the types of the R value, G value and/or B value of the threshold are the same, to determine whether the connected area meets the preset rule. The same type refers to the two first pixels, which are respectively marked as pixel A and pixel B.
  • the R value of pixel A is greater than the preset threshold, then only the R value of pixel B is greater than the preset threshold. ; If the R and G values of pixel A are greater than the preset threshold, then only the R and G values of pixel B are greater than the preset threshold; if the R, G, and B values of pixel A are greater than the preset Threshold, then the R value, G value, and B value of pixel B are all greater than the preset threshold.
  • the different types mean that for the two first pixels, they are marked as pixel point C and pixel point D respectively.
  • V value can be one of R value, G value, and B value
  • M value is R value, G value and B value One of the two values excluding the V value
  • the preset rule is that the R value, G value, and B value of the first pixel in each connected area are of the same R value, G value, and/or B value that are greater than the preset threshold.
  • the denoising image may include multiple target areas
  • the target area can be filtered according to the area of the target area to obtain the highlight area.
  • the area of the target area refers to the area of the area where the target area is located in the denoised image, and the area is calculated in the pixel coordinate system of the denoised image.
  • the area of each target area can be compared, and the target area with the largest area is selected, and the target area is regarded as the highlight area, so that the target area with the largest area is regarded as the highlight area. Go to the area with the largest brightness area in the denoising image, and determine the exposure according to the area with the largest brightness area, which can improve the accuracy of the exposure.
  • the determining the exposure of the denoised image according to the highlight area specifically includes:
  • the corresponding relationship between the ratio interval and the exposure degree is preset. After obtaining the ratio, first obtain the ratio area where the ratio is located, and then determine the The ratio interval corresponds to the exposure to obtain the exposure of the denoised image.
  • the corresponding relationship between the ratio interval and the exposure is: when the interval is [0,1/100), the exposure corresponds to 0 level; when the interval is [1/100,1/50), the exposure corresponds to ⁇ 1 level; when the interval is [1/50,1/20), the exposure corresponds to -2 level; when the interval is [1/20,1/50), the exposure corresponds to -3 level; when the interval is [1 /20,1], the exposure corresponds to -4 level. Then when the ratio of the first area to the second area is 1/10, the ratio is in the interval [1/20, 1], so the exposure degree corresponding to the denoised image is -4 level.
  • F102 Determine model parameters corresponding to the denoising image according to the exposure, and use the model parameters to update the model parameters of the first image processing model.
  • the corresponding relationship between the exposure and the model parameters is established during the training of the first image processing model, so that after the exposure of the denoised image is obtained, the exposure corresponding to the exposure can be determined according to the corresponding relationship between the exposure and the model parameters.
  • Model parameters where the exposure refers to an exposure level, that is, the corresponding relationship between the exposure and the model parameter is the corresponding relationship between the exposure level and the model parameter.
  • each exposure level corresponds to a ratio interval, then after the denoised image is obtained, the ratio of the area area of the highlight area to the image area in the denoised image can be obtained, and the ratio is determined.
  • the exposure level corresponding to the denoised image is determined according to the ratio area, and finally the model parameters corresponding to the denoised image are determined according to the exposure level, so as to obtain the model parameters corresponding to the denoised image.
  • the acquired model parameters are used to update the model parameters of the first image processing model configuration to update the first image processing model, that is, the first image corresponding to the acquired model parameters Processing model.
  • F103 Input the denoising image to the updated first image processing model.
  • the denoised image is used as an input item of the updated first image processing model, and the denoised image is output to the updated first image processing model to process the denoised image.
  • the model parameters of the image processing model corresponding to the image to be processed are model parameters determined according to the exposure of the image to be processed, and the model parameters are model parameters obtained by training a preset network model, In this way, the accuracy of the image processing to be processed by the updated image processing model can be ensured.
  • the generation of the output image corresponding to the denoised image through the first image processing model refers to the use of the denoised image as the first image processing model.
  • Input items are input to the first image processing model, and the image color of the denoised image is adjusted by the first image processing model to obtain an output image, wherein the output image corresponds to the image to be denoised
  • the image after de-casting processing For example, after the denoising image shown in FIG. 13 is processed through the image, the output image shown in FIG. 14 is obtained.
  • the first image processing model includes a down-sampling module and a transformation module, so that when processing the image to be processed through the first image processing model, it needs to pass through The down-sampling module and the transformation module perform processing.
  • the first image processing model includes; the generating the output image corresponding to the denoising image through the first image processing model specifically includes:
  • F201 Input the denoising image into the down-sampling module, and obtain the bilateral grid corresponding to the image to be processed and the guidance image corresponding to the image to be processed through the down-sampling module, wherein The resolution is the same as the resolution of the image to be processed;
  • F202 Input the guidance image, the bilateral grid, and the denoising image to the transformation module, and generate an output image corresponding to the denoising image through the transformation module.
  • the input item of the down-sampling module is a denoised image
  • the output item is a bilateral grid corresponding to the image to be denoised and a guide image
  • the input items of the transformation module are a guide image, a bilateral grid, and the image to be processed
  • the output item is the output image.
  • the structure of the down-sampling module is the same as the structure of the down-sampling module in the first preset network model. For details, reference may be made to the description of the structure of the down-sampling module in the first preset network model.
  • the processing of the image to be processed by the down-sampling module of the first image processing model is the same as the processing of the first image by the down-sampling module in the first preset network model, so the specific execution process of step F201 can refer to step Q11.
  • the structure of the transformation module is the same as the structure of the transformation module in the first preset network model.
  • the specific execution process of step F202 can refer to step Q12.
  • the down-sampling module includes a down-sampling unit and a convolution unit.
  • the inputting the denoised image into the down-sampling module, and obtaining the bilateral grid corresponding to the denoised image and the guidance image corresponding to the image to be processed through the down-sampling module specifically includes:
  • F2011 input the denoising image into the down-sampling unit and the convolution unit respectively;
  • F2012 Obtain a bilateral grid corresponding to the denoising image through the down-sampling unit, and obtain a guide image corresponding to the image to be processed through the convolution unit.
  • the input item of the down-sampling unit is a denoised image
  • the output item is a bilateral grid
  • the input item of the convolution unit is a denoised image
  • the output item is a guide image.
  • the structure of the down-sampling unit is the same as the structure of the down-sampling unit in the first preset network model.
  • the processing of the image to be processed by the down-sampling unit of the first image processing model is the same as the processing of the first image by the down-sampling unit in the first preset network model, so the specific execution process of step F2011 can refer to step Q111.
  • the structure of the convolution unit is the same as the structure of the convolution unit in the first preset network model.
  • the processing of the denoising image by the convolution unit of the first image processing model is the same as the processing of the first image by the convolution unit in the first preset network model, so the specific execution process of step F2012 can be referred to Step Q112.
  • the transformation module includes a segmentation unit and a transformation unit.
  • the inputting the guidance image, the bilateral grid, and the image to be processed into the transformation module, and generating the output image corresponding to the denoising image through the transformation module specifically includes:
  • F2022 input the denoised image and the color transformation matrix of each pixel in the image to be processed into the transformation unit, and generate an output image corresponding to the denoised image through the transformation unit.
  • the input items of the segmentation unit are the guidance image and the bilateral grid
  • the output items are the color transformation matrix of each pixel in the image to be processed
  • the input items of the transformation unit are the denoising image and the denoising image.
  • the color transformation matrix of each pixel, and the output item is the output image.
  • the structure of the segmentation unit is the same as the structure of the segmentation unit in the first preset network model. For details, reference may be made to the description of the structure of the segmentation unit in the first preset network model.
  • the segmentation unit of the first image processing model processes the bilateral grid corresponding to the image to be processed and the guidance image
  • the down-sampling unit in the first preset network model performs the processing of the bilateral grid corresponding to the first image and the guidance image
  • the processing procedure is the same, so the specific execution procedure of step F2021 can refer to step Q121.
  • the structure of the transformation unit is the same as the structure of the transformation unit in the first preset network model. For details, reference may be made to the description of the structure of the transformation unit in the first preset network model.
  • the transformation unit of the first image processing model is based on the color transformation matrix of each pixel in the image to be processed, and the transformation unit in the first preset network model is based on the color transformation matrix of each pixel in the first image
  • the processing procedure for the first image is the same, so the specific execution procedure of step F2022 can refer to step Q122.
  • the network structure corresponding to the first image processing model during the training process is the same as the network structure corresponding to the application process (removing the color cast carried by the image to be processed).
  • the first image processing model includes a down-sampling module and a transformation module. Accordingly, when the denoising image is de-pigmented by the first image processing model, the first image processing model also includes Down-sampling module and transformation module.
  • the down-sampling module of the first image processing model includes a down-sampling unit and a convolution unit, and the transformation module includes a segmentation unit and a transformation unit; accordingly, the denoising image is processed by the first image processing model.
  • the down-sampling module can also include down-sampling unit and convolution unit, and the transformation module includes segmentation unit and transformation unit; and in the application process, the working principle of each layer is the same as that of each layer in the training process. The working principle is the same. Therefore, the input and output conditions of each layer of neural network in the application process of the first image processing model can be referred to the relevant introduction in the training process of the first image processing model, which will not be repeated here.
  • the present disclosure provides an image processing method, a storage medium, and a terminal device.
  • the image processing method includes acquiring a set of images to be processed, and generating the set of images to be processed based on the set of images to be processed Corresponding denoising image; inputting the denoising image to a trained first image processing model, and generating an output image corresponding to the denoising image through the first image processing model.
  • the present disclosure first obtains multiple images, and generates a denoised image based on the multiple images.
  • the first image processing model obtained by deep learning based on the training image set is used to adjust the image color of the denoised image. This makes it possible to improve the color quality and noise quality of the output image, thereby improving the image quality.
  • H40 Input the processed image to a trained second image processing model, and perform de-ghosting processing on the processed image through the second image processing model to obtain an output image.
  • the second image processing model may be pre-trained by an image device that processes the image set to be processed (for example, a mobile phone equipped with an under-screen camera), or it may be trained by other models corresponding to the second image processing model.
  • the file is transplanted to the imaging device.
  • the imaging device may use the second image processing model as a de-ghosting function module. When the imaging device obtains the processed image, the de-ghosting function module is activated to output the image to be processed to the second image. Processing model.
  • the training process of the second image processing model may include:
  • the second preset network model generates a generated image corresponding to the third image according to the third image in the training image set.
  • the second preset network model may adopt a deep learning network model
  • the training image set includes multiple sets of training image groups with different image content
  • each training image group includes a third image and a fourth image.
  • the third image corresponds to the fourth image, they present the same image scene
  • the fourth image is the normally displayed image (or original image)
  • the image content of the third image corresponds to the fourth image but the object in the image content ghosting or blurring similar to ghosting appears.
  • the ghosting refers to the formation of a virtual image around the object in the image, for example, it may include the situation where one or multiple contours or virtual images appear on the edge of the object in the image, for example, when the object in the image has a double image (That is, when a double contour or virtual image appears at the edge of the object), the image column with a smaller pixel value can be understood as the real image of the object, and the other column image with a larger pixel value can be understood as the contour or virtual image of the object.
  • the third image and the fourth image correspond to the same image scene.
  • the third image and the fourth image corresponding to the same image scene means that the similarity between the image content carried by the third image and the image content carried by the fourth image reaches a preset threshold, and the image size of the third image is equal to that of the first image.
  • the image sizes of the four images are the same, so that when the third image and the fourth image overlap, the coverage rate of the object carried by the third image to the corresponding object in the fourth image reaches the preset condition.
  • the preset threshold may be 99%, and the preset condition may be 99.5% and so on.
  • the image content of the third image and the image content of the fourth image It can be exactly the same.
  • the third image is an image with a ghost image with an image size of 600*800
  • the image content of the third image is a square
  • the positions of the four vertices of the square in the third image in the third image are respectively (200,300), (200,400), (300,400) and (300,300).
  • the image size of the fourth image is 600*800 without ghost image
  • the image content of the fourth image is a square
  • the positions of the four vertices of the square in the fourth image in the fourth image are respectively (200,300), (200,400), (300,400) and (300,300)
  • the third image covers the fourth image
  • the The squares in the three images overlap with the squares in the fourth image up and down.
  • the fourth image may be an image obtained through normal shooting, for example, an image taken by an under-screen camera after removing the display panel in the under-screen imaging system, or by making a light-shielding structure without data lines and scan lines.
  • the experimental display panel replaces the actual display panel, and then uses it as the display panel of the under-screen imaging system. Images sent by other external devices (such as smart phones).
  • the third image may be captured by an under-screen imaging system (for example, an under-screen camera), or may be obtained by processing the fourth image.
  • the processing of the fourth image refers to forming a ghost on the fourth image. In a possible implementation manner, the image size and image content of the fourth image can be kept unchanged during the processing.
  • the third image is captured by an off-screen imaging system
  • the shooting parameters of the third image and the fourth image are the same
  • the shooting scene corresponding to the third image is the same as that of the first image.
  • the shooting scenes of the four images are the same.
  • the third image is an image obtained by shooting a scene through an under-screen camera as shown in FIG. 25.
  • the content of the image is relatively blurred due to the influence of the shading structure in the display panel.
  • the fourth image is as shown in FIG. 26 Normally displayed image.
  • the shooting parameters may include the exposure parameters of the imaging system, where the exposure parameters may include aperture, door opening speed, sensitivity, focus, white balance, and the like.
  • the shooting parameters may also include ambient light, shooting angle, and shooting range.
  • the method further includes:
  • each training image group in the training image set refers to performing alignment processing on each training image group in the training image set.
  • Perform alignment processing for each training image group to obtain an aligned training image group and perform the step of inputting the third image in each training image group into the second preset network model after all the training image groups are aligned ;
  • alignment processing is performed on the training image group to obtain the aligned training image corresponding to the training image.
  • the third image in the aligned training image group is input into the second preset network model.
  • the alignment process is performed on each training image group after the training image set is acquired, and after all the training image groups are aligned, the third image input in the training image set is executed. Operation of the second preset network model.
  • the process of aligning the third image with the fourth image corresponding to the third image is the same as the process of aligning the reference image with the reference image.
  • the third image is used as a reference.
  • the fourth image is used as the reference image. Therefore, the process of aligning the third image with the fourth image is not described in detail here. For details, please refer to the description of the process of aligning the reference image with the reference image.
  • the second preset network model includes an encoder and a decoder; the second preset network model generates a third image based on the training image set.
  • the generated image corresponding to the third image specifically includes:
  • L102 Input the characteristic image to the decoder, and output the generated image through the decoder, wherein the image size of the generated image is equal to the image size of the third image.
  • the second preset network model adopts a decoding-encoding structure
  • the decoding-encoding structure is a convolutional neural network CNN structure
  • the encoder 1000 is used to convert an input image into an image whose spatial size is smaller than that of the input image.
  • the decoder 2000 is used to convert the characteristic image into a generated image with the same size as the image size of the input image.
  • the encoder includes a first redundant learning layer 101 and a down-sampling layer 102 arranged in sequence.
  • the third image in the training image group is input to the first redundant learning layer 101, and the first redundant learning layer 101 is passed through the first redundant learning layer.
  • Layer 101 outputs a first feature map with the same image size as the third image; the first feature image is input to the downsampling layer 102 as an input item of the downsampling layer 102, and the first feature image is downsampled through the downsampling layer 102 to output
  • the second feature image corresponding to the third image (the second feature image is the feature image of the third image generated by the encoder), wherein the image size of the second feature image is smaller than the image size of the third image.
  • the decoder 2000 includes an up-sampling layer 201 and a second redundant learning layer 202 arranged in sequence.
  • the feature image output by the encoder 1000 is input to the up-sampling layer 201, and the third feature is output after being up-sampled by the up-sampling layer 201
  • the image, the third characteristic image is input to the second redundant learning layer 202, and the generated image is output after passing through the second redundant learning layer 202, wherein the image size of the generated image is the same as the image size of the third image.
  • multi-scale training can be performed on the second preset network model, so that the de-ghosting effect of the second image processing model obtained by training can be improved.
  • the first redundant learning layer 101 includes a first convolutional layer 11 and a first redundant learning module 12
  • the down-sampling layer 102 includes a first coding redundancy learning module 110 and The second coding redundancy learning module 120
  • the first coding redundancy learning module 110 includes a first down-sampling convolution layer 13 and a second redundancy learning module 14
  • the second coding redundancy learning module 120 includes a second down-sampling convolution Layer 15 and the third redundant learning module 16.
  • the input item of the first convolutional layer 11 is a third image
  • the third image is sampled to obtain the first feature image
  • the first feature image is input to the first redundant learning module 12 Perform feature extraction
  • the first feature image passing through the first redundant learning module 12 sequentially passes through the first down-sampling convolutional layer, the second redundant learning module 14, the second down-sampling convolutional layer 15, and the third redundant learning module 16 performs down-sampling to obtain the second feature image.
  • the first convolutional layer 11 samples the third image
  • the first down-sampling convolutional layer 13 and the second down-sampling convolutional layer 15 are both used to down-sample the input feature image.
  • the first redundant learning module 12, the second redundant learning module 14 and the third redundant learning module 16 are used to extract image features.
  • the first down-sampling convolutional layer 13 and the second down-sampling convolutional layer 15 may both use convolutional layers with a step size of 2.
  • a redundant learning module 12, a second redundant learning module 14 and a third redundant learning module 16 each include three redundant learning blocks arranged in sequence, and the three redundant learning blocks sequentially extract image features of the input image.
  • the third image is a 256*256 image
  • the third image is input to the first redundant learning layer 101 through the input layer, and after the first redundant learning layer 101, the first feature image of 256*256 is output;
  • the feature image is input to the first down-sampling convolutional layer 13 of the first coding redundancy learning module 110.
  • the fourth feature image with an image size of 128*128 is passed through the fourth feature image.
  • the first redundancy learning module 12 of a coding redundancy learning module 110 performs feature extraction; the fourth feature image of the first redundancy learning module 12 is input into the second down-sampling convolution layer 15 of the second coding redundancy learning module 120 , The second feature image with an image size of 64*64 after being processed by the second down-sampling convolution layer 15, and the second feature image is subjected to feature extraction through the second redundancy learning module 16 of the second encoding redundancy learning module 120.
  • the up-sampling layer 201 includes a first decoding redundancy learning module 210 and a second decoding redundancy learning module 220, and the first decoding redundancy learning module 210 includes a fourth redundancy learning module 21 and The first up-sampling convolutional layer 22, the second decoding redundancy learning module 220 includes a fifth redundancy learning module 23 and a second up-sampling convolutional layer 24, the second redundancy learning layer 202 includes a sixth redundancy learning Module 25 and second convolutional layer 26.
  • the input item of the first up-sampling convolutional layer 22 is the first feature image, and the input first feature image sequentially passes through the fourth redundant learning module 21, the first up-sampling convolutional layer 22, and the fifth redundant learning module.
  • the module 23 and the second up-sampling convolutional layer 24 perform up-sampling to obtain a third feature image, and input the third feature image to the sixth redundant learning module 25, and perform feature extraction through the sixth redundant learning module 25
  • the latter third characteristic image is input to the second convolution layer 26, and the generated image is obtained through the second convolution layer 26.
  • the first up-sampling convolutional layer 22 and the second up-sampling convolutional layer 24 are used for up-sampling the input feature image
  • the module 23 and the sixth redundant learning module 25 are both used to extract image features
  • the second convolutional layer 26 is used to sample the input feature images.
  • the first up-sampling convolutional layer 22 and the second up-sampling convolutional layer 24 are both deconvolutional layers with a step length of 2
  • the module 21, the fifth redundant learning module 23, and the sixth redundant learning module 25 each include three redundant learning blocks, and the three redundant learning blocks sequentially extract image features of the input image.
  • the third redundant learning block of the redundant learning module in the first redundant learning layer 101 is hop-connected to the first redundant learning block of the redundant learning module in the second redundant learning layer 202
  • the third redundant learning block of the redundant learning module in the first encoding redundancy learning module 110 is hop-connected to the first redundant learning block of the redundant learning module in the second decoding redundancy learning module 220.
  • the third image is a 256*256 image through the encoder 1000 to obtain a 64*64 second feature image
  • the 64*64 second feature image is input through the fourth redundancy of the first decoding redundancy learning module 210.
  • the co-learning module 21 performs feature extraction.
  • the second feature image of 64*64 after feature extraction is input to the first up-sampling convolutional layer 22 of the first decoding redundant learning module 210, and the first up-sampling convolutional layer 22 is output.
  • a fifth feature image with an image size of 128*128.
  • the fifth feature image is subjected to feature extraction through the fifth redundant learning module 23 of the second decoding redundant learning module 220; the fifth feature image that passes through the fifth redundant learning module 23
  • the second up-sampling convolutional layer 24 of the second decoding redundancy learning module 220 is output.
  • a third feature image with an image size of 256*256 is output.
  • the third feature image is input to the second redundant
  • the remaining learning layer 202 outputs a 256*256 generated image after passing through the second redundant learning layer 202.
  • the encoder and the decoder include a first convolutional layer, a second convolutional layer, a first up-sampling convolutional layer, a second up-sampling convolutional layer, a first down-convolution layer, and a second down-convolution layer.
  • the convolutional layers all use linear rectification functions as the activation functions and the convolution kernels are all 5*5, which can improve the gradient transfer efficiency of each layer, and after multiple back propagation, the gradient amplitude changes little, which improves the training
  • the accuracy of the generator can also increase the receptive field of the network.
  • the second preset network model corrects the model parameters according to the fourth image corresponding to the third image and the generated image corresponding to the third image, and continues to execute according to the next training image in the training image set
  • the third image in the group generates an image corresponding to the third image until the training condition of the second preset network model satisfies a preset condition, so as to obtain the second image processing model.
  • said correcting the second preset network model refers to correcting the model parameters of the second preset network model until the model parameters meet a preset condition.
  • the preset condition includes that the loss function value meets a preset requirement or the number of training times reaches a preset number.
  • the preset requirement may be determined according to the accuracy of the second image processing model, which will not be described in detail here.
  • the preset number of times may be the maximum number of training times of the second preset network model, for example, 4000 times.
  • the generated image is output in the second preset network model
  • the loss function value of the second preset network model is calculated based on the generated image and the fourth image, and after the loss function value is calculated, the loss is determined Whether the function value meets the preset requirements; if the loss function value meets the preset requirements, the training ends; if the loss function value does not meet the preset requirements, it is determined whether the training times of the second preset network model reaches the predicted times, if If the preset number of times is not reached, the network parameters of the second preset network model are corrected according to the loss function value; if the preset number of times is reached, the training is ended.
  • the network parameters of the second preset network model are modified when the training situation of the second preset network model does not meet the preset conditions (for example, the loss function value does not meet the preset requirements and the number of training times does not reach the preset number of times ), so that after the network parameters of the second preset network model are corrected according to the loss function value, it is necessary to continue training the network model, that is, continue to execute inputting the third image in the training image set into the second preset network model A step of.
  • the third image that continues to input the third image in the training image set into the second preset network model may be a third image that has not been input to the second preset network model as an input item.
  • all third images in the training image set have unique image identifiers (for example, image numbers), the image identifiers of the third images input into the second preset network model for the first training and the second preset network model for the second training input
  • the image ID of the third image is different, for example, the image number of the third image input to the second preset network model for the first training is 1, and the image number of the third image input for the second preset network model for the second training Is 2, and the image number of the third image input to the second preset network model for the Nth training is N.
  • the third images in the training image set can be sequentially input to the second preset network model to correct The second preset network model is trained. After all the third images in the training image set have been input to the second preset network model, the process of sequentially inputting the third images in the training image set to the second preset network model can be continued. Operate so that the training image group in the training image set is cyclically input to the second preset network model. It should be noted that in the process of inputting the third image into the second preset network model training, it can be input in the order of the image number of each third image or not in the order of the image number of each third image.
  • the same third image may be used repeatedly to train the second preset network model, or the same third image may not be used repeatedly to train the second preset network model.
  • the specific implementation mode of "the step of inputting the third image in the image set into the second preset network model" is defined.
  • the loss function value is calculated from a structural similarity loss function and a content bidirectional loss function.
  • the second preset network model compares the model of the second preset network model according to the fourth image corresponding to the third image and the generated image corresponding to the third image. The parameters are corrected, and the step of generating the generated image corresponding to the third image according to the third image in the training image set is continued until the training situation of the second preset network model meets the preset conditions, so as to obtain the trained
  • the second image processing model specifically includes:
  • the second preset network model adopts a structural similarity index (Structural similarity index, SSIM) loss function and a content bidirectional (Contextual bilateral loss, CoBi) loss based on VGG (Visual Geometry Group Network, VGG network) extraction features
  • SSIM structural similarity index
  • CoBi Content bidirectional loss
  • VGG Visual Geometry Group Network
  • the stochastic gradient descent method is used to train the second preset network model, where the initial network parameters for training are set to 0.0001, And the network parameter adopts the way of exponential decay to revise when revising.
  • the structural similarity loss function value is used to measure the structural similarity between the generated image and the fourth image.
  • the expression of the structural similarity loss function corresponding to the structural similarity loss function value may be:
  • ⁇ x is the average value of the pixel values of all pixels in the generated image
  • ⁇ y is the average value of the pixel values of all pixels in the second image
  • ⁇ x is the variance of the pixel values of all pixels in the generated image
  • ⁇ y is the variance of the pixel values of all pixels in the second image
  • ⁇ xy is the covariance between the generated image and the second image.
  • the value of the content bidirectional loss function is calculated by the CoBi loss function based on the VGG feature
  • the CoBi loss function based on the VGG feature is obtained by extracting several sets of VGG features of the generated image and the fourth image respectively, and is specific to the generated image
  • search for a second VGG feature match close to the first VGG feature in the second VGG feature of the fourth image, and finally calculate the distance sum between each first VGG feature and its matching second VGG feature .
  • the content two-way loss function value so that the two-sided distance is searched through the content two-way loss function, considering the spatial loss of the first VGG feature and its matching second VGG feature, so that the third image and the fourth image can be avoided
  • the impact of incomplete alignment improves the speed and accuracy of training the second preset network model.
  • the content two-way loss function value is determined according to the distance and position relationship between the first VGG feature and the second VGG feature, which improves the accuracy of matching. Thereby, the influence of the misalignment of the third image and the fourth image on the training of the second preset network model is further reduced.
  • the expression of the content bidirectional loss function may be:
  • D is the cosine distance between the VVG feature of the generated image and the VVG feature of the second image
  • D′ is the spatial position distance between the VVG feature of the generated image and the VVG feature of the second image
  • N is the VVG of the generated image
  • N200 Perform de-ghosting processing on the processed image by using the second image processing model, and use the de-ghosting processed image as the processed image.
  • the de-ghosting of the image to be processed by the second image processing model refers to inputting the image to be processed into the second image as an input item of the second image processing model
  • the second image processing model is used to remove the ghost of the image to be processed to obtain a processed image, where the processed image is the image to be processed corresponding to the image processed by the second image processing model The resulting image.
  • the image to be processed is an image with ghosting corresponding to the output image, that is, the output image corresponds to the image to be processed, and they present the same image scene, the output image is the image that is normally displayed, the image of the image to be processed The content corresponds to the output image, but the object in the image content to be processed has ghosting or a blur effect similar to ghosting. For example, after the image to be processed as shown in FIG. 25 is processed through the image, the output image as shown in FIG. 26 is obtained.
  • the second image processing model includes an encoder and a decoder, so that when the second image processing model corresponds to the image to be processed for processing, it needs to be encoded separately.
  • Processor and decoder for processing the de-ghosting the image to be processed by the second image processing model to obtain the processed image corresponding to the image to be processed specifically includes:
  • N201 Input the image to be processed into the encoder, and obtain a characteristic image of the image to be processed through the encoder, wherein the image size of the characteristic image is smaller than the image size of the image to be processed;
  • N202 Input the characteristic image to the decoder, and output a processed image corresponding to the image to be processed through the decoder, wherein the image size of the processed image is equal to the image size of the image to be processed.
  • the encoder converts the input image to be processed into a feature image with an image space size smaller than the input image and more channels than the input image, and the feature image is input to the decoder, and the decoder converts the input image to a feature image.
  • the feature image is converted into a generated image with the same size as the image to be processed.
  • the structure of the encoder is the same as the structure of the encoder in the second preset network model. For details, reference may be made to the description of the structure of the encoder in the second preset network model.
  • the processing of the image to be processed by the encoder of the second image processing model is the same as the processing of the third image by the encoder in the second preset network model, so the specific execution process of step N201 can refer to step L101.
  • the structure of the decoder is the same as the structure of the decoder in the second preset network model.
  • the process of processing the characteristic image corresponding to the image to be processed by the decoder of the second image processing model is the same as the process of processing the characteristic image corresponding to the third image by the decoder in the second preset network model, so that the process of step N202 For the specific execution process, please refer to step L202.
  • the network structure corresponding to the second image processing model during the training process is the same as the network structure corresponding to the application process (removal of ghost images carried by the processed image).
  • the second image processing model includes an encoder and an encoder. Accordingly, when the ghost image carried by the processed image is removed by the second image processing model, the second image processing model also includes an encoder And encoder.
  • the encoder of the second image processing model includes the encoder including a first redundant learning layer and a down-sampling layer, and the decoder includes an up-sampling layer and a second redundant learning layer; correspondingly Specifically, when the ghost image carried by the processed image is removed by the second image processing model, the encoder may also include a first redundant learning layer and a down-sampling layer, and the decoder may include an up-sampling layer and a second redundant learning layer; and
  • the working principle of each layer is the same as the working principle of each layer in the training process. Therefore, the input and output of each layer of neural network in the application process of the second image processing model can be referred to in the second image processing The relevant introduction in the training process of the model will not be repeated here.
  • post-processing may be performed on the output image, where the post-processing may include sharpening processing and noise reduction processing.
  • the method further includes:
  • the sharpening processing refers to compensating the contour of the output image, enhancing the edges of the output image and the part where the gray level jumps, so as to improve the image quality of the processed image.
  • the sharpening processing may adopt an existing sharpening processing method, for example, a high-pass filtering method.
  • the noise reduction processing refers to removing noise in the image and improving the signal-to-noise ratio of the image.
  • the noise reduction processing may adopt an existing noise reduction algorithm or a trained noise reduction network model, etc., for example, the noise reduction processing adopts a Gaussian low-pass filtering method or the like.
  • this embodiment provides an image processing device, and the image processing device includes:
  • the third acquiring module 501 is configured to acquire a set of images to be processed, where the set of images to be processed includes multiple images;
  • the third generating module 502 is configured to generate a denoising image corresponding to the to-be-processed image set according to the to-be-processed image set;
  • the third processing module 503 is configured to input the denoised image into the trained first image processing model, and perform de-color cast processing on the denoised image through the image processing model to obtain the denoised image The corresponding processed image;
  • the fourth processing module 504 is configured to input the processed image to a second image processing model that has been trained, and perform de-ghosting processing on the processed image through the second image processing model to obtain an output image .
  • one of the multiple images included in the image set to be processed is a base image, and the remaining images are neighboring images of the base image, and the third generation module is specifically configured to:
  • the weight parameter set corresponding to the basic image block includes a first weight parameter and a second weight parameter
  • the first weight parameter is the weight parameter of the basic image block
  • the second weight parameter is The weight parameter of the adjacent image block corresponding to the basic image block in the adjacent image
  • the denoising image is determined according to the image set to be processed and the weight parameter set corresponding to each basic image block.
  • the number of images in the image set to be processed is determined according to shooting parameters corresponding to the image set to be processed.
  • the image definition of the base image is greater than or equal to the image definition of the adjacent image.
  • the third generation module is specifically configured to:
  • For each basic image block determine the second weight parameter of each adjacent image block corresponding to the basic image block, and obtain the first weight parameter corresponding to the basic image block to obtain the weight parameter set corresponding to the basic image block.
  • the third generation module is specifically configured to:
  • the third generation module is specifically configured to:
  • the preset second preset parameter is used as the second weight parameter of the adjacent image block.
  • the first threshold and the second threshold are both determined according to the similarity values of adjacent image blocks corresponding to the basic image block.
  • the image processing device further includes:
  • the spatial denoising module is used to perform spatial denoising on the denoised image, and use the image obtained after spatial denoising as the denoised image.
  • the first image processing model is obtained by training based on a first training image set.
  • the first training image set includes a plurality of training image groups, and each training image group includes a first image and a second image.
  • Image the first image is a color cast image corresponding to the second image.
  • the first image is an image captured by an off-screen imaging system.
  • the image processing device further includes:
  • the third alignment module is used for aligning the first image in the training image group with the second image corresponding to the first image for each training image group in the first training sample set to obtain the The second image is aligned with the alignment image, and the alignment image is used as the first image.
  • the first training image set includes several training sub-image sets, each training sub-image set includes several training sample image groups, and any two training sample image groups in the several training image groups
  • the exposure of the first image is the same, the exposure of the second image in each training sample image group in the several training image groups is within the preset range, and the exposure of the first image in any two training sub-image sets The degrees are not the same.
  • the first image processing model corresponds to a number of model parameters
  • each model parameter is obtained by training according to a training sub-image set in the first training image set
  • any two model parameters respectively correspond to The training sub-image sets of are different from each other.
  • the third processing module is specifically configured to:
  • the third processing module is specifically configured to:
  • a third pixel that meets a preset condition is determined, where the preset condition is at least one of the R value, the G value, and the B value Greater than a preset threshold; determining the highlight area of the denoised image according to all third pixel points that meet the preset condition, and determining the exposure of the denoised image according to the highlight area.
  • the third processing module is specifically configured to:
  • the connected areas formed by all the third pixel points that meet the preset conditions select target areas that meet the preset rules from all the obtained connected areas, calculate the respective areas corresponding to each target area obtained by screening, and The target area with the largest area is selected as the highlight area, where the preset rule is that the R value, G value and/or B value of the third pixel in the target area are greater than the preset threshold.
  • the values are of the same type.
  • the third processing module is specifically configured to:
  • the first image processing model includes a down-sampling module and a transformation module; the third processing module is specifically configured to:
  • the denoised image is input to the downsampling module, and the bilateral grid corresponding to the denoised image and the guide image corresponding to the denoised image are obtained through the downsampling module, wherein the resolution of the guide image The same as the resolution of the denoising image;
  • the guide image, the bilateral grid, and the denoising image are input to the transformation module, and the processed image corresponding to the denoising image is generated by the transformation module.
  • the down-sampling module includes a down-sampling unit and a convolution unit; the third processing module is specifically configured to:
  • the two-sided grid corresponding to the denoising image is obtained by the down-sampling unit, and the guide image corresponding to the denoising image is obtained by the convolution unit.
  • the transformation module includes a segmentation unit and a transformation unit
  • the third processing module is specifically configured to:
  • the denoising image and the color transformation matrix of each pixel in the denoising image are input to the transformation unit, and the processed image corresponding to the denoising image is generated by the transformation unit.
  • the second image processing model is obtained by training based on a second training image set.
  • the second training image set includes a plurality of training image groups, and each training image group includes a third image and a fourth image.
  • the third image is an image with ghosting corresponding to the fourth image.
  • the third image is generated based on the fourth image and a point spread function, wherein the point spread function is generated based on a grayscale image generated by a light shielding structure in an under-screen imaging system.
  • the third image is an image captured by an off-screen imaging system.
  • the under-screen imaging system is an under-screen camera.
  • the image processing device further includes:
  • the fourth alignment module is used for aligning the third image in the training image group with the fourth image corresponding to the third image for each training image group in the second training image set to obtain the The fourth image is aligned with the alignment image, and the alignment image is used as the third image.
  • the second image processing model includes an encoder and a decoder; the fourth processing module is specifically configured to:
  • the processed image is input to the encoder, and the characteristic image of the processed image is obtained through the encoder; and the characteristic image is input to the decoder, and the processed image is output through the decoder
  • the corresponding output image wherein the image size of the characteristic image is smaller than the image size of the processed image; the image size of the output image is equal to the image size of the processed image.
  • both the third alignment module and/or the fourth alignment module are specifically configured to:
  • the alignment method aligns the reference image with the reference image, wherein when the reference image is the second image, the reference image is the first image; when the reference image is the fourth image, the reference image is the third image .
  • both the third alignment module and/or the fourth alignment module are specifically configured to:
  • the reference pixel point set of the reference image and the reference pixel point set of the reference image are extracted, and the reference pixel point set includes the reference pixel point set in the reference image.
  • a number of reference pixels the reference pixel point set includes a number of reference pixels in the base image, the reference pixel points in the reference pixel point set correspond to the reference pixels in the reference pixel point set in a one-to-one correspondence;
  • the coordinate difference between the reference pixel point and its corresponding reference pixel point is calculated, and the position of the reference pixel point is adjusted according to the coordinate difference value corresponding to the reference pixel point, so as to The reference pixel point is aligned with the reference pixel point corresponding to the reference pixel point.
  • the image processing device further includes:
  • the sharpening and noise reduction module is used to perform sharpening and noise reduction processing on the processed image, and use the processed image after the sharpening and noise reduction processing as the output image.
  • this embodiment provides a computer-readable storage medium, the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors , In order to realize the steps in the image processing method described in the above embodiments.
  • the present disclosure also provides a terminal device, as shown in FIG. 39, which includes at least one processor (processor) 30; a display panel 31; and a memory (memory) 32, and may also include a communication interface ( Communications Interface) 33 and bus 34.
  • the processor 30, the display panel 31, the memory 32, and the communication interface 33 can communicate with each other through the bus 34.
  • the display panel 31 is configured to display a user guide interface preset in the initial setting mode.
  • the communication interface 33 can transmit information.
  • the processor 30 may call the logic instructions in the memory 32 to execute the method in the foregoing embodiment.
  • logic instructions in the memory 32 can be implemented in the form of a software functional unit and when sold or used as an independent product, they can be stored in a computer readable storage medium.

Abstract

一种图像处理模型的生成方法、处理方法、存储介质及终端，所述生成方法通过将预设的训练图像集中第一图像输入预设网络模型，通过所述预设网络模型生成的生成图像以及第一图像对应的第二图像对预设模型进行序列，得到图像处理模型。所述图像处理模型为通过对具有多组训练图像组的训练图像集的去偏色过程进行深度学习得到，每一组训练图像组包括第一图像和第二图像，第一图像为对应第二图像的偏色图像。由此可知，采用基于训练图像集进行深度学习得到已训练的图像处理模型进行偏色处理，这样可以快速对图像进行偏色调整，提高图像的色彩质量，从而提高图像质量。

Description

图像处理模型生成方法、处理方法、存储介质及终端
优先权
本公开要求于申请日为2020年3月10日提交中国专利局、申请号为“2020101626840”、申请名称为“一种图像处理方法、存储介质及终端设备”的中国专利申请，申请日为2020年3月10日提交中国专利局、申请号为“2020101634724”、申请名称为“图像处理模型的生成方法、处理方法、存储介质及终端”的中国专利申请，以及申请日为2020年3月10日提交中国专利局、申请号为“2020101627097”、申请名称为“图像处理模型的训练方法、图像处理方法、介质及终端”的中国专利申请的优先权，其全部内容通过引用结合在本公开中。
技术领域
本公开涉及图像处理技术领域，特别涉及一种图像处理模型生成方法、处理方法、存储介质及终端。
背景技术
现有的全面屏终端普遍包括显示面板区域以及摄像头区域，摄像头区域位于显示面板区域的顶部，这样虽然可以增大屏占比，但是摄像头区域还是会占用部分显示区域，无法真实达到全面屏。因而，为了实现全面屏终端需要在显示面板下安装成像系统，现有显示面板普遍包括基板以及偏光片等，而当光线通过显示面板时，显示面板一方面折射光线而使得光线透射率低，另一方显示面板会吸收光线，这样会影响拍摄得到的图像质量，例如，拍摄得到的图像色彩与拍摄场景不符以及图像噪声增多等。
公开内容
本公开要解决的技术问题在于，针对现有技术的不足，提供一种图像处理模型生成方法、处理方法、存储介质及终端。
本公开第一方面提供了一种图像处理模型的生成方法，其中，所述图像处理模型的生成方法具体包括：
预设网络模型根据训练图像集中的第一图像，生成所述第一图像对应的生成图像，其中，所述训练图像集包括多组训练图像组，每一组训练图像组包括第一图像和第二图 像，第一图像为对应第二图像的偏色图像；
所述预设网络模型根据所述第一图像对应的第二图像和所述第一图像对应的生成图像，对模型参数进行修正，并继续执行根据所述训练图像集中的下一训练图像组中的第一图像，生成所述第一图像对应的生成图像的步骤，直至所述预设网络模型的训练情况满足预设条件，以得到所述图像处理模型。
本公开第二方面提供了一种图像处理方法，其中，应用如上所述的图像处理模型的生成方法生成的图像处理模型，所述图像处理方法包括：
获取待处理图像，并将所述待处理图像输入至所述图像处理模型；
通过所述图像处理模型对所述待处理图像进行偏色处理，以得到所述待处理图像对应的处理后的图像。
本公开第三方面提供了一种图像处理模型的生成方法，其中，所述图像处理模型的生成方法具体包括：
预设网络模型根据训练图像集中的第一图像生成所述第一图像对应的生成图像；其中，所述训练图像集包括多组训练图像组，每一组训练图像组包括第一图像和第二图像，第一图像为第二图像对应的具有重影的图像；
所述预设网络模型根据所述第一图像对应的第二图像和所述第一图像对应的生成图像，对所述预设网络模型的模型参数进行修正，并继续执行根据训练图像集中的下一训练图像组中的第一图像，生成所述第一图像对应的生成图像的步骤，直至所述预设网络模型的训练情况满足预设条件，以得到已训练的图像处理模型。
本公开第四方面提供了一种图像处理方法，其中，应用如上所述的图像处理模型的生成方法生成的图像处理模型，所述图像处理方法包括：
获取待处理图像，并将所述待处理图像输入至所述图像处理模型；
通过所述图像处理模型对所述待处理图像进行去重影处理，以得到所述待处理图像对应的输出图像。
本公开第五方面提供了一种图像处理方法，其中，所述方法包括：
获取待处理图像集，其中，所述待处理图像集包括多张图像；
根据所述待处理图像集，生成所述待处理图像集对应的去噪图像；
将所述去噪图像输入至以已训练的第一图像处理模型，通过所述图像处理模型对所述去噪图像进行去偏色处理，得到所述去噪图像对应的处理后图像；
将所述处理后图像输入至以已训练的第二图像处理模型，通过所述第二图像处理 模型对所述处理后图像进行去重影处理，以得到输出图像。
在一个实施例中，所述待处理图像集包括的多张图像中一张图像为基础图像，其余图像为基础图像的临近图像,所述根据所述待处理图像集，生成所述待处理图像集对应的去噪图像具体包括：
将所述基础图像划分为若干基础图像块，分别确定各基础图像在各临近图像中对应的临近图像块；
确定各个基础图像块分别对应的权重参数集；其中，基础图像块对应的权重参数集包括第一权重参数和第二权重参数，第一权重参数为基础图像块的权重参数，第二权重参数为临近图像中与基础图像块对应的临近图像块的权重参数；
根据所述待处理图像集以及各个基础图像块分别对应的权重参数集，确定去噪图像。
在一个实施例中，所述待处理图像集的图像数量为根据所述待处理图像集对应的拍摄参数确定的。
在一个实施例中，所述基础图像的图像清晰度大于或等于所述临近图像的图像清晰度。
在一个实施例中，所述确定各个基础图像块分别对应的权重参数集具体包括：
针对每个基础图像块，确定该基础图像块对应的各临近图像块的第二权重参数，以及，获取该基础图像块对应的第一权重参数，以得到该基础图像块对应的权重参数集。
在一个实施例中，所述确定该基础图像块对应的各临近图像块的第二权重参数具体包括：
针对每个临近图像块，计算该基础图像块与该临近图像块的相似度值；
根据所述相似度值计算该临近图像块的第二权重参数。
在一个实施例中，所述根据所述相似度值计算该临近图像块的第二权重参数具体包括：
当所述相似度值小于或等于第一阈值时，将第一预设参数作为该临近图像块的第二权重参数；
当所述相似度值大于第一阈值，且小于或等于第二阈值时，根据所述相似度值、所述第一阈值及所述第二阈值计算该临近图像块的第二权重参数；
当所述相似度值大于第二阈值时，将预设第二预设参数作为该临近图像块的第二权重参数。
在一个实施例中，所述第一阈值和第二阈值均为根据该基础图像块对应临近图像块的相似度值确定的。
在一个实施例中，所述根据所述待处理图像集以及各个基础图像块分别对应的权重参数集，确定去噪图像之后还包括：
对所述去噪图像进行空域降噪，并将空域降噪后得到的图像作为去噪图像。
在一个实施例中，所述第一图像处理模型为基于第一训练图像集训练得到，所述第一训练图像集包括多组训练图像组，每一组训练图像组包括第一图像和第二图像，第一图像为对应第二图像的偏色图像。
在一个实施例中，所述第一图像为通过屏下成像系统拍摄得到的图像。
在一个实施例中，基于第一训练图像集训练第一图像处理模型之前，所述方法包括：
针对所述第一训练样本集中每组训练图像组，将该组训练图像组中的第一图像与所述第一图像对应的第二图像进行对齐处理，得到与所述第二图像对齐的对齐图像，并将所述对齐图像作为第一图像。
在一个实施例中，所述第一训练图像集包括若干训练子图像集，每个训练子图像集包括若干组训练样本图像组，若干训组训练图像组中的任意两组训练样本图像组中的第一图像的曝光度相同，若干组训练图像组中的每组训练样本图像组中的第二图像的曝光度均处于预设范围内，任意两个训练子图像集中的第一图像的曝光度不相同。
在一个实施例中，所述第一图像处理模型对应若干模型参数，每个模型参数均为根据所述第一训练图像集中一个训练子图像集训练得到的，并且任意两个模型参数各自分别对应的训练子图像集互不相同。
在一个实施例中，所述将所述去噪图像输入至以已训练的第一图像处理模型具体包括：
提取所述去噪图像的曝光度；
根据所述曝光度确定所述去噪图像对应的模型参数，并采用所述模型参数更新所述第一图像处理模型的模型参数；
将所述去噪图像输入至更新后的第一图像处理模型。
在一个实施例中，所述提取所述去噪图像的曝光度具体包括：
根据所述去噪图像中各像素点的R值、G值以及B值确定满足预设条件的第三像素点，其中，所述预设条件为R值、G值以及B值中至少一个值大于预设阈值；
根据满足预设条件的所有第三像素点确定所述去噪图像的高光区域，并根据所述高光区域确定所述去噪图像的曝光度。
在一个实施例中，所述根据满足预设条件的所有第三像素点确定所述去噪图像的高光区域具体包括：
获取所述满足预设条件的所有第三像素点所形成的连通区域，并在获取到的所有连通区域进行选取满足预设规则的目标区域，其中，所述预设规则为目标区域中的第三像素点的R值、G值和B值中大于预设阈值的R值、G值和/或B值的类型相同；
计算筛选得到的各目标区域分别对应的面积，并选取面积最大的目标区域作为高光区域。
在一个实施例中，所述根据所述高光区域确定所述去噪图像的曝光度具体包括：
计算所述高光区域的第一面积以及去噪图像的第二面积；
根据所述第一面积和第二面积的比值确定所述去噪图像对应的曝光度。
在一个实施例中，所述第一图像处理模型包括下采样模块以及变换模块；所述通过所述第一图像处理模型对所述去噪图像进行去偏色处理，以得到所述去噪图像对应的处理后图像具体包括：
将所述去噪图像输入所述下采样模块，通过所述下采样模块得到所述去噪图像对应的双边网格以及所述去噪图像对应的指导图像，其中，所述指导图像的分辨率与所述去噪图像的分辨率相同；
将所述指导图像、所述双边网格以及所述去噪图像输入所述变换模块，通过变换模块生成所述去噪图像对应的处理后图像。
在一个实施例中，所述下采样模块包括下采样单元和卷积单元；所述将所述去噪图像输入所述下采样模块，通过所述下采样模块得到所述去噪图像对应的双边网格以及所述去噪图像对应的指导图像具体包括：
将所述去噪图像分别输入所述下采样单元以及所述卷积单元；
通过所述下采样单元得到所述去噪图像对应的双边网格，并通过所述卷积单元得到所述去噪图像对应的指导图像。
在一个实施例中，所述变换模块包括切分单元以及变换单元，所述将所述指导图像、所述双边网格以及所述去噪图像输入所述变换模块，通过变换模块生成所述去噪图像对应的处理后图像具体包括：
将所述指导图像输入所述切分单元，通过所述切分单元对所述双边网格进行切分， 以得到所述去噪图像中各像素点的颜色变换矩阵；
将所述去噪图像以及所述去噪图像中各像素点的颜色变换矩阵输入所述变换单元，通过所述变换单元生成所述去噪图像对应的处理后图像。
在一个实施例中，所述第二图像处理模型为基于第二训练图像集训练得到，所述第二训练图像集包括多组训练图像组，每一组训练图像组包括第三图像和第四图像，第三图像为第四图像对应的具有重影的图像。
在一个实施例中，所述第三图像为根据第四图像和点扩散函数生成的，其中，所述点扩散函数为根据屏下成像系统中的遮光结构生成的灰度图生成的。
在一个实施例中，所述第三图像为通过屏下成像系统拍摄得到的图像。
在一个实施例中，所述屏下成像系统为屏下摄像头。
在一个实施例中，基于第二训练图像集训练第二图像处理模型之前，所述方法还包括：
针对所述第二训练图像集中每组训练图像组，将该组训练图像组中的第三图像与所述第三图像对应的第四图像进行对齐处理，得到与所述第四图像对齐的对齐图像，并将所述对齐图像作为第三图像。
在一个实施例中，所述第二图像处理模型包括编码器和解码器；所述通过所述第二图像处理模型对所述处理后图像进行去重影，以得到所述处理后图像对应的输出图像具体包括：
将所述处理后图像输入所述编码器，通过所述编码器得到所述处理后图像的特征图像，其中，所述特征图像的图像尺寸小于所述处理后图像的图像尺寸；
将所述特征图像输入所述解码器，通过所述解码器输出所述处理后图像对应的输出图像，其中，所述输出图像的图像尺寸等于所述处理后图像的图像尺寸。
在一个实施例中，所述对齐处理的过程具体包括：
获取训练图像组中的基准图像和参考图像，并计算所述参考图像与所述基准图像之间的像素偏差量；其中，当基准图像为第二图像时，参考图像为第一图像；当基准图像为第四图像时，参考图像为第三图像；
根据所述像素偏差量确定所述参考图像对应的对齐方式，并采用所述对齐方式将所述参考图像与所述基准图像进行对齐处理。
在一个实施例中，所述根据所述像素偏差量确定所述述参考图像对应的对齐方式，并采用所述对齐方式将所述参考图像与所述基准图像进行对齐处理具体包括：
当所述像素偏差量小于或等于预设偏差量阈值时，根据所述参考图像与所述基准图像的互信息，以所述基准图像为基准对所述参考图像进行对齐处理；
当所述像素偏差量大于所述预设偏差量阈值时，提取所述参考图像的基准像素点集和所述基准图像的参考像素点集，所述基准像素点集包含所述参考图像中的若干参考像素点，所述基准像素点集包括所述基图像中的若干基准像素点，所述参考像素点集中的参考像素点与所述基准像素点集中的基准像素点一一对应；针对所述基准像素点集中每个基准像素点，计算该基准像素点与其对应的参考像素点的坐标差值，并根据该参考像素点对应的坐标差值对该参考像素点进行位置调整，以将该参考准像素点与该参考像素点对应的基准像素点对齐。
在一个实施例中，所述将所述处理后图像输入至以已训练的第二图像处理模型，通过所述第二图像处理模型对所述处理后图像进行去重影处理，以得到输出图像之后还包括：
对所述处理后图像进行锐化以及降噪处理，并将锐化以及降噪处理后的处理后图像作为所述输出图像。
本公开第六方面提供了一种图像处理模型的生成装置，其中，所述图像处理模型的生成装置包括：
第一生成模块，用于利用预设网络模型根据训练图像集中的第一图像，生成所述第一图像对应的生成图像，其中，所述训练图像集包括多组训练图像组，每一组训练图像组包括第一图像和第二图像，第一图像为对应第二图像的偏色图像；
第一修正模块，用于利用预设网络模型根据所述第一图像对应的第二图像和所述第一图像对应的生成图像，对模型参数进行修正，并继续执行根据所述训练图像集中的下一训练图像组中的第一图像，生成所述第一图像对应的生成图像的步骤，直至所述预设网络模型的训练情况满足预设条件，以得到所述图像处理模型。
在一个实施例中，所述第一图像中满足预设偏色条件的第一目标像素点的数量满足预设数量条件；所述预设偏色条件为第一图像中第一目标像素点的显示参数与第二图像中第二目标像素点的显示参数之间的误差满足预设误差条件，其中，所述第一目标像素点与所述第二目标像素点之间具有一一对应关系。
在一个实施例中，所述第一目标像素点为所述第一图像中任意一个像素点或者所述第一图像的目标区域中任意一个像素点。
在一个实施例中，所述训练图像集包括若干训练子图像集，每个训练子图像集包 括若干组训练样本图像组，若干训组训练图像组中的任意两组训练样本图像组中的第一图像的曝光度相同，若干组训练图像组中的每组训练样本图像组中的第二图像的曝光度均处于预设范围内，任意两个训练子图像集中的第一图像的曝光度不相同。
在一个实施例中，所述图像处理模型对应若干模型参数，每个模型参数均为根据所述训练图像集中的一个训练子图像集训练得到的，并且任意两个模型参数各自分别对应的训练子图像集互不相同。
在一个实施例中，所述预设网络模型包括下采样模块以及变换模块；所述第一生成模块具体用于：
将所述训练图像集中的第一图像输入所述下采样模块，通过所述下采样模块得到所述第一图像对应的双边网格以及所述第一图像对应的指导图像；以及将所述指导图像、所述双边网格以及所述第一图像输入所述变换模块，通过变换模块生成所述第一图像对应的生成图像，其中，所述指导图像的分辨率与所述第一图像的分辨率相同。
在一个实施例中，所述下采样模块包括下采样单元和卷积单元；所述第一生成模块具体用于：
将所述训练图像集中第一图像分别输入所述下采样单元以及所述卷积单元；以及通过所述下采样单元得到所述第一图像对应的双边网格，并通过所述卷积单元得到所述第一图像对应的指导图像。
在一个实施例中，所述变换模块包括切分单元以及变换单元，所述第一生成模块具体用于：
将所述指导图像输入所述切分单元，通过所述切分单元对所述双边网格进行切分，以得到所述第一图像中各像素点的颜色变换矩阵；
将所述第一图像以及所述第一图像中各像素点的颜色变换矩阵输入所述变换单元，通过所述变换单元生成所述第一图像对应的生成图像。
在一个实施例中，所述第一图像为通过屏下成像系统拍摄得到的图像。
在一个实施例中，所述屏下成像系统为屏下摄像头。
在一个实施例中，所述图像处理模型的生成装置还包括：
第一对齐模块，用于针对所述训练图像集中每组训练图像组，将该组训练图像组中的第一图像与所述第一图像对应的第二图像进行对齐处理，得到与所述第二图像对齐的对齐图像，并将所述对齐图像作为第一图像。
在一个实施例中，所述第一对齐模块具体用于：
针对所述训练图像集中每组训练图像组，获取该组训练图像组中的第一图像与所述第一图像对应的第二图像之间的像素偏差量；根据所述像素偏差量确定所述第一图像对应的对齐方式，并采用所述对齐方式将所述第一图像与第二图像进行对齐处理，得到与所述第二图像对齐的对齐图像；以及将所述对齐图像作为第一图像
在一个实施例中，所述第一对齐模块具体用于：
当所述像素偏差量小于等于预设偏差量阈值时，根据所述第一图像与所述第二图像的互信息，以所述第二图像为基准对所述第一图像进行对齐处理；
当所述像素偏差量大于所述预设偏差量阈值时，提取所述第一图像的第一像素点集和所述第二图像的第二像素点集，所述第一像素点集包含所述第一图像中的若干第一像素点，所述第二像素点集包括所述第二图像中的若干个第二像素点，所述第二像素点集中的第二像素点与所述第一像素点集中的第一像素点一一对应；针对第一像素点集中每个第一像素点，计算该第一像素点与其对应的第二像素点的坐标差值，并根据该第一像素点对应的坐标差值对该第一像素点进行位置变换，以将该第一像素点与该第一像素点对应的第二像素点对齐。
本公开第七方面提供了一种图像处理装置，其中，应用如第一方法所述的图像处理模型的生成方法或者如第六方面所述的图像处理模型的生成装置生成的图像处理模型，所述图像处理装置包括：
第一获取模块，用于获取待处理图像，并将所述待处理图像输入至所述图像处理模型；
第一处理模块，用于通过所述图像处理模型对所述待处理图像进行偏色处理，以得到所述待处理图像对应的处理后的图像。
所述图像处理模型对应若干模型参数，每个模型参数均为根据一个训练子图像集训练得到的，并且任意两个模型参数各自分别对应的训练子图像集互不相同。
在一个实施例中，所述第一获取模块具体用于：
获取待处理图像，并提取所述待处理图像的曝光度；根据所述曝光度确定所述待处理图像对应的模型参数，并采用所述模型参数更新所述图像处理模型的模型参数；以及将所述待处理图像输入至更新后的图像处理模型。
在一个实施例中，所述第一获取模块具体用于：
根据所述待处理图像中各像素点的R值、G值以及B值确定满足预设条件的第三像素点，其中，所述预设条件为R值、G值以及B值中至少一个值大于预设阈值；
根据满足预设条件的所有第三像素点确定所述待处理图像的高光区域，并根据所述高光区域确定所述待处理图像的曝光度。
在一个实施例中，所述第一获取模块具体用于：
获取所述满足预设条件的所有第三像素点所形成的连通区域，并在获取到的所有连通区域进行选取满足预设规则的目标区域，其中，所述预设规则为目标区域中的第三像素点的R值、G值和B值中大于预设阈值的R值、G值和/或B值的类型相同；
计算筛选得到的各目标区域分别对应的面积，并选取面积最大的目标区域作为高光区域。
在一个实施例中，所述第一获取模块具体用于：
计算所述高光区域的第一面积以及待处理图像的第二面积；
根据所述第一面积和第二面积的比值确定所述待处理图像对应的曝光度。
在一个实施例中，所述图像处理模型包括下采样模块以及变换模块；所述第一处理模块具体用于：
将所述待处理图像输入所述下采样模块，通过所述下采样模块得到所述待处理图像对应的双边网格以及所述待处理图像对应的指导图像，其中，所述指导图像的分辨率与所述待处理图像的分辨率相同；以及将所述指导图像、所述双边网格以及所述待处理图像输入所述变换模块，通过变换模块生成所述第一图像对应的处理后的图像。
在一个实施例中，所述下采样模块包括下采样单元和卷积单元；所述第一处理模块具体用于：
将所述待处理图像分别输入所述下采样单元以及所述卷积单元；通过所述下采样单元得到所述待处理图像对应的双边网格，并通过所述卷积单元得到所述待处理图像对应的指导图像。
在一个实施例中，所述变换模块包括切分单元以及变换单元，所述第一处理模块具体用于：
将所述指导图像输入所述切分单元，通过所述切分单元对所述双边网格进行切分，以得到所述待处理图像中各像素点的颜色变换矩阵；以及将所述待处理图像以及所述待处理图像中各像素点的颜色变换矩阵输入所述变换单元，通过所述变换单元生成所述待处理图像对应的处理后的图像。
在一个实施例中，所述图像处理装置还包括：
降噪处理单元，用于对所述处理后的图像进行锐化以及降噪处理，并将锐化以及 降噪处理后的图像作为所述待处理图像对应的处理后的图像。
本公开第八方面提供了一种图像处理模型的生成装置，其中，所述图像处理模型的生成装置包括：
第二生成模块，用于利用预设网络模型根据训练图像集中的第一图像生成所述第一图像对应的生成图像；其中，所述训练图像集包括多组训练图像组，每一组训练图像组包括第一图像和第二图像，第一图像为第二图像对应的具有重影的图像；
第二修正模块，用于利用所述预设网络模型根据所述第一图像对应的第二图像和所述第一图像对应的生成图像，对所述预设网络模型的模型参数进行修正，并继续执行根据训练图像集中的下一训练图像组中的第一图像，生成所述第一图像对应的生成图像的步骤，直至所述预设网络模型的训练情况满足预设条件，以得到已训练的图像处理模型。
在一个实施例中，所述预设网络模型包括编码器和解码器；所述第二生成模块具体用于：
将所述训练图像集中第一图像输入所述编码器，通过所述编码器得到所述第一图像的特征图像；以及将所述特征图像输入所述解码器，通过所述解码器输出所述生成图像，其中，所述特征图像的图像尺寸小于所述第一图像的图像尺寸；所述生成图像的图像尺寸等于第一图像的图像尺寸。
在一个实施例中，所述第二修正模块具体用于：
根据所述第一图像对应的第二图像和所述第一图像对应的生成图像分别计算所述预设网络模型对应的结构相似性损失函数值和内容双向损失函数值；根据所述结构相似性损失函数值和所述内容双向损失函数值得到所述预设网络模型的总损失函数值；以及基于所述总损失函数值训练所述预设网络模型，并继续执行根据训练图像集中的下一训练图像组中的第一图像，生成所述第一图像对应的生成图像的步骤，直至所述预设网络模型的训练情况满足预设条件，以得到已训练的图像处理模型。
在一个实施例中，所述第一图像为根据第二图像和点扩散函数生成的，其中，所述点扩散函数为根据屏下成像系统中的遮光结构生成的灰度图生成的。
在一个实施例中，所述第一图像为通过屏下成像系统拍摄得到的图像。
在一个实施例中，所述屏下成像系统为屏下摄像头。
在一个实施例中，所述图像处理模型的生成装置还包括：
第二对齐模块，用于针对所述训练图像集中每组训练图像组，将该组训练图像组 中的第一图像与所述第一图像对应的第二图像进行对齐处理，得到与所述第二图像对齐的对齐图像，并将所述对齐图像作为第一图像。
在一个实施例中，所述第二对齐模块具体用于：
针对所述训练图像集中每组训练图像组，获取该组训练图像组中的第一图像与所述第一图像对应的第二图像之间的像素偏差量；根据所述像素偏差量确定所述第一图像对应的对齐方式，并采用所述对齐方式将所述第一图像与所述第二图像进行对齐处理，得到与所述第二图像对齐的对齐图像；以及将所述对齐图像作为第一图像。
在一个实施例中，所述第二对齐模块具体用于：
当所述像素偏差量小于或等于预设偏差量阈值时，根据所述第一图像与所述第二图像的互信息，以所述第二图像为基准对所述第一图像进行对齐处理；
当所述像素偏差量大于所述预设偏差量阈值时，提取所述第一图像的第一像素点集和所述第二图像的第二像素点集，所述第一像素点集包含所述第一图像中的若干第一像素点，所述第二像素点集包括所述第二图像中的若干第二像素点，所述第二像素点集中的第二像素点与所述第一像素点集中的第一像素点一一对应；针对所述第一像素点集中每个第一像素点，计算该第一像素点与其对应的第二像素点的坐标差值，并根据该第一像素点对应的坐标差值对该第一像素点进行位置调整，以将该第一像素点与该第一像素点对应的第二像素点对齐。
本公开第九方面提供了一种图像处理装置，应用如第三方面所述的图像处理模型的生成方法或者应用如第八方面所述的图像处理模型的生成装置所生成的图像处理模型，所述图像处理装置包括：
第二获取模块，用于获取待处理图像，并将所述待处理图像输入至所述图像处理模型；
第二处理模块，用于通过所述图像处理模型对所述待处理图像进行去重影处理，以得到所述待处理图像对应的输出图像。
在一个实施例中，所述图像处理模型包括编码器和解码器；所述第二处理模块具体包括：
将所述待处理图像输入所述编码器，通过所述编码器得到所述待处理图像的特征图像；以及将所述特征图像输入所述解码器，通过所述解码器输出所述待处理图像对应的输出图像，其中，所述特征图像的图像尺寸小于所述待处理图像的图像尺寸；所述输出图像的图像尺寸等于所述待处理图像的图像尺寸。
在一个实施例中，所述图像处理装置还包括：
锐化模块，用于对所述输出图像进行锐化以及降噪处理，并将锐化以及降噪处理后的输出图像作为所述待处理图像对应的输出图像。
本公开第十方面提供了一种图像处理装置，其中，所述图像处理装置包括：
第三获取模块，用于获取待处理图像集，其中，所述待处理图像集包括多张图像；
第三生成模块，用于根据所述待处理图像集，生成所述待处理图像集对应的去噪图像；
第三处理模块，用于将所述去噪图像输入至以已训练的第一图像处理模型，通过所述图像处理模型对所述去噪图像进行去偏色处理，得到所述去噪图像对应的处理后图像；
第四处理模块，用于将所述处理后图像输入至以已训练的第二图像处理模型，通过所述第二图像处理模型对所述处理后图像进行去重影处理，以得到输出图像。
在一个实施例中，所述待处理图像集包括的多张图像中一张图像为基础图像，其余图像为基础图像的临近图像,所述第三生成模块具体用于：
将所述基础图像划分为若干基础图像块，分别确定各基础图像在各临近图像中对应的临近图像块；
确定各个基础图像块分别对应的权重参数集；其中，基础图像块对应的权重参数集包括第一权重参数和第二权重参数，第一权重参数为基础图像块的权重参数，第二权重参数为临近图像中与基础图像块对应的临近图像块的权重参数；
根据所述待处理图像集以及各个基础图像块分别对应的权重参数集，确定去噪图像。
在一个实施例中，所述待处理图像集的图像数量为根据所述待处理图像集对应的拍摄参数确定的。
在一个实施例中，所述基础图像的图像清晰度大于或等于所述临近图像的图像清晰度。
在一个实施例中，所述第三生成模块具体用于：
针对每个基础图像块，确定该基础图像块对应的各临近图像块的第二权重参数，以及，获取该基础图像块对应的第一权重参数，以得到该基础图像块对应的权重参数集。
在一个实施例中，所述第三生成模块具体用于：
针对每个临近图像块，计算该基础图像块与该临近图像块的相似度值；
根据所述相似度值计算该临近图像块的第二权重参数。
在一个实施例中，所述第三生成模块具体用于：
当所述相似度值小于或等于第一阈值时，将第一预设参数作为该临近图像块的第二权重参数；
当所述相似度值大于第一阈值，且小于或等于第二阈值时，根据所述相似度值、所述第一阈值及所述第二阈值计算该临近图像块的第二权重参数；
当所述相似度值大于第二阈值时，将预设第二预设参数作为该临近图像块的第二权重参数。
在一个实施例中，所述第一阈值和第二阈值均为根据该基础图像块对应临近图像块的相似度值确定的。
在一个实施例中，所述图像处理装置还包括：
空域降噪模块，用于对所述去噪图像进行空域降噪，并将空域降噪后得到的图像作为去噪图像。
在一个实施例中，所述第一图像处理模型为基于第一训练图像集训练得到，所述第一训练图像集包括多组训练图像组，每一组训练图像组包括第一图像和第二图像，第一图像为对应第二图像的偏色图像。
在一个实施例中，所述第一图像为通过屏下成像系统拍摄得到的图像。
在一个实施例中，所述图像处理装置还包括：
第三对齐模块，用于针对所述第一训练样本集中每组训练图像组，将该组训练图像组中的第一图像与所述第一图像对应的第二图像进行对齐处理，得到与所述第二图像对齐的对齐图像，并将所述对齐图像作为第一图像。
在一个实施例中，所述第一训练图像集包括若干训练子图像集，每个训练子图像集包括若干组训练样本图像组，若干训组训练图像组中的任意两组训练样本图像组中的第一图像的曝光度相同，若干组训练图像组中的每组训练样本图像组中的第二图像的曝光度均处于预设范围内，任意两个训练子图像集中的第一图像的曝光度不相同。
在一个实施例中，所述第一图像处理模型对应若干模型参数，每个模型参数均为根据所述第一训练图像集中一个训练子图像集训练得到的，并且任意两个模型参数各自分别对应的训练子图像集互不相同。
在一个实施例中，所述第三处理模块具体用于：
提取所述去噪图像的曝光度；根据所述曝光度确定所述去噪图像对应的模型参数， 并采用所述模型参数更新所述第一图像处理模型的模型参数；以及将所述去噪图像输入至更新后的第一图像处理模型。
在一个实施例中，所述第三处理模块具体用于：
根据所述去噪图像中各像素点的R值、G值以及B值确定满足预设条件的第三像素点，其中，所述预设条件为R值、G值以及B值中至少一个值大于预设阈值；根据满足预设条件的所有第三像素点确定所述去噪图像的高光区域，并根据所述高光区域确定所述去噪图像的曝光度。
在一个实施例中，所述第三处理模块具体用于：
获取所述满足预设条件的所有第三像素点所形成的连通区域，并在获取到的所有连通区域进行选取满足预设规则的目标区域，计算筛选得到的各目标区域分别对应的面积，并选取面积最大的目标区域作为高光区域，其中，所述预设规则为目标区域中的第三像素点的R值、G值和B值中大于预设阈值的R值、G值和/或B值的类型相同。
在一个实施例中，所述第三处理模块具体用于：
计算所述高光区域的第一面积以及去噪图像的第二面积；以及根据所述第一面积和第二面积的比值确定所述去噪图像对应的曝光度。
在一个实施例中，所述第一图像处理模型包括下采样模块以及变换模块；所述第三处理模块具体用于：
将所述去噪图像输入所述下采样模块，通过所述下采样模块得到所述去噪图像对应的双边网格以及所述去噪图像对应的指导图像，其中，所述指导图像的分辨率与所述去噪图像的分辨率相同；
将所述指导图像、所述双边网格以及所述去噪图像输入所述变换模块，通过变换模块生成所述去噪图像对应的处理后图像。
在一个实施例中，所述下采样模块包括下采样单元和卷积单元；所述第三处理模块具体用于：
将所述去噪图像分别输入所述下采样单元以及所述卷积单元；
通过所述下采样单元得到所述去噪图像对应的双边网格，并通过所述卷积单元得到所述去噪图像对应的指导图像。
在一个实施例中，所述变换模块包括切分单元以及变换单元，所述第三处理模块具体用于：
将所述指导图像输入所述切分单元，通过所述切分单元对所述双边网格进行切分， 以得到所述去噪图像中各像素点的颜色变换矩阵；
将所述去噪图像以及所述去噪图像中各像素点的颜色变换矩阵输入所述变换单元，通过所述变换单元生成所述去噪图像对应的处理后图像。
在一个实施例中，所述第二图像处理模型为基于第二训练图像集训练得到，所述第二训练图像集包括多组训练图像组，每一组训练图像组包括第三图像和第四图像，第三图像为第四图像对应的具有重影的图像。
在一个实施例中，所述第三图像为根据第四图像和点扩散函数生成的，其中，所述点扩散函数为根据屏下成像系统中的遮光结构生成的灰度图生成的。
在一个实施例中，所述第三图像为通过屏下成像系统拍摄得到的图像。
在一个实施例中，所述屏下成像系统为屏下摄像头。
在一个实施例中，所述图像处理装置还包括：
第四对齐模块，用于针对所述第二训练图像集中每组训练图像组，将该组训练图像组中的第三图像与所述第三图像对应的第四图像进行对齐处理，得到与所述第四图像对齐的对齐图像，并将所述对齐图像作为第三图像。
在一个实施例中，所述第二图像处理模型包括编码器和解码器；所述第四处理模块具体用于：
将所述处理后图像输入所述编码器，通过所述编码器得到所述处理后图像的特征图像；以及将所述特征图像输入所述解码器，通过所述解码器输出所述处理后图像对应的输出图像，其中，所述特征图像的图像尺寸小于所述处理后图像的图像尺寸；所述输出图像的图像尺寸等于所述处理后图像的图像尺寸。
在一个实施例中，所述第三对齐模块和/或所述第四对齐模块均具体用于：
获取训练图像组中的基准图像和参考图像，并计算所述参考图像与所述基准图像之间的像素偏差量；以及根据所述像素偏差量确定所述参考图像对应的对齐方式，并采用所述对齐方式将所述参考图像与所述基准图像进行对齐处理，其中，当基准图像为第二图像时，参考图像为第一图像；当基准图像为第四图像时，参考图像为第三图像。
在一个实施例中，所述第三对齐模块和/或所述第四对齐模块均具体用于：
当所述像素偏差量小于或等于预设偏差量阈值时，根据所述参考图像与所述基准图像的互信息，以所述基准图像为基准对所述参考图像进行对齐处理；
当所述像素偏差量大于所述预设偏差量阈值时，提取所述参考图像的基准像素点集和所述基准图像的参考像素点集，所述基准像素点集包含所述参考图像中的若干参考 像素点，所述基准像素点集包括所述基图像中的若干基准像素点，所述参考像素点集中的参考像素点与所述基准像素点集中的基准像素点一一对应；针对所述基准像素点集中每个基准像素点，计算该基准像素点与其对应的参考像素点的坐标差值，并根据该参考像素点对应的坐标差值对该参考像素点进行位置调整，以将该参考准像素点与该参考像素点对应的基准像素点对齐。
在一个实施例中，所述图像处理装置还包括：
锐化降噪模块，用于对所述处理后图像进行锐化以及降噪处理，并将锐化以及降噪处理后的处理后图像作为所述输出图像。
本公开第十一方面提供了一种计算机可读存储介质，所述计算机可读存储介质存储有一个或者多个程序，所述一个或者多个程序可被一个或者多个处理器执行，以实现如上任一所述的图像处理模型的生成方法和/或图像处理方法中的步骤。
本公开第十二方面提供了一种终端设备，其包括：处理器、存储器及通信总线；所述存储器上存储有可被所述处理器执行的计算机可读程序；
所述通信总线实现处理器和存储器之间的连接通信；
所述处理器执行所述计算机可读程序时实现如上任一所述的图像处理模型的生成方法和/或图像处理方法中的步骤。
有益效果：与现有技术相比，本公开提供了一种图像处理模型的生成方法、处理方法、存储介质及终端，所述生成方法通过将预设的训练图像集中第一图像输入预设网络模型，通过所述预设网络模型生成的生成图像以及第一图像对应的第二图像对预设模型进行序列，得到图像处理模型。所述图像处理模型为通过对具有多组训练图像组的训练图像集的去偏色过程进行深度学习得到，每一组训练图像组包括第一图像和第二图像，第一图像为对应第二图像的偏色图像。由此可知，本公开是采用基于训练图像集进行深度学习得到已训练的图像处理模型进行偏色处理，这样可以快速对图像进行偏色调整，提高图像的色彩质量，从而提高图像质量。
附图说明
图1为公开提供的一种图像处理模型的生成方法的应用场景示意图。
图2为公开提供一种图像处理模型的生成方法的流程图。
图3为公开提供的一种图像处理模型的生成方法的预设网络模型的一个原理示意图。
图4为公开提供的一种图像处理模型的生成方法的流程示意图。
图5为公开提供的第一图像的一个示例图。
图6为公开提供的第二图像的一个示例图。
图7为公开提供的确定对齐方式过程的流程图。
图8为公开提供的一种图像处理模型的生成方法的中步骤S10的流程图。
图9为公开提供的一种图像处理模型的生成方法的中步骤S11的流程图。
图10为公开提供的一种图像处理模型的生成方法的中步骤S12的流程图。
图11为公开提供的一种图像处理方法的流程图。
图12为公开提供的一种图像处理方法中步骤A100的流程图。
图13为公开提供的待处理图像的一个示例图。
图14为公开提供的待处理图像对应的处理后的图像的一个示例图。
图15为公开提供的一种图像处理模型的生成方法的流程图。
图16为公开提供的一种图像处理模型的生成方法的流程示意图。
图17为公开提供的第一图像的一个示例图。
图18为公开提供的第二图像的一个示例图。
图19为公开提供的一种图像处理模型的生成方法的信号线路的一个示例图。
图20为公开提供的一种图像处理模型的生成方法的灰度图的一个示例图。
图21为公开提供的一种图像处理模型的生成方法的中步骤N10的流程图。
图22为公开提供的一种图像处理模型的生成方法的中预设网络模型的结构示意图。
图23为公开提供的一种图像处理模型的生成方法的中步骤N20的流程图。
图24为公开提供的一种图像处理方法的流程图。
图25为公开提供的待处理图像的一个示例图。
图26为公开提供的待处理图像对应的输出图像的一个示例图。
图27为本公开提供的图像处理方法的一个实施例的流程图。
图28为本公开提供的图像处理方法的一个实施例中步骤H20的流程图。
图29为本公开提供的图像处理方法的一个实施例中临近图像块的获取过程的流程图。
图30为本公开提供的图像处理方法的一个实施例中指定区域的一个示例图。
图31为本公开提供的图像处理方法的一个实施例中第二权重参数的计算过程的流程图。
图32为本公开提供的图像处理方法的一个实施例中第二图像处理模型的训练过程 的流程图。
图33为本公开提供的图像处理方法的一个实施例中步骤L200的流程图。
图34为本公开提供的一种图像处理模型的生成装置的一实施例的结构原理图。
图35为本公开提供的一种图像处理装置的一实施例的结构原理图。
图36为本公开提供的一种图像处理模型的生成装置的一实施例的结构原理图。
图37为本公开提供的一种图像处理装置的一实施例的结构原理图。
图38为本公开提供的一种图像处理装置的一实施例的结构原理图。
图39为本公开提供的终端设备的结构原理图。
具体实施方式
本公开提供一种图像处理模型生成方法、处理方法、存储介质及终端，为使本公开的目的、技术方案及效果更加清楚、明确，以下参照附图并举实施例对本公开进一步详细说明。应当理解，此处所描述的具体实施例仅用以解释本公开，并不用于限定本公开。
本技术领域技术人员可以理解，除非特意声明，这里使用的单数形式“一”、“一个”、“所述”和“该”也可包括复数形式。应该进一步理解的是，本公开的说明书中使用的措辞“包括”是指存在所述特征、整数、步骤、操作、元件和/或组件，但是并不排除存在或添加一个或多个其他特征、整数、步骤、操作、元件、组件和/或它们的组。应该理解，当我们称元件被“连接”或“耦接”到另一元件时，它可以直接连接或耦接到其他元件，或者也可以存在中间元件。此外，这里使用的“连接”或“耦接”可以包括无线连接或无线耦接。这里使用的措辞“和/或”包括一个或更多个相关联的列出项的全部或任一单元和全部组合。
本技术领域技术人员可以理解，除非另外定义，这里使用的所有术语(包括技术术语和科学术语)，具有与本公开所属领域中的普通技术人员的一般理解相同的意义。还应该理解的是，诸如通用字典中定义的那些术语，应该被理解为具有与现有技术的上下文中的意义一致的意义，并且除非像这里一样被特定定义，否则不会用理想化或过于正式的含义来解释。
发明人经过研究发现，为了实现终端设备的全面屏，需要将终端设备的前置摄像头安装在显示面板下方。而现有显示面板普遍包括基板以及偏光片等，而当光线通过显示面板时，显示面板一方面折射光线而使得光线透射率低，另一方显示面板会吸收光线，这样会影响拍摄得到的图像质量，例如，拍摄得到的图像色彩与拍摄场景不符、图像噪 声增多以及图像影像模糊等。
为了解决上述问题，在本公开实施例中，采用第二图像作为目标图像，并采用第二图像的偏色图像(指的是第一图像)作为训练样本图像，将第一图像输入预设网络模型，通过所述预设网络模型输出第一图像对应的生成图像，再根据第一图像对应的第二图像和第一图像对应的生成图像对所述预设网络模型进行训练，以得到已训练的图像处理模型。可见，本发明实施例中，通过对预设网络模型进行深度学习来得到图像处理模型，使得训练得到的图像处理模型可以去除图像中偏色，进而可以通过训练得到的图像处理模型对屏下成像系统拍摄得到的图像进行处理，以去除图像携带的偏色，提高屏下成像系统拍摄图像的图像质量。
举例说明，本发明实施例可以应用到如图1所示的场景。在该场景中，首先，终端设备1可以采集训练图像集，并将所述训练图像集输入服务器2，以使得服务器2依据所述训练图像集对预设网络模型进行训练。服务器2可以预先存储有预设网络模型，并响应终端设备1的输入的训练图像集，将所述训练图像集中的第一图像作为输入项输入预设网络模型，然后，获取所述预设网络模型输出的生成图像，通过所述第一图像对应的第二图像以及第一图像对应的生成图像对所述预设网络模型进行修正，并继续执行将训练图像集中第一图像输入预设网络模型的操作，并继续执行根据所述训练图像集中第一图像，生成所述第一图像对应的生成图像的步骤，直至所述预设网络模型的训练情况满足预设条件，以得到所述图像处理模型。
可以理解的是，在上述应用场景中，虽然将本发明实施方式的动作描述为部分由终端设备1执行、部分由服务器2执行，但是这些动作可以完全由服务器执行，或者完全由终端设备1执行。本发明在执行主体方面不受限制，只要执行了本发明实施方式所公开的动作即可。
进一步，在获取到已训练的图像处理模型后，可以将所述已训练的图像处理模型用于处理通过具有屏下成像系统(如，屏下摄像头)的终端设备拍摄的照片。例如，将通过具有屏下成像系统(如，屏下摄像头)的终端设备拍摄的照片作为输入项输入所述已训练的图像处理模型，通过所述已训练的图像处理模型对该照片进行处理，以得到处理后的照片，从而可以快速对所述照片进行去偏色处理，以提高屏下摄像头拍摄的照片的图像质量。当然，在实际应用中，所述已训练的图像处理模型可作为一个去偏色功能模块配置于具有屏下成像系统(如，屏下摄像头)的终端设备，当具有屏下成像系统(如，屏下摄像头)的终端设备拍摄到照片时，启动所述去偏色功能模块，通过所述去偏色功 能模块对该照片进行去偏色处理，使得去具有屏下成像系统(如，屏下摄像头)的终端设备输出去除偏色后的照片，使得具有屏下成像系统(如，屏下摄像头)的终端设备可以直接输出经过去偏色处理的图像。
需要注意的是，上述应用场景仅是为了便于理解本公开而示出，本公开的实施方式在此方面不受任何限制。相反，本公开的实施方式可以应用于适用的任何场景。
下面结合附图，通过对实施例的描述，对公开内容作进一步说明。
实施例一
本实施例提供了一种图像处理模型的生成方法，如图2和4所示，所述方法包括：
S10、预设网络模型根据训练图像集中第一图像，生成所述第一图像对应的生成图像。
具体地，所述预设网络模型为深度学习网络模型，并且该预设网络模型基于预设的训练图像集进行训练的。所述训练图像集包括多组具有不同图像内容的训练图像组，每一组训练图像组均包括第一图像和第二图像，第一图像为对应第二图像的偏色图像。其中，所述第一图像为对应第二图像的偏色图像指的是第一图像与第二图像相对应，第一图像和第二图像呈现同一图像场景，并且所述第一图像中满足预设偏色条件的第一目标像素点的数量满足预设数量条件。可以理解的是，第二图像为正常显示图像，第一图像中存在若干满足预设偏色条件的第一目标像素点，并且若干第一目标像素点的数量满足预设条件。例如，第二图像为如图6所示的图像，第一图像为如图5所示的图像，其中，第一图像的图像内容与第二图像的图像内容相同，但在第一图像中苹果对应的呈现的色彩与第二图像中苹果呈现的色彩不同，例如，在图5中，第一图像中苹果在第一图像中呈现的色彩为绿色偏蓝色；在图6中，第二图像中苹果在第二图像中呈现的色彩为深绿色。
进一步，所述预设偏色条件为第一图像中第一目标像素点的显示参数与第二图像中第二目标像素点的显示参数之间的误差满足预设误差条件，所述第一目标像素点与所述第二目标像素点之间具有一一对应关系。其中，所述显示参数为用于反映像素点对应的色彩的参数，例如，所述显示参数可以为像素点的RGB值，其中，R值为红色通道值、G值为绿色通道值、B值为蓝色通道值；也可以为像素点的hsl值，其中，h值为色相值，l为亮度值，s为饱和度值。此外，当显示参数为像素点的RGB值时，第一图像和第二图像中任一像素点的显示参数均包括R值、G值和B值三个显示参量；当显示显示为像素点的hls值，第一图像和第二图像中任一像素点的显示参数均包括h值、l值和s值 三个显示参量。
所述预设误差条件用于衡量第一目标像素点是否为满足预设偏色条件的像素点，其中，所述预设误差条件为预设误差阈值，误差满足预设误差条件为误差大于或等于预设误差阈值。此外，所述显示参数包括若干显示参数，例如显示参数为像素点的RGB值，显示参数包括R值、G值和B值三个显示参，当显示参数为像素点的hsl值时，显示参数包括h值、l值和s值三个显示参量。由此，所述误差可以为显示参数中各显示参量的误差最大值，也可以为显示参数中各显示参量的误差的最小值，还可以是所有显示参量的误差平均值。例如，这里以显示参数为像素点的RGB值进行说明，第一目标像素点的显示参数为(55,86,108)，第二目标像素点的显示参数为(58,95,120)，那么各显示参量的误差值分为3,9以及12；由此，当第一目标像素点与第二目标像素点的误差为各显示参量的误差最大值时，该误差为12；当第一目标像素点与第二目标像素点的误差为各显示参量的误差最小值时，该误差为3；当第一目标像素点与第二目标像素点的误差为所有显示参量的误差平均值时，该误差为8；需要说明的是，在一种可能的实现方式中，也可以仅参考RGB中一个参数(例如R、G或B)或任意两个参数的误差，当显示参数为像素点的hsl值时，同理。
进一步，用于与第一目标像素点计算误差的第二目标像素点与第一目标显示点之间存在一一对应关系。可以理解的是，对于第一目标像素点，第二图像中存在唯一的第二目标像素点与第一目标像素点对应，其中，第一目标像素点与第二目标像素点对应指的是第一目标像素点在第一图像中的像素位置，与第二目标像素点在第二图像中的像素位置相对应。例如，第一目标像素点在第一图像中的像素位置为(5,6)，第二目标像素点在第二图像中的像素位置为(5,6)。此外，所述第一目标像素点可以为第一图像中任一像素点，也可以是第一图像中目标区域中任一像素点，其中，所述目标区域可以为第一图像中物品所处区域，其中，所述物品所处区域可以为人或物在图像中对应的区域。例如，如图5所示，所述目标区域为第一图像中苹果所处区域。也就是说，第一图像中可以全部像素点与第二图像相比较出现偏色，即第一图像中全部像素点均为第一目标像素点，也可以只有一部分像素点与第二图像相比较出现偏色，即第一图像中部分像素点为第一目标像素点，例如，当一图像中只有部分区域(例如图中苹果对应的区域)中的像素点与第二图像相比较出现偏色时，该图像也可以理解为对应第二图像的偏色图像，即第一图像。
进一步，所述第一图像和第二图像相对应指的是第一图像的图像尺寸与第二图像 的图像尺寸相等，并且所述第一图像和第二图像对应相同的图像场景。所述第一图像和第二图像对应相同的图像场景指的是第一图像携带的图像内容与第二图像携带的图像内容的相似度达到预设阈值，所述第一图像的图像尺寸与第二图像的图像尺寸相同，以使得当第一图像和第二图像重合时，第一图像携带的物体对第二图像中与其对应的物体的覆盖率达到预设条件。其中，所述预设阈值可以为99％，所述预设条件可以为99.5％等。在实际应用中，所述第一图像可以是通过屏下成像系统拍摄得到；所述第二图像可以是通过正常屏上成像系统(如，屏上摄像头)拍摄得到，也可以是通过网络(如，百度)获取到，还可以是通过其他外部设备(如，智能手机)发送的。
在本实施例的一个可能实现方式中，所述第二图像为通过正常屏上成像系统拍摄得到，所述第二图像和第一图像的拍摄参数相同。其中，所述拍摄参数可以包括成像系统的曝光参数，所述曝光参数可以包括光圈、快门速度、感光度、对焦以及白平衡等。当然，在实际应用中，所述拍摄参数还可以包括环境光、拍摄角度以及拍摄范围等。例如，所述第一图像为如图5所示的通过屏下摄像头拍摄一场景得到的图像，第二图像为如图6所示的通过屏上摄像头拍摄该场景得到的图像。
进一步，在本实施例的一个实现方式中，为了减少第一图像和第二图像的图像差异对预设网络模型训练的影响，所述第一图像的图像内容和第二图像的图像内容可以完全相同。即所述第一图像和第二图像具有相同图像内容指的是第一图像具有的物体内容与第二图像具有的物体内容相同，所述第一图像的图像尺寸与第二图像的图像尺寸相同，并且当第一图像和第二图像重合时，第一图像具有的物体可以覆盖第二图像中与其对应的物体。
举例说明：所述第一图像的图像尺寸为400*400，第一图像的图像内容为一个圆，并且第一图像中圆的圆心在第一图像中的位置为(200，200)、半径长度为50像素。那么，所述第二图像的图像尺寸为400*400，第二图像的图像内容也为一个圆，第二图像中圆的圆心在第二图像中的位置为(200，200)，半径为50像素；当第一图像放置于第二图像上并与第二图像重合时，所述第一图像覆盖所述第二图像，并且第一图像中的圆与第二图像的圆上下重叠。
进一步，当第二图像为通过正常屏上成像系统拍摄得到时，由于第一图像和第二图像是通过两个不同的成像系统拍摄得到，而在更换成像系统时，可能会造成屏上成像系统和屏下成像系统的拍摄角度和/或拍摄位置的变化，使得第一图像和第二图像在空间上存在不对齐的问题。由此，在本实施例的一个可能实现方式中，在通过屏上成像系 统拍摄第二图像以及通过屏下成像系统拍摄第一图像时，可以将屏上成像系统和屏下成像系统设置于同一固定架上，将屏上成像系统和屏下成像系统并排布置在固定架上，并保持屏上成像系统和屏下成像系统相接触。同时，将屏上成像系统和屏下成像系统分别与无线设置(如，蓝牙手表等)相连接，通过无线设置触发屏上成像系统和屏下成像系统的快门，这样可以减少拍摄过程中屏上成像系统和屏下成像系统的位置变化，提高第一图像和第二图像在空间上的对齐性。当然，屏上成像系统和屏下成像系统的拍摄时间和拍摄范围均相同。
此外，虽然在第一图像和第二图像的拍摄过程中，可以通过固定屏下成像系统和屏上成像系统的拍摄位置、拍摄角度、拍摄时间以及曝光系数等。但是，由于环境参数(如，光线强度、风吹动成像系统等)，屏下成像系统拍摄得到的第一图像和屏上成像系统拍摄得到的第二图像在空间上还可能存在不对齐的问题。由此，在将训练图像集中第一图像输入预设网络模型之前，可以训练图像集中的各训练图像组中的第一图像和第二图像进行对齐处理，从而在本实施例的一个实现方式中，所述预设网络模型根据训练图像集中第一图像，生成所述第一图像对应的生成图像之前还包括
M10、针对所述训练图像集中每组训练图像组，将该组训练图像组中的第一图像与所述第一图像对应的第二图像进行对齐处理，得到与所述第二图像对齐的对齐图像，并将所述对齐图像作为第一图像。
具体地，所述针对所述训练图像集中每组训练图像组指的是对训练图像集中每一组训练图像组均执行对齐处理，所述对齐处理可以是在获取到训练图像集之后，分别对每一组训练图像组进行对齐处理，以得到对齐后的训练图像组，并在所有组训练图像组对齐后执行将每一组训练图像组中的第一图像输入预设网络模型的步骤；当然也可以是在将每一组训练图像组中的第一图像输入预设网络模型之前，对该组训练图像组进行对齐处理，以得到该组训练图像对应的对齐后的训练图像组，之后将对齐后的训练图像组中的第一图像输入预设网络模型。在本实施例中，所述对齐处理是在获取到训练图像集后，分别对每一组训练图像组进行，并在所有训练图像组完成对齐处理后，在执行将训练图像集中第一图像输入预设网络模型的操作。
进一步，所述将该组训练图像组中的第一图像与所述第一图像对应的第二图像进行对齐处理指的是将以第二图像为基准，将第一图像中像素点与第二图像中与其对应的像素点对齐，以使得第一图像中像素点与第二图像中像素点的对齐率可以达到预设值，例如，99％等。其中，所述第一图像中像素点与第二图像中与其对应的像素点对齐指的 是：对于第一图像中的第一像素点和第二图像中与第一像素点相对应的第二像素点，若第一像素点对应的像素坐标与第二像素点对应的像素坐标相同，那么第一像素点与第二像素点对齐；若第一像素点对应的像素坐标与第二像素点对应的像素坐标不相同那么第一像素点与第二像素点对齐。所述对齐图像指的通过对第一图像进行对齐处理得到图像，并且对齐图像中每个像素点与第二图像中其对应的像素点的像素坐标相同。此外，在得到对齐图像后，采用所述对齐图像替换其对应的第一图像以更新训练图像组，以使得更新后的训练图像组中的第一图像和第二图像在空间上对齐。
进一步，由于不同组训练图像组中的第一图像和第二图像的对齐程度不同，从而可以在实现对齐的基础上，针对不同对齐程度的第一图像和第二图像可以采用不同的对齐方式，以使得各组训练图像组均可以采用复杂度低的对齐方式进行对齐处理。由此，在本实施例的一个实现方式中，如图7所示，所述将该组训练图像组中的第一图像与所述第一图像对应的第二图像进行对齐处理具体包括：
M11、获取该组训练图像组中的第一图像与所述第一图像对应的第二图像之间的像素偏差量；
M12、根据所述像素偏差量确定所述第一图像对应的对齐方式，并采用所述对齐方式将所述第一图像与所述第二图像进行对齐处理。
具体地，所述像素偏差量指的是第一图像中第一像素点与第二图像中与该第一像素点对应的第二像素点不对齐的第一像素点的总数量。所述像素偏差量可以通过获取第一图像中各第一像素点的第一坐标，以及第二图像中各第二像素点的第二坐标，然后将第一像素点的第一坐标与其对应的第二像素点的第二坐标进行比较，若第一坐标与第二坐标相同，则判定第一像素点与其对应的第二像素点对齐；若第一坐标与第二坐标不相同，则判定第一像素点与其对应的第二像素点不对齐，最后获取所有不对齐的第一像素点的总数量，以得到所述像素偏差量。例如，当所述第一图像中的第一像素点的第一坐标为(200,200)，第二图像中与所述第一像素点对应的第二像素点的第二坐标为(201,200)时，所述第一像素点与第二像素点不对齐，不对齐的第一像素点的总数量加一；当所述第一图像中的第一像素点的第一坐标为(200,200)，第二图像中与所述第一像素点对应的第二像素点的第二坐标为(200,200)时，所述第一像素点与第二像素点对齐，不对齐的第一像素点的总数量不变。
进一步，为了确定像素偏差量与对齐方式的对应关系，可以需要设置偏差量阈值，在获取到第一图像的像素偏差量时，可以通过将获取到的像素偏差量与预设偏差量阈值 进行比较，以确定像素偏差量对应的对齐方式。由此，在本实施例的一个实现方式中，所述根据所述像素偏差量确定所述述第一图像对应的对齐方式，并采用所述对齐方式将所述第一图像与所述第二图像进行对齐处理具体包括：
M121、当所述像素偏差量小于或等于预设偏差量阈值时，根据所述第一图像与所述第二图像的互信息，以所述第二图像为基准对所述第一图像进行对齐处理；
M122、当所述像素偏差量大于所述预设偏差量阈值时，提取所述第一图像的第一像素点集和所述第二图像的第二像素点集，所述第一像素点集包含所述第一图像中的若干第一像素点，所述第二像素点集包括所述第二图像中的若干第二像素点，所述第二像素点集中的第二像素点与所述第一像素点集中的第一像素点一一对应；针对所述第一像素点集中每个第一像素点，计算该第一像素点与其对应的第二像素点的坐标差值，并根据该第一像素点对应的坐标差值对该第一像素点进行位置调整，以将该第一像素点与该第一像素点对应的第二像素点对齐。
具体地，所述预设偏差量阈值为预先设置，例如，预设偏差量阈值为20。所述当所述像素偏差量小于或等于预设偏差量阈值时指的是当将所述像素偏差量与所述预设偏差量阈值时，所述像素偏差量小于等于预设偏差量阈值。而当所述像素偏差量小于等于预设偏差量阈值时，说明第一图像和第二图像在空间上的偏差较小，此时可以采用根据所述第一图像与所述第二图像的互信息对第一图像和第二图像进行对齐。在本实施例中，以所述第一图像和其对应的第二图像之间互信息对第一图像和第二图像进行对齐的过程可以采用图像配准方法，所述图像配准方法中以互信息作为度量准则，通过优化器对度量准则迭代进行优化以得到对齐参数，通过所述配准所述对齐参数的配准器将第一图像与第二图像进行对齐，这保证第一图像与第二图像的对齐效果的基础，降低了第一图像与第二图像对齐的复杂性，从而提高了对齐效率。在本实施例中，所述优化器主要采用平移和旋转变换，以通过所述平移和旋转变换来优化度量准则。
进一步，所述像素偏差量大于所述预设偏差量阈值，说明第一图像和第二图像在空间上不对齐程度较高，此时需要着重考虑对齐效果。从而此时可以采用通过选取第一图像中的第一像素点集和第二图像中第二像素点集的方式对第一图像和第二图像进行对齐。所述第一像素点集的第一像素点与第二像素点集中第二像素点一一对应，以使得对于第一像素点集中的任一第一像素点，在第二像素点集中均可以找到一个第二像素点，所述第二像素点在第二图像中的位置与第一像素点在第一图像中的位置相对应。此外，所述第一像素点集和第二像素点集可以是在获取到第一像素点集/第二像素点集后，根 据第一像素点与第二像素点的对应关系确定第二像素点集/第一像素点集，例如，所述第一像素点集通过在第一图像中随机选取多个第一像素点的方式生成，第二像素点则是根据第一像素点集包含的各第一像素点确定的。
同时在本实施例中，所述第一像素点集和第二像素点集均是通过尺度不变特征变换(Scale-invariant feature transform，sift)的方式获取得到，即所述第一像素点集中第一像素点为第一图像中第一sift特征点，所述第二像素点集中第二像素点为第二图像的第二sift特征点。相应的，所述计算该第一像素点与其对应的第二像素点的坐标差值为将第一像素点中第一sift特征点与第二像素点集中第二sift特征点进行点对点匹配，以得到各第一sift特征点与其对应的各第二sift特征点的坐标差值，并根据该第一sift特征点对应的坐标差值对该第一sift特征点进行位置变换，以将该第一像素点与该第一sift特征点对应的第二sift特征点对齐，从而使得第一图像中第一sift特征点与第二图像中第二sift特征点位置相同，从而实现了第一图像与第二图像的对齐。
进一步，在本实施例的一个实现方式中，如图3、4和图7所示，所述预设网络模型包括下采样模块100以及变换模块200，相应的，所述预设网络模型根据训练图像集中第一图像，生成所述第一图像对应的生成图像可以具体包括：
S11、将所述训练图像集中第一图像输入所述下采样模块，通过所述下采样模块得到所述第一图像对应的双边网格以及所述第一图像对应的指导图像，其中，所述指导图像的分辨率与所述第一图像的分辨率相同；
S12、将所述指导图像、所述双边网格以及所述第一图像输入所述变换模块，通过变换模块生成所述第一图像对应的生成图像。
具体地，所述双边网格10为在二维图像的像素坐标中增加一维代表像素强度的维度而得到的三维双边网格，其中，所述三维双边网络的三维分别为二维图像的像素坐标中横轴和纵轴，以及增加的代表像素强度的维度。所述指导图像为通过对第一图像进行像素级操作得到的，所述指导图像50的分辨率与所述第一图像的分辨率相同，例如，所述指导图像50为所述第一图像对应的灰阶图像。
进一步，由于所述下采样模块100用于输出第一图像对应的双边网格10和指导图像50，从而所述下采样模块100包括下采样单元70和卷积单元30，所述下采样单元70用于输出所述第一图像对应的双边网格10，所述卷积单元30用于输出所述第一图像对应的指导图像50。相应的，如图3、4和图8所示，所述将所述训练图像集中第一图像 输入所述下采样模块，通过所述下采样模块得到所述第一图像对应的双边网格参数以及所述第一图像对应的指导图像具体包括：
S111、将所述训练图像集中第一图像分别输入所述下采样单元以及所述卷积单元；
S112、通过所述下采样单元得到所述第一图像对应的双边网格，并通过所述卷积单元得到所述第一图像对应的指导图像。
具体地，所述下采样单元70用于对第一图像进行下采样，以得到第一图像对应的特征图像，并根据所述特征图像生成所述第一图像对应的双边网格，特征图像的空间通道数大于第一图像的空间通道数。所述双边网格是根据所述特征图像的局部特征和全局特征生成的，其中，所述局部特征为从图像局部区域中抽取的特征，例如，边缘、角点、线、曲线和属性区域等，在本实施例中，所述局部特征可以为区域颜色特征。所述全局特征指的是表示整幅图像属性的特征，例如，颜色特征、纹理特征和形状特征。在本实施例中，所述全局特征可以为整幅图像的颜色特征。
进一步，在本实施例的一个可能实现方式中，所述下采样单元70包括下采样层、局部特征提取层、全局特征提取层以及全连接层，所述局部特征提取层连接于所述下采样层与全连接层之间，所述全局特征提取层连接于下采样层与全连接层之间，并且所述全局特征提取层与所述局部特征提取层并联。由此可知，第一图像作为输入项输入下采样层，经过下采样层输出特征图像；下采样层的特征图像分别输入至局部特征提取层和全局特征提取层，局部特征提取层提取特征图像的局部特征，全局特征提取层提取特征图像的全局特征；局部特征提取层输出的局部特征和全局特征提取层输出的全局特征分别输入全连接层，以通过全连接层输出第一图像对应的双边网格。此外，在本实施例的一个可能实现方式中，所述下采样层包括下采样卷积层和四个第一卷积层，第一卷积层的卷积核为1*1，步长为1；所述局部特征提取层可以包括两个第二卷积层，两个第二卷积层的卷积核均为3*3，步长均为1；所述全局特征提取层可以包括两个第三卷积层和三个全连接层，两个第三卷积层的卷积核均为3*3，步长均为2。
进一步，所述卷积单元30包括第四卷积层，第一图像输入第四卷积层，经过第四卷积层输入指导图像，其中，所述指导图像与第一图像的分辨率相同。例如，第一图像为彩色图像，所述第四卷积层对第一图像进行像素级操作，以使得指导图像为第一图像的灰阶图像。
举例说明：第一图像I输入下采样卷积层，经过下采样卷积层输出256x256大小的三通道低分辨率图像，256x256大小的三通道低分辨率图像依次经过四个第一卷积层， 得到16x16大小的64通道特征图像；16x16大小的64通道特征图像输入局部特征提取层得到局部特征L，16x16大小的64通道特征图像输入全局特征提取层得到全局特征；局部特征和全局特征输入全连接层，经过全连接层输出双边网格。此外，将第一图像输入至卷积单元，经过卷积单元输入第一图像对应的指导图像。
进一步，所述在本实施例的一个实现方式中，所述变换模块200包括切分单元40以及变换单元60，相应的，如图3、4和9所示，所述将所述指导图像、所述双边网格以及所述第一图像输入所述变换模块，通过变换模块生成所述第一图像对应的生成图像具体包括：
S121、将所述指导图像输入所述切分单元，通过所述切分单元对所述双边网格进行切分，以得到所述第一图像中各像素点的颜色变换矩阵；
S122、将所述第一图像以及所述第一图像中各像素点的颜色变换矩阵输入所述变换单元，通过所述变换单元生成所述第一图像对应的生成图像。
具体地，所述切分单元40包括上采样层，所述上采样层的输入项为指导图像和双边网格，通过所述指导图像对双边网格进行上采样，以得到第一图像中各像素点的颜色变换矩阵。其中，所述上采样层的上采样过程可以为将所述双边网格参考指导图进行上采样，以得到第一图像中各像素点的颜色变换矩阵。此外，所述变换单元60的输入项为各像素点的颜色变换矩阵以及第一图像，通过各像素点的颜色变换矩阵对第一图像中其对应的像素点的颜色进行变换，以得到所述第一图像对应的生成图像。
S20、所述预设网络模型根据所述第一图像对应的第二图像和所述第一图像对应的生成图像，对模型参数进行修正，并继续执行根据所述训练图像集中第一图像，生成所述第一图像对应的生成图像的步骤，直至所述预设网络模型的训练情况满足预设条件，以得到所述图像处理模型。
具体地，所述预设条件包括损失函数值满足预设要求或者训练次数达到预设次数。所述预设要求可以是根据图像处理模型的精度来确定，这里不做详细说明，所述预设次数可以为预设网络模型的最大训练次数，例如，5000次等。由此，在预设网络模型输出生成图像，根据所述生成图像以及所述第二图像来计算预设网络模型的损失函数值，在计算得到损失函数值后，判断所述损失函数值是否满足预设要求；若损失函数值满足预设要求，则结束训练；若损失函数值不满足预设要求，则判断所述预设网络模型的训练次数是否达到预测次数，若未达到预设次数，则根据所述损失函数值对所述预设网络模型的网络参数进行修正；若达到预设次数，则结束训练。这样通过损失函数值和训练次 数来判断预设网络模型训练是否结束，可以避免因损失函数值无法达到预设要求而造成预设网络模型的训练进入死循环。
进一步，由于对预设网络模型的网络参数进行修改是在预设网络模型的训练情况未满足预设条件(即，损失函数值未满足预设要求并且训练次数未达到预设次数)，从而在根据损失函数值对所述预设网络模型的网络参数进行修正后，需要继续对网络模型进行训练，即继续执行将训练图像集中的第一图像输入预设网络模型的步骤。其中，继续执行将训练图像集中第一图像输入预设网络模型中的第一图像为未作为输入项输入过预设网络模型的第一图像。例如，训练图像集中所有第一图像具有唯一图像标识(例如，图像编号)，第一次训练输入的第一图像为图像标识与第二次训练输入的第一图像的图像标识不同，如，第一次训练输入的第一图像的图像编号为1，第二次训练输入的第一图像的图像编号为2，第N次训练输入的第一图像的图像编号为N。当然，在实际应用中，由于训练图像集中的第一图像的数量有限，为了提高图像处理模型的训练效果，可以依次将训练图像集中的第一图像输入至预设网络模型以对预设网络模型进行训练，当训练图像集中的所有第一图像均输入过预设网络模型后，可以继续执行依次将训练图像集中的第一图像输入至预设网络模型的操作，以使得训练图像集中的训练图像组按循环输入至预设网络模型。
此外，对于不同曝光度下拍摄的图像的高光部分的扩散程度不同，从而屏下成像系统在不同光线强度下拍摄的图像的高光部分的扩散程度不同，从而使得屏下成像系统拍摄得到的图像图像质量不同。由此，在对图像处理模型进行训练时，可以获取多个训练图像集，每个训练图像集对应不同的曝光度，并采用每个训练图像集对预设网络模型进行训练，以得到每个训练图像集对应的模型参数。这样采用具有相同曝光度的第一图像作为训练样本图像，可以提高网络模型的训练速度，同时使得不同曝光度对应不同的模型参数，在采用图像处理模型对具有偏色的待处理图像进行处理时，可以根据待处理图像对应的曝光度选取相应的模型参数，抑制各曝光度下图像高光部分的扩散，以提高待处理图像对应的处理后图像的图像质量。
进一步，在本实施例的一个实现方式中，所述训练图像集包括若干训练子图像集，每个训练子图像集包含多组训练样本图像组，每个训练子图像集包括若干组训练样本图像组，若干训组训练图像组中的任意两组训练样本图像组中的第一图像的曝光度相同(即对于每组训练图像组而言，该组中的每组训练样本图像组中的第一图像的曝光度均相同)，若干组训练图像组中的每组训练样本图像组中的第二图像的曝光度均处于预设 范围内，任意两个训练子图像集中的第一图像的曝光度不相同。其中，所述第二图像的曝光度的预设范围可以根据曝光时间和ISO(现有的手机的光圈为固定值)确定，所述曝光度的预设范围表示在无需曝光补偿下拍摄图像的曝光度，屏上摄像头在曝光度的预设范围内的第一曝光度下拍摄得到的第二图像为正常曝光图像，这通过采用正常曝光图像作为第二图像，可以使得根据训练图像集训练得到的图像处理模型输出的图像具有正常曝光度，从而使得图像处理模型具有提亮的功能。例如，当输入图像处理模的图像A为低曝光度的图像，那么图像A通过所述图像处理模型处理后，可以使得处理后的图像A的曝光度为正常曝光度，从而提高了图像A的图像亮度。
举例说明：假设图像的曝光度包括5个等级，分别记为0,-1,-2,-3和-4，其中，所述曝光度随着曝光度等级的降低而增强，例如，曝光等级0对应的曝光度低于曝光等级-4对应的曝光度。所述训练图像集可以包括5个训练子图像集，分别记为第一训练子图像集、第二训练子图像集、第三训练子图像集、第四训练子图像集以及第五训练子图像集，所述第一训练子图像集包含的每组训练图像组中第一图像的曝光度对应0等级，第二图像为曝光度在预设范围内的图像；所述第二训练子图像集包含的每组训练图像组中第一图像的曝光度对应-1等级，第二图像为曝光度在预设范围内的图像；所述第三训练子图像集包含的每组训练图像组中第一图像的曝光度对应-2等级，第二图像为曝光度在预设范围内的图像；所述第四训练子图像集包含的每组训练图像组中第一图像的曝光度对应-3等级，第二图像为曝光度在预设范围内的图像；所述第五训练子图像集包含的每组训练图像组中第一图像的曝光度对应-4等级，第二图像为曝光度在预设范围内的图像。当然，值得说明的，所述第一训练子图像集、第二训练子图像集、第三训练子图像集、第四训练子图像集以及第五训练子图像集包含的训练图像组的数量可以相同，也可以不同。例如，所述第一训练子图像集、第二训练子图像集、第三训练子图像集、第四训练子图像集以及第五训练子图像集均包括5000组训练图像组。
此外，针对于每个训练子图像集，该训练子图像集为预设网络模型的一个训练图像集，通过该训练子图像集对预设网络模型进行训练，以得到该训练子图像集对应的模型参数。其中，该训练子图像集作为训练图像集对预设网络模型进行训练的过程包括：所述预设网络模型根据训练子图像集中第一图像，生成第一图像对应的生成图像；所述预设网络模型根据所述第一图像对应的第二图像和所述第一图像对应的生成图像，对模型参数进行修正，并且预设网络模型继续执行根据训练子图像集中第一图像，生成第一图像对应的生成图像的步骤，直至所述预设网络模型的训练情况满足预设条件，以得到 该训练子图像对应的模型参数，具体地可以参数步骤S10和步骤S20，这里就不再赘述。
进一步，每个训练子图像集对所述预设网络模型的训练过程为相互独立，即分别采用每个训练子图像集对所述预设网络模型进行训练。同时，分别采用个训练子图像集对所述预设网络模型进行训练可以得到若干模型参数，每个模型参数均为根据一个训练子图像集训练得到，并且任意两个模型参数各自对应的训练子图像集互不相同。由此可知，图像处理模型对应若干模型参数，若干模型参数与若干训练子图像集一一对应。
举例说明：以上述训练样本图像包括第一训练子图像集、第二训练子图像集、第三训练子图像集、第四训练子图像集以及第五训练子图像集为例，那么图像处理模型包括5个模型参数，分别记为第一模型参数、第二模型参数、第三模型参数、第四模型参数以及第五模型参数，其中，第一模型参数对应第一训练子图像集，第二模型参数对应第二训练子图像集，第三模型参数对应第三训练子图像集，第四模型参数对应第四训练子图像集，第五模型参数对应第五训练子图像集。
进一步，当训练图像集包括若干训练子图像集时，预设网络模型根据每个训练子图像集进行训练。这里以训练图像集包括5个训练子图像集为例加以说明。采用第一训练子图像集、第二训练子图像集、第三训练子图像集、第四训练子图像集以及第五训练子图像集分别对预设网络模型进行训练的过程可以为：首先采用第一训练子图像集对预设网络模型进行训练，得到第一训练子图像集对应的第一模型参数，之后再采用第二训练子图像集对预设网络模型进行训练，得到第二训练子图像集对应的第二模型参数，依次类推得到第五训练子图像集对应的第五模型参数。
此外，当使用同一个预设网络模型对多个训练子图像集分别进行训练时，会存在各个训练子图像集对于预设网络模型的模型参数产生影响的问题，举例来说，假设训练子图像集A包括1000组训练图像组，训练子图像集B包括200组训练图像组，那么，先用训练子图像集A对预设网络模型进行训练，再紧接着用训练子图像集B对预设网络模型进行训练所得到的该训练子图像集B所对应的模型参数，与仅用训练子图像集B对预设网络模型进行训练所得到的该训练子图像集B所对应的模型参数，是不同的。
故此，在本实施例的一种实现方式中，预设网络模型在训练完一训练子图像集之后，可以先对该预设网络模型进行初始化，再使用该初始化后的预设网络模型对下一训练子图像集进行训练。举例来说，预设网络模型根据第一训练子图像集进行训练，得到第一训练子图像集对应的第一模型参数后，所述预设网络模型可以进行初始化，以使得用于训练第二模型参数的预设网络模型的初始模型参数以及模型结构均与用于训练第 一模型参数的预设网络模型相同，当然，在训练第三模型参数、第四模型参数和第五模型参数之前，均可以对预设网络模型进行初始化，以使得每个训练子图像集对应的预设网络模型的初始模型参数以及模型结构均相同。当然，在实际应用中，预设网络模型根据第一训练子图像集进行训练，得到第一训练子图像集对应的第一模型参数后，也可以直接采用基于第一训练子图像集训练后的预设网络模型(配置第一模型参数)对第二训练子图像集进行训练，以得到第二训练子图像集对应的第二模型参数，继续执行预设网络模型(配置第二模型参数)根据第三训练子图像集进行训练的步骤，直至第五训练子图像集训练完毕，得到第五训练子图像集对应的第五模型参数。
此外，第一训练子图像集、第二训练子图像集、第三训练子图像集、第四训练子图像集以及第五训练子图像集均包括一定数量的训练图像组，以使得每组训练子图像均可以满足预设网络模型的训练需求。当然，在实际应用中，在基于每一训练子图像集对预设网络模型进行训练时，可以循环将该训练子图像集中的训练图像组输入至预设网络模型，以对所述预设网络模型进行训练，使得预设网络模型满足预设要求。
进一步，在本实施例的一个实现按时，所述获取包含各个训练子图像集的训练样本的获取过程可以为：首先将屏下成像系统设置为第一曝光度，通过屏下成像系统获取第一训练子图像集中的第一图像，以及通过屏上成像系统获取第一训练子图像集中和第一图像对应的第二图像；在第一训练子图像集获取完成后，将屏下成像系统设置为第二曝光度，通过屏下成像系统和屏上成像系统获取第二训练子图像集中第一图像和第一图像对应的第二图像；在第二训练子图像集获取完成后；继续执行设置屏下成像系统的曝光度以及获取训练子图像集的步骤，直至获取到训练图像集包含的所有训练子图像集。其中，训练图像集包含的每个训练子图像集包含的训练图像组的数量可以相同，也可以不相同。在本实施例的一个实现方式中，所述训练图像集包含的每个训练子图像集包含的训练图像组的数量可以相同，例如，每个训练子图像集包含的训练图像组的数量为5000。
进一步，由于各训练子图像集均对应不同的曝光度，从而在获取到每个训练子图像集对应的模型参数后，针对于每个训练子图像集，可以将该训练子图像集对应的模型参数与该训练子图像集对应的曝光度相关联，以建立曝光度与模型参数的对应关系。这样在采用图像处理模型对待处理图像进行处理时，可以先获取待处理图像的曝光度，再根据曝光度确定待处理图像对应的模型参数，然后将待处理图像对应的模型参数配置于预设网络模型，以得到待处理图像对应的图像处理模型，以便于采用该图像处理模型对 待处理图像进行处理。这样对于不同曝光度的待处理图像可以确定配置不同网络参数的图像处理模型，并采用待处理图像对应的图像处理模型对待处理图像进行处理，避免曝光度对偏色的影响，从而可以提高去除待处理图像的偏色的效果。此外，所述第二图像可以为采用正常曝光度，使得所述图像处理模型输出的处理后的图像为正常曝光度，对待处理图像起到提亮的效果。
基于上述图像处理模型的生成方法，本实施例还提供了一种图像处理方法，如图10所示，所述图像处理方法包括：
A100、获取待处理图像，并将所述待处理图像输入至所述图像处理模型。
具体地，所述待处理图像可以为用于处理所述待处理图像的图像设备拍摄的图像，也可以是其他外部设备并存储于图像处理设备的图像，通过云端发送的图像。在本实施例中，所述待处理图像为通过屏下成像系统(例如屏下摄像头)拍摄得到的图像，其中，所述屏下成像系统可以为图像设备自身配置的，可也可以为其他设备配置的。例如，所述待处理图像为通过配置有屏下成像系统的手机拍摄得到的人物图像。
此外，所述图像处理模型可是处理所述待处理图像的图像设备(例如，配置屏下摄像头的手机)预先训练的，也可以是由其他训练好后将图像处理模型对应的文件移植到图像设备中。此外，图像设备可以将所述图像处理模型可作为一个去偏色功能模块，当图像设备获取到待处理图像时，启动所述去偏色功能模块，将待处理图像输出至图像处理模型。
进一步，由图像处理模型的生成过程可以知道，在一种可能的实现方式中，所述图像处理模型包括可以若干模型参数，并且每个模型参数均对应一个曝光度。因此，在该实现方式中，在获取到待处理图像后，可以先检测所述图像处理模型包括的模型参数的数量，当模型参数的数量为一个时，直接将所述待处理图像输入到所述图像处理模型内，以通过所述图像处理对待处理图像进行处理；当模型参数为多个时，可以先获取待处理图像的曝光度，再根据曝光度确定该待处理图像对应的模型参数，将该待处理图像对应的模型参数配置于所述图像处理模型，以对图像处理参数配置的模型参数进行更新，并将待处理图像输入更新后图像处理模型。
进一步，在本实施例的一个实现方式中，所述图像处理模型对应若干模型参数，每个模型参数均为根据一个训练子图像集训练得到的，并且任意两个模型参数各自分别对应的训练子图像集互不相同(例如，模型参数A对应的训练子图像集与模型参数B对应的训练子图像集是不同的)。相应的，如图11所示，所述获取待处理图像，并将所述 待处理图像输入至所述图像处理模型具体包括：
A101、获取待处理图像，并提取所述待处理图像的曝光度。
具体地，所述曝光度为图像采集装置的感光元件被光线照射的程度，用于反映成像时的曝光程度。所述待处理图像可以为RGB三通道图像，所述待处理图像的曝光度为根据待处理图像的高光区域确定的，所述高光区域包含的各像素点的R(即红色通道)值、G(即绿色通道)值以及B(即蓝色通道)值中至少存在一个值大于预设阈值。当然，在实际应用中，所述待处理图像还可以是Y通道图像或者贝尔格式图像，而当所述待处理图像为Y通道图像或者贝尔格式图像(Raw格式)时，在提取所述待处理图像之前，需要将所述Y通道图像或者贝尔格式图像转换为RGB三通道图像，以便于根据待处理图像的红色通道R值、绿色通道G值以及蓝色通道B值确定待处理图像的高光区域。
进一步，在本实施例的一个实现方式中，所述提取所述待处理图像的曝光度具体包括：
B10、根据所述待处理图像中各像素点的红色通道R值、绿色通道G值以及蓝色通道B值确定满足预设条件的第三像素点，其中，所述预设条件为R值、G值以及B值中至少一个值大于预设阈值；
B20、根据满足预设条件的所有第三像素点确定所述待处理图像的高光区域，并根据所述高光区域确定所述待处理图像的曝光度。
具体地，所述待处理图像为RGB三通道图像，从而对于待处理图像中的每个像素点，该像素点均包括红色通道R值、绿色通道G值和蓝色通道B值，即对于待处理图像中的每个像素点，均可以获取到该像素点的红色通道R值、绿色通道G值以及蓝色通道B值。由此，在提取所述待处理图像的曝光的过程中，首先针对于每个待处理图像的每个像素点，获取该像素点的红色通道R值、绿色通道G值以及蓝色通道B值，之后再分别将各像素点的R值、G值以及B值分别与预设阈值进行比较，以获取待处理图像中满足预设条件的第三像素点。所述预设条件为预设条件为R值、G值以及B值中至少一个值大于预设阈值，第三像素点满足预设条件指的是第三像素点的R值大于预设阈值，第三像素点的G值大于预设阈值，第三像素点的B值大于预设阈值，第三像素点的R值和G值均大于预设阈值，第三像素点的R值和B值均大于预设阈值，第三像素点的G值和B值均大于预设阈值，或者第三像素点的R值、B值和G值均大于预设阈值。
进一步，在获取到满足预设条件的所有第三像素点后，将获取到所有第三像素点记为第三像素点集，第三像素点集中存在相邻的像素点，也存在不相邻的像素点，其中， 像素点相邻指的是像素点在待处理图像中的位置相邻，所述不相邻指的是像素点在待处理图像中的位置不相邻，所述位置相邻指的在待处理的像素坐标中，相邻两个像素点的横坐标和纵坐标中存在一个相同。例如，第三像素点集中包括像素点(100,101)、像素点(100,100)，像素点(101,101)以及像素点(200,200)，那么像素点(100,101)、像素点(100,100)为相邻像素点，并且像素点(100,101)、像素点(101,101)为相邻像素点，而像素点(100,101)、像素点(100,100)和像素点(101,101)和像素点(200,200)均为不相邻像素点。
进一步，所述高光区域根据第三像素点集中相邻像素点构成的连通区域，即高光区域包含的每个第三像素点的像素值均满足预设条件。由此，在本实施例一个实现方式中，所述根据满足预设条件的所有第三像素点确定所述待处理图像的高光区域具体包括：
C10、获取所述满足预设条件的所有第三像素点所形成的连通区域，并在获取到的所有连通区域进行选取满足预设规则的目标区域，其中，所述预设规则为目标区域中的第三像素点的R值、G值和B值中大于预设阈值的R值、G值和/或B值的类型相同；
C20、计算筛选得到的各目标区域分别对应的面积，并选取面积最大的目标区域作为高光区域。
具体地，所述连通区域是第三像素点集中所有相邻第三像素点形成的闭合区域，所述连通区域包含的每个像素点均为第三像素点，并且对于连通区域内的每个第三像素点A，连通区域内至少一个第三像素点B与该第三像素点A相邻。同时，针对于第三像素点集中去除该连通区域包含的第三像素点外的每个第三像素点C，该第三像素点C与连通区域内的任一第三像素点A均不相邻。例如，第三像素点集中包括像素点(100,101)、像素点(100,100)、像素点(101,100)、像素点(101,101)、像素点(100,102)以及像素点(200,200)，那么，像素点(100,101)、像素点(100,100)、像素点(101,100)、像素点(101,101)、像素点(100,102)形成一个连通区域。
此外，由于待处理图像的连通区域是有光源形成，并且光源会产生光线颜色相同。从而在获取到待处理图像包含的所有连通区域后，可以根据各连通区域对应的区域颜色对连通区域进行选取。由此，在获取到待处理图像的连通区域后，判断连通区域内各第三像素点的R值、G值和B值中第三像素点的R值、G值和B值中大于预设阈值的R值、G值和/或B值的类型是否相同，以判断连通区域是否满足预设规则。所述类型相同指的是对于两个第三像素点，分别记为像素点A和像素点B，若像素点A为R值大于预设阈值，那么像素点B也只有R值大于预设阈值；若像素点A的R值和G值均大于预设阈值， 那么像素点B也只有R值和G值大于预设阈值；若像素点A的R值、G值和B值均大于预设阈值，那么像素点B的R值、G值和B值均大于预设阈值。所述类型不同指的是，对于两个第三像素点，分别记为像素点C和像素点D,若像素点C为V值(V值可以为R值，G值，B值中一种)大于预设阈值，那么像素点D中V值小于或等于预设阈值，或者像素点D中V值大于预设阈值且至少存在一个M值(M值为R值，G值和B值中去除V值外的两个值中一个)大于预设阈值。例如，像素点C的R值大于预设阈值，像素点D的R值小于等于预设阈值，那么像素点C和像素点D的类型不同；再如，像素点C的R值大于预设阈值，像素点D的R值大于预设阈值，并且像素点D的G值大于预设阈值，那么像素点C和像素点D的类型不同。本实施例中，所述预设规则为各连通区域中的第三像素点的R值、G值和B值中大于预设阈值的R值、G值和/或B值的类型相同。
进一步，由于待处理图像中可能包括多个目标区域，从而在获取到目标区域后，可以根据目标区域的面积对目标区域进行筛选以得到高光区域。其中，所述目标区域的面积指的是目标区域在待处理图像中所在区域的面积，所述面积是在待处理图像的像素坐标系内计算的。在获取到各目标区域的面积后，可以将各目标区域的面积进行比较，并选取面积最大的目标区域，将所述目标区域作为高光区域，这样将面积最大的目标区域作为高光区域，可以获取到待处理图像中亮度面积最大的区域，根据亮度面积最大的区域确定曝光度，可以提高曝光度的准确性。
进一步，在本实施例的一个实现方式中，所述根据所述高光区域确定所述待处理图像的曝光度具体包括：
D10、计算所述高光区域的第一面积以及待处理图像的第二面积；
D20、根据所述第一面积和第二面积的比值确定所述待处理图像对应的曝光度。
具体地，所述待处理图像的第二面积指的是根据待处理图像的图像尺寸计算得到，例如，待处理图像的图像尺寸为400*400，那么待处理图像的图像面积为400*400＝160000。所述高光区域的第一面积为高光区域在待处理图像的像素坐标系中区域面积，例如，高光区域为边长为20的正方形区域，那么高光区域的第一面积为20*20＝400。
进一步，为了根据第一面积和第二面积的比值确定曝光度，预先设定了比值区间与曝光度的对应关系，在获取到比值后，首先取得比值所处比值区域，在根据该对应关系确定该比值区间对应曝光度，以得到待处理图像的曝光度。例如，所述比值区间与曝光度的对应关系为：当区间为[0,1/100)时，曝光度对应0等级；当区间为[1/100,1/50)时，曝光度对应-1等级；当区间为[1/50,1/20)时，曝光度对应-2等级；当区间为 [1/20,1/50)时，曝光度对应-3等级；当区间为[1/20,1]时，曝光度对应-4等级。那么当第一面积与第二面积的比值为1/10时，该比值处于区间[1/20,1]，从而该待处理图像对应的曝光度为-4等级。
A102、根据所述曝光度确定所述待处理图像对应的模型参数，并采用所述模型参数更新所述图像处理模型的模型参数。
具体地，在图像处理模型训练时建立了曝光度与模型参数的对应关系，从而在获取到待处理图像的曝光度后，可以根据曝光度与模型参数的对应关系确定该曝光度对应的模型参数，其中，所述曝光度指的是曝光度等级，即所述曝光度与模型参数的对应关系为曝光度等级与模型参数的对应关系。此外，由上述可以知道，每个曝光等级对应一个比值区间，那么在获取到待处理图像后，可以获取待处理图像中高光区域的区域面积与图像图像的比值，并确定所述比值所处的比值区间，再根据比值区域确定待处理图像对应的曝光等级，最后根据曝光等级确定待处理图像对应的模型参数，从而得到待处理图像对应的模型参数。此外，在获取到曝光度对应的模型参数后，采用获取到的模型参数更新图像处理模型配置的模型参数，以更新图像处理模型，即获取到的模型参数所对应的图像处理模型。
A103、将所述待处理图像输入至更新后的图像处理模型。
具体地，将待处理图像作为更新后的图像处理模型的输入项，并将待处理图像输出至更新后的图像处理模型对待处理图像进行处理。可以理解的是，所述待处理图像对应的图像处理模型的模型参数为根据所述待处理图像的曝光度确定模型参数，并且该模型参数为通过对预设网络模型进行训练得到的模型参数，这样可以保证更新后的图像处理模型对待处理图像处理的精确度。至此，完成对步骤A100(即，获取待处理图像，并将所述待处理图像输入至所述图像处理模型)的介绍，下面对所述步骤A100的后续步骤说明。
A200、通过所述图像处理模型对所述待处理图像进行偏色处理，以得到所述待处理图像对应的处理后的图像。
具体地，所述通过所述图像处理模型对所述待处理图像进行去偏色指的是将所述待处理图像作为所述图像处理模型的输入项输入至所述图像处理模型中，通过所述图像处理模型去除所述待处理图像的偏色，即去除所述待处理图像的第一目标像素点，以得到处理后的图像，其中，所述处理后的图像为所述待处理图像对应的通过所述处理模型对待处理图像进行偏色处理后的图像，即，待处理图像为对应处理后的图像的偏色图像。 例如，如图12所示的待处理图像通过所述图像处理图像后得到如图13所示的处理后的图像。
进一步，由所述图像处理模型的训练过程可以知道，所述图像处理模型包括下采样模块以及变换模块，从而在通过图像处理模型对应待处理图像进行处理时，需要依次通过下采样模块以及变换模块进行处理。相应的，所述图像处理模型包括；所述通过所述图像处理模型对所述待处理图像进行偏色处理，以得到所述待处理图像对应的处理后的图像具体包括：
A201、将所述待处理图像输入所述下采样模块，通过所述下采样模块得到所述待处理图像对应的双边网格以及所述待处理图像对应的指导图像，其中，所述指导图像的分辨率与所述待处理图像的分辨率相同；
A202、将所述指导图像、所述双边网格以及所述待处理图像输入所述变换模块，通过变换模块生成所述第一图像对应的处理后的图像。
具体地，所述下采样模块的输入项为待处理图像，输出项为待处理图像对应的双边网格以及指导图像，所述变换模块的输入项为指导图像、双边网格以及待处理图像，输出项为处理后的图像。其中，所述下采样模块的结构与预设网络模型中的下采样模块的结构相同，具体可以参照预设网络模型中的下采样模块的结构的说明。所述图像处理模型的下采样模块的对待处理图像的处理与预设网络模型中的下采样模块对第一图像的处理过程相同，从而所述步骤A201的具体执行过程可以参照步骤S11。同样的，所述变换模块的结构与预设网络模型中的变换模块的结构相同，具体可以参照预设网络模型中的变换模块的结构的说明。所述图像处理模型的变换模块的对待处理图像的处理与预设网络模型中的变换模块对第一图像的处理过程相同，从而所述步骤A202的具体执行过程可以参照步骤S12。
进一步，在本实施例的一个实现方式中，所述下采样模块包括下采样单元以及卷积单元。相应的，所述将所述待处理图像输入所述下采样模块，通过所述下采样模块得到所述待处理图像对应的双边网格以及所述待处理图像对应的指导图像具体包括：
A2011、将所述待处理图像分别输入所述下采样单元以及所述卷积单元；
A2012、通过所述下采样单元得到所述待处理图像对应的双边网格，并通过所述卷积单元得到所述待处理图像对应的指导图像。
具体地，所述下采样单元的输入项为待处理图像，输出项为双边网格，所述卷积单元的输入项为待处理图像，输出项为指导图像。其中，其中，所述下采样单元的结构 与预设网络模型中的下采样单元的结构相同，具体可以参照预设网络模型中的下采样单元的结构的说明。所述图像处理模型的下采样单元的对待处理图像的处理与预设网络模型中的下采样单元对第一图像的处理过程相同，从而所述步骤A2011的具体执行过程可以参照步骤S111。同样的，所述卷积单元的结构与预设网络模型中的卷积单元的结构相同，具体可以参照预设网络模型中的卷积单元的结构的说明。所述图像处理模型的卷积单元的对待处理图像的处理与预设网络模型中的卷积单元对第一图像的处理过程相同，从而所述步骤A2012的具体执行过程可以参照步骤S112。
进一步，在本实施例的一个实现方式中，所述变换模块包括切分单元以及变换单元。相应的，所述将所述指导图像、所述双边网格以及所述待处理图像输入所述变换模块，通过变换模块生成所述待处理图像对应的处理后的图像具体包括：
A2021、将所述指导图像输入所述切分单元，通过所述切分单元对所述双边网格进行切分，以得到所述待处理图像中各像素点的颜色变换矩阵；
A2022、将所述待处理图像以及所述待处理图像中各像素点的颜色变换矩阵输入所述变换单元，通过所述变换单元生成所述待处理图像对应的处理后的图像。
具体地，所述切分单元的输入项为指导图像和双边网格，输出项为待处理图像中各像素点的颜色变换矩阵，所述变换单元的输入项为待处理图像和待处理图像中各像素点的颜色变换矩阵，输出项为处理后的图像。其中，其中，所述切分单元的结构与预设网络模型中的切分单元的结构相同，具体可以参照预设网络模型中的切分单元的结构的说明。所述图像处理模型的切分单元对待处理图像对应的双边网格以及指导图像的处理，与预设网络模型中的下采样单元对第一图像对应的双边网格以及指导图像的处理过程相同，从而所述步骤A2021的具体执行过程可以参照步骤S121。同样的，所述变换单元的结构与预设网络模型中的变换单元的结构相同，具体可以参照预设网络模型中的变换单元的结构的说明。所述图像处理模型的变换单元基于待处理图像中各像素点的颜色变换矩阵对待处理图像的处理与预设网络模型中的变换单元基于第一图像中各像素点的颜色变换矩阵对第一图像的处理过程相同，从而所述步骤A2022的具体执行过程可以参照步骤S122。
可以理解的是，图像处理模型在训练过程中对应的网络结构，与在应用过程(对待处理图像进行去偏色处理)中所对应的网络结构相同。例如，在训练的过程中，图像处理模型包括下采样模块和变换模块，那么相应地，在通过图像处理模型对待处理图像进行去偏色处理时，图像处理模型也包括下采样模块和变换模块。
例如，在训练过程中，图像处理模型的下采样模块包括下采样单元以及卷积单元，变换模块包括切分单元和变换单元；相应地，在通过图像处理模型对待处理图像进行去偏色处理时，下采样模块也可以包括下采样单元以及卷积单元，变换模块包括切分单元和变换单元；并且在应用过程中，每一层的工作原理与在训练过程中每一层的工作原理相同，因此，图像处理模型应用过程中的每一层神经网络的输入输出情况可以参见图像处理模型的训练过程中的相关介绍，这里不再赘述。
与现有技术相比，本发明提供了一种图像处理模型的生成方法、处理方法。所述生成方法包括：所述方法通过将预设的训练图像集中第一图像输入预设网络模型，通过所述预设网络模型生成的生成图像以及第一图像对应的第二图像对预设模型进行序列，得到图像处理模型。所述图像处理模型为通过对具有多组训练图像组的训练图像集的去偏色过程进行深度学习得到，每一组训练图像组包括第一图像和第二图像，第一图像为对应第二图像的偏色图像。由此可知，本发明是采用基于训练图像集进行深度学习得到已训练的图像处理模型进行偏色处理，这样可以快速对图像进行偏色调整，即纠正偏色，提高图像的色彩质量，从而提高图像质量。
进一步，为了进一步提高图像处理模型的图像质量，在获取到图像处理模型输出的处理后的图像后，还可以对所述处理后的图像进行后处理，其中，所述后处理可以包括锐化处理以及降噪处理等。相应的，所述通过所述图像处理模型对所述待处理图像进行偏色处理，以得到所述待处理图像对应的处理后的图像之后还包括：
对所述处理后的图像进行锐化以及降噪处理，并将锐化以及降噪处理后的图像作为所述待处理图像对应的处理后的图像。
具体地，所述锐化处理指的是补偿处理后的图像的轮廓、增强处理后的图像的边缘及灰度跳变的部分，以提高处理后图像的图像质量。其中，所述锐化处理可以采用现有的锐化处理方法，例如，高通滤波方法等。所述降噪处理指的是去除图像中的噪音，提高图像的信噪比。其中，所述降噪处理可以采用现有的降噪算法或已训练的降噪网络模型等，例如，所述降噪处理采用高斯低通滤波方法等。
基于上述图像处理模型的生成方法，如图34所示，本实施例提供了一种图像处理模型的生成装置，所述图像处理模型的生成装置包括：
第一生成模块101，用于利用预设网络模型根据训练图像集中的第一图像，生成所述第一图像对应的生成图像，其中，所述训练图像集包括多组训练图像组，每一组训练图像组包括第一图像和第二图像，第一图像为对应第二图像的偏色图像；
第一修正模块102，用于利用预设网络模型根据所述第一图像对应的第二图像和所述第一图像对应的生成图像，对模型参数进行修正，并继续执行根据所述训练图像集中的下一训练图像组中的第一图像，生成所述第一图像对应的生成图像的步骤，直至所述预设网络模型的训练情况满足预设条件，以得到所述图像处理模型。
在一个实施例中，所述第一图像中满足预设偏色条件的第一目标像素点的数量满足预设数量条件；所述预设偏色条件为第一图像中第一目标像素点的显示参数与第二图像中第二目标像素点的显示参数之间的误差满足预设误差条件，其中，所述第一目标像素点与所述第二目标像素点之间具有一一对应关系。
在一个实施例中，所述第一目标像素点为所述第一图像中任意一个像素点或者所述第一图像的目标区域中任意一个像素点。
在一个实施例中，所述训练图像集包括若干训练子图像集，每个训练子图像集包括若干组训练样本图像组，若干训组训练图像组中的任意两组训练样本图像组中的第一图像的曝光度相同，若干组训练图像组中的每组训练样本图像组中的第二图像的曝光度均处于预设范围内，任意两个训练子图像集中的第一图像的曝光度不相同。
在一个实施例中，所述图像处理模型对应若干模型参数，每个模型参数均为根据所述训练图像集中的一个训练子图像集训练得到的，并且任意两个模型参数各自分别对应的训练子图像集互不相同。
在一个实施例中，所述预设网络模型包括下采样模块以及变换模块；所述第一生成模块具体用于：
将所述训练图像集中的第一图像输入所述下采样模块，通过所述下采样模块得到所述第一图像对应的双边网格以及所述第一图像对应的指导图像；以及将所述指导图像、所述双边网格以及所述第一图像输入所述变换模块，通过变换模块生成所述第一图像对应的生成图像，其中，所述指导图像的分辨率与所述第一图像的分辨率相同。
在一个实施例中，所述下采样模块包括下采样单元和卷积单元；所述第一生成模块具体用于：
将所述训练图像集中第一图像分别输入所述下采样单元以及所述卷积单元；以及通过所述下采样单元得到所述第一图像对应的双边网格，并通过所述卷积单元得到所述第一图像对应的指导图像。
在一个实施例中，所述变换模块包括切分单元以及变换单元，所述第一生成模块具体用于：
将所述指导图像输入所述切分单元，通过所述切分单元对所述双边网格进行切分，以得到所述第一图像中各像素点的颜色变换矩阵；
将所述第一图像以及所述第一图像中各像素点的颜色变换矩阵输入所述变换单元，通过所述变换单元生成所述第一图像对应的生成图像。
在一个实施例中，所述第一图像为通过屏下成像系统拍摄得到的图像。
在一个实施例中，所述屏下成像系统为屏下摄像头。
在一个实施例中，所述图像处理模型的生成装置还包括：
第一对齐模块，用于针对所述训练图像集中每组训练图像组，将该组训练图像组中的第一图像与所述第一图像对应的第二图像进行对齐处理，得到与所述第二图像对齐的对齐图像，并将所述对齐图像作为第一图像。
在一个实施例中，所述第一对齐模块具体用于：
针对所述训练图像集中每组训练图像组，获取该组训练图像组中的第一图像与所述第一图像对应的第二图像之间的像素偏差量；根据所述像素偏差量确定所述第一图像对应的对齐方式，并采用所述对齐方式将所述第一图像与第二图像进行对齐处理，得到与所述第二图像对齐的对齐图像；以及将所述对齐图像作为第一图像
在一个实施例中，所述第一对齐模块具体用于：
当所述像素偏差量小于等于预设偏差量阈值时，根据所述第一图像与所述第二图像的互信息，以所述第二图像为基准对所述第一图像进行对齐处理；
当所述像素偏差量大于所述预设偏差量阈值时，提取所述第一图像的第一像素点集和所述第二图像的第二像素点集，所述第一像素点集包含所述第一图像中的若干第一像素点，所述第二像素点集包括所述第二图像中的若干个第二像素点，所述第二像素点集中的第二像素点与所述第一像素点集中的第一像素点一一对应；针对第一像素点集中每个第一像素点，计算该第一像素点与其对应的第二像素点的坐标差值，并根据该第一像素点对应的坐标差值对该第一像素点进行位置变换，以将该第一像素点与该第一像素点对应的第二像素点对齐。
基于上述实施例提供的图像处理方法，如图35所示，本实施例提供了一种图像处理装置，应用如上所述的图像处理模型的生成方法或生成装置所生成的图像处理模型，所述图像处理装置包括：
第一获取模块201，用于获取待处理图像，并将所述待处理图像输入至所述图像处理模型；
第一处理模块202，用于通过所述图像处理模型对所述待处理图像进行偏色处理，以得到所述待处理图像对应的处理后的图像。
所述图像处理模型对应若干模型参数，每个模型参数均为根据一个训练子图像集训练得到的，并且任意两个模型参数各自分别对应的训练子图像集互不相同。
在一个实施例中，所述第一获取模块具体用于：
获取待处理图像，并提取所述待处理图像的曝光度；根据所述曝光度确定所述待处理图像对应的模型参数，并采用所述模型参数更新所述图像处理模型的模型参数；以及将所述待处理图像输入至更新后的图像处理模型。
在一个实施例中，所述第一获取模块具体用于：
根据所述待处理图像中各像素点的R值、G值以及B值确定满足预设条件的第三像素点，其中，所述预设条件为R值、G值以及B值中至少一个值大于预设阈值；
根据满足预设条件的所有第三像素点确定所述待处理图像的高光区域，并根据所述高光区域确定所述待处理图像的曝光度。
在一个实施例中，所述第一获取模块具体用于：
获取所述满足预设条件的所有第三像素点所形成的连通区域，并在获取到的所有连通区域进行选取满足预设规则的目标区域，其中，所述预设规则为目标区域中的第三像素点的R值、G值和B值中大于预设阈值的R值、G值和/或B值的类型相同；
计算筛选得到的各目标区域分别对应的面积，并选取面积最大的目标区域作为高光区域。
在一个实施例中，所述第一获取模块具体用于：
计算所述高光区域的第一面积以及待处理图像的第二面积；
根据所述第一面积和第二面积的比值确定所述待处理图像对应的曝光度。
在一个实施例中，所述图像处理模型包括下采样模块以及变换模块；所述第一处理模块具体用于：
将所述待处理图像输入所述下采样模块，通过所述下采样模块得到所述待处理图像对应的双边网格以及所述待处理图像对应的指导图像，其中，所述指导图像的分辨率与所述待处理图像的分辨率相同；以及将所述指导图像、所述双边网格以及所述待处理图像输入所述变换模块，通过变换模块生成所述第一图像对应的处理后的图像。
在一个实施例中，所述下采样模块包括下采样单元和卷积单元；所述第一处理模块具体用于：
将所述待处理图像分别输入所述下采样单元以及所述卷积单元；通过所述下采样单元得到所述待处理图像对应的双边网格，并通过所述卷积单元得到所述待处理图像对应的指导图像。
在一个实施例中，所述变换模块包括切分单元以及变换单元，所述第一处理模块具体用于：
将所述指导图像输入所述切分单元，通过所述切分单元对所述双边网格进行切分，以得到所述待处理图像中各像素点的颜色变换矩阵；以及将所述待处理图像以及所述待处理图像中各像素点的颜色变换矩阵输入所述变换单元，通过所述变换单元生成所述待处理图像对应的处理后的图像。
在一个实施例中，所述图像处理装置还包括：
降噪处理单元，用于对所述处理后的图像进行锐化以及降噪处理，并将锐化以及降噪处理后的图像作为所述待处理图像对应的处理后的图像。
实施例二
本实施例提供了一种图像处理模型的生成方法，如图15和16所示，所述方法包括：
N10、预设网络模型根据训练图像集中的第一图像，生成所述第一图像对应的生成图像。
具体地，所述预设网络模型为深度学习网络模型，所述训练图像集包括多组具有不同图像内容的训练图像组，每一组训练图像组均包括第一图像和第二图像，第一图像与第二图像相对应，它们呈现的是同一图像场景，第二图像为正常显示的图像(即原始图像)，第一图像的图像内容与第二图像对应但图像内容中的物体出现重影或者与重影类似的模糊效果。其中，所述重影指的是图像中的物体周围形成了虚像，例如，可以包括图像中物体的边缘出现一重或多重轮廓或虚像的情况，举例来说，当图像中的物体出现了双重影像(即出现物体边缘出现一重轮廓或虚像)时，其中，像素值较小的一列影像可以理解为物体的实像，像素值较大的另一列影像可以理解为物体的轮廓或虚像。
所述第一图像和第二图像对应相同的图像场景。所述第一图像和第二图像对应相同的图像场景指的是第一图像携带的图像内容与第二图像携带的图像内容的相似度达到预设阈值，所述第一图像的图像尺寸与第二图像的图像尺寸相同，以使得当第一图像和第二图像重合时，第一图像携带的物体对第二图像中与其对应的物体的覆盖率达到预设条件。其中，所述预设阈值可以为99％，所述预设条件可以为99.5％等。
此外，在本实施例的一个实现方式中，为了减少第一图像和第二图像的图像差异对预设网络模型训练的影响，所述第一图像的图像内容和第二图像的图像内容可以完全相同。例如，所述第一图像为图像尺寸为600*800的具有重影的图像，第一图像的图像内容为一个正方形，并且第一图像中正方形的四个顶点在第一图像中的位置分别为(200，300)、(200,400)，(300,400)以及(300,300)。那么，所述第二图像的图像尺寸为600*800的图像，第二图像的图像内容为一个正方形，第二图像中正方形的四个顶点在第二图像中的位置分别为(200，300)、(200,400)，(300,400)以及(300,300)，当第一图像放置于第二图像上并与第二图像重合时，所述第一图像覆盖所述第二图像，并且第一图像中的正方形与第二图像的正方形上下重叠。
进一步，所述第二图像可以是通过正常拍摄得到的图像，例如将屏下成像系统中的显示面板移除后由屏下摄像头拍摄的图像，或者通过制作不带数据线和扫描线等遮光结构的实验性质的显示面板替代实际的显示面板，然后利用其作为屏下成像系统的显示面板而由屏下摄像头拍摄的图像，也可以是通过网络(如，百度)获取的图像，还可以是通过其他外部设备(如，智能手机)发送的图像。所述第一图像可以为通过屏下成像系统(例如，屏下摄像头)拍摄得到，也可以是通过对第二图像进行处理得到。所述对第二图像进行处理指的是在第二图像上形成重影，在一种可能的实现方式中，在处理的过程中可以同时保持第二图像的图像尺寸以及图像内容不变。
在本实施例的一个实现方式中，所述第一图像为通过屏下成像系统拍摄得到，所述第一图像和第二图像的拍摄参数相同，并且所述第一图像对应的拍摄场景与第二图像的拍摄场景相同。例如，所述第一图像为如图17所示的图像，因显示面板内遮光结构的影响导致图像内容较为模糊，第二图像为如图18所示的正常显示的图像。同时在实施例的一个可能实现方式中，所述拍摄参数可以包括成像系统的曝光参数，其中，所述曝光参数可以包括光圈、开门速度、感光度、对焦以及白平衡等。当然，在实际应用中，所述拍摄参数还可以包括环境光、拍摄角度以及拍摄范围等。
进一步，当所述第一图像为通过屏下成像系统拍摄得到的图像时，由于第一图像和第二图像可以是通过两个不同的成像系统拍摄，而在更换成像系统时，可能会造成拍摄位置或拍摄角度的变化，使得所述第一图像和第二图像在空间上存在不对齐的问题。由此，在所述预设网络模型根据训练图像集中第一图像，生成所述第一图像对应的生成图像之前还包括：
P10、针对所述训练图像集中每组训练图像组，将该组训练图像组中的第一图像与 所述第一图像对应的第二图像进行对齐处理，得到与所述第二图像对齐的对齐图像，并将所述对齐图像作为第一图像。
具体地，所述针对所述训练图像集中每组训练图像组指的是对训练图像集中每一组训练图像组均执行对齐处理，所述对齐处理可以是在获取到训练图像集之后，分别对每一组训练图像组进行对齐处理，以得到对齐后的训练图像组，并在所有组训练图像组对齐后执行将每一组训练图像组中的第一图像输入预设网络模型的步骤；当然也可以是在将每一组训练图像组中的第一图像输入预设网络模型之前，对该组训练图像组进行对齐处理，以得到该组训练图像对应的对齐后的训练图像组，之后将对齐后的训练图像组中的第一图像输入预设网络模型。在本实施例中，所述对齐处理是在获取到训练图像集后，分别对每一组训练图像组进行，并在所有训练图像组完成对齐处理后，在执行将训练图像集中第一图像输入预设网络模型的操作。
进一步，所述将该组训练图像组中的第一图像与所述第一图像对应的第二图像进行对齐处理指的是将以第二图像为基准，将第一图像中像素点与第二图像中与其对应的像素点对齐，以使得第一图像中像素点与第二图像中像素点的对齐率可以达到预设值，例如，99％等。其中，所述第一图像中像素点与第二图像中与其对应的像素点对齐指的是：对于第一图像中的第一像素点和第二图像中与第一像素点相对应的第二像素点，若第一像素点对应的像素坐标与第二像素点对应的像素坐标相同，那么第一像素点与第二像素点对齐；若第一像素点对应的像素坐标与第二像素点对应的像素坐标不相同那么第一像素点与第二像素点对齐。所述对齐图像指的通过对第一图像进行对齐处理得到图像，并且对齐图像中每个像素点与第二图像中其对应的像素点的像素坐标相同。此外，在得到对齐图像后，采用所述对齐图像替换其对应的第一图像以更新训练图像组，以使得更新后的训练图像组中的第一图像和第二图像在空间上对齐。
进一步，由于不同组训练图像组中的第一图像和第二图像的对齐程度不同，从而可以在实现对齐的基础上，针对不同对齐程度的第一图像和第二图像可以采用不同的对齐方式，以使得各组训练图像组均可以采用复杂度低的对齐方式进行对齐处理。由此，在本实施例的一个实现方式中，所述将该组训练图像组中的第一图像与所述第一图像对应的第二图像进行对齐处理具体包括：
P11、获取该组训练图像组中的第一图像与所述第一图像对应的第二图像之间的像素偏差量；
P12、根据所述像素偏差量确定所述第一图像对应的对齐方式，并采用所述对齐方 式将所述第一图像与所述第二图像进行对齐处理。
具体地，所述像素偏差量指的是第一图像中第一像素点与第二图像中与该第一像素点对应的第二像素点不对齐的第一像素点的数量。所述像素偏差量可以通过获取第一图像中各第一像素点的第一坐标，以及第二图像中各第二像素点的第二坐标，然后将第一像素点的第一坐标与其对应的第二像素点的第二坐标进行比较，若第一坐标与第二坐标相同，则判定第一像素点与其对应的第二像素点对齐；若第一坐标与第二坐标不相同，则判定第一像素点与其对应的第二像素点不对齐，最后获取所有不对齐的第一像素点的数量，以得到所述像素偏差量。例如，当所述第一图像中的第一像素点的第一坐标为(100,100)，第二图像中与所述第一像素点对应的第二像素点的第二坐标为(101,100)时，所述第一像素点与第二像素点不对齐，不对齐第一像素点的数量加一；当所述第一图像中的第一像素点的第一坐标为(100,100)，第二图像中与所述第一像素点对应的第二像素点的第二坐标为(100,100)是，所述第一像素点与第二像素点对齐，不对齐第一像素点的不变。
进一步，为了确定像素偏差量与对齐方式的对应关系，可以需要设置偏差量阈值，在获取到第一图像的像素偏差量时，可以通过将获取到的像素偏差量与预设偏差量阈值进行比较，以确定像素偏差量对应的对齐方式。由此，在本实施例的一个实现方式中，所述根据所述像素偏差量确定所述述第一图像对应的对齐方式，并采用所述对齐方式将所述第一图像与所述第二图像进行对齐处理具体包括：
P121、当所述像素偏差量小于或等于预设偏差量阈值时，根据所述第一图像与所述第二图像的互信息，以所述第二图像为基准对所述第一图像进行对齐处理；
P122、当所述像素偏差量大于所述预设偏差量阈值时，提取所述第一图像的第一像素点集和所述第二图像的第二像素点集，所述第一像素点集包含所述第一图像中的若干第一像素点，所述第二像素点集包括所述第二图像中的若干第二像素点，所述第二像素点集中的第二像素点与所述第一像素点集中的第一像素点一一对应；针对所述第一像素点集中每个第一像素点，计算该第一像素点与其对应的第二像素点的坐标差值，并根据该第一像素点对应的坐标差值对该第一像素点进行位置调整，以将该第一像素点与该第一像素点对应的第二像素点对齐。
具体地，所述预设偏差量阈值为预先设置，例如，预设偏差量阈值为20。所述当所述像素偏差量小于或等于预设偏差量阈值时指的是当将所述像素偏差量与所述预设偏差量阈值时，所述像素偏差量小于等于预设偏差量阈值。而当所述像素偏差量小于等 于预设偏差量阈值时，说明第一图像和第二图像在空间上的偏差较小，此时可以采用根据所述第一图像与所述第二图像的互信息对第一图像和第二图像进行对齐。在本实施例中，以所述第一图像和其对应的第二图像之间互信息对第一图像和第二图像进行对齐的过程可以采用图像配准方法，所述图像配准方法中以互信息作为度量准则，通过优化器对度量准则迭代进行优化以得到对齐参数，通过所述配准所述对齐参数的配准器将第一图像与第二图像进行对齐，这保证第一图像与第二图像的对齐效果的基础，降低了第一图像与第二图像对齐的复杂性，从而提高了对齐效率。在本实施例中，所述优化器主要采用平移和旋转变换，以通过所述平移和旋转变换来优化度量准则。
进一步，所述像素偏差量大于所述预设偏差量阈值，说明第一图像和第二图像在空间上不对齐程度较高，此时需要着重考虑对齐效果。从而此时可以采用通过选取第一图像中的第一像素点集和第二图像中第二像素点集的方式对第一图像和第二图像进行对齐。所述第一像素点集的第一像素点与第二像素点集中第二像素点一一对应，以使得对于第一像素点集中的任一第一像素点，在第二像素点集中均可以找到一个第二像素点，所述第二像素点在第二图像中的位置与第一像素点在第一图像中的位置相对应。此外，所述第一像素点集和第二像素点集可以是在获取到第一像素点集/第二像素点集后，根据第一像素点与第二像素点的对应关系确定第二像素点集/第一像素点集，例如，所述第一像素点集通过在第一图像中随机选取多个第一像素点的方式生成，第二像素点则是根据第一像素点集包含的各第一像素点确定的。
同时在本实施例中，所述第一像素点集和第二像素点集均是通过尺度不变特征变换(Scale-invariant feature transform，sift)的方式获取得到，即所述第一像素点集中第一像素点为第一图像中第一sift特征点，所述第二像素点集中第二像素点为第二图像的第二sift特征点。相应的，所述计算该第一像素点与其对应的第二像素点的坐标差值为将第一像素点中第一sift特征点与第二像素点集中第二sift特征点进行点对点匹配，以得到各第一sift特征点与其对应的各第二sift特征点的坐标差值，并根据该第一sift特征点对应的坐标差值对该第一sift特征点进行位置变换，以将该第一像素点与该第一sift特征点对应的第二sift特征点对齐，从而使得第一图像中第一sift特征点与第二图像中第二sift特征点位置相同，从而实现了第一图像与第二图像的对齐。
进一步，在本实施的一个实现方式中，所述第一图像为通过对第二图像进行预处理得到的具有重影的图像，并且第一图像和第二图像的图像尺寸和图像内容均相同，这 样可以提高第一图像对应的场景和成像参数与第二图像对应的场景和成像参数的相似程度，而通过场景相似程度高的训练图像组对预设网络模型进行训练，可以提高预设网络模型的训练速度以及训练得到的图像处理模型的处理效果。
进一步，所述对所述第二图像进行预处理的具体过程可以为：首先根据遮光结构生成灰度图，其次根据所述灰度图生成点扩散函数，最后根据点扩散函数和第二图像生成第一图像。也就是说，所述第一图像为根据第二图像和点扩散函数生成的，所述点扩散函数为根据按照遮光结构生成的灰度图生成的。所述点扩散函数PSF(point spread function)用于描述成像系统对点光源或点对象的响应，所述点扩散函数为成像系统光传递函数的空间域形成的。
进一步，如图19所示，所述遮光结构可以包括终端设备的显示面板的信号线路、电容线以及电源线等。所述信号线路可以包括若干数据线(如，S1，S2，...，Sn，其中，n为正整数)和若干扫描线(如，G1，G2，...，Gm，其中，m为正整数)，所述若干数据线与若干扫描线横纵交错布置而形成多个网格。所述信号线路形成的多个网格与显示面板配置的多个像素点相对应，当光线通过显示面板时，光线可以透射每个像素点，而无法透射遮光结构，而使得照射在遮光结构上的光线产生衍射。那么当成像系统设置于显示面板下方时，位于成像系统上方的显示面板内的若干遮光结构及像素点会出现在拍摄区域内，而在成像系统拍摄时，位于成像系统上方的遮光结构会造成拍摄图像产生重影等模糊不清的问题。由此，在对第二图像进行处理时，根据遮光结构来生成的灰度图对应的点扩散函数对第二图像进行处理，来生成第二图像对应的具有重影的第一图像，这样可以保证第一图像对应的图像内容和成像参数与第二图像对应的图像内容和成像参数的相同，从而提高图像处理模型的训练速度以及训练得到的图像处理模型的处理效果。需要强调的是，第一图像和第二图像的图像尺寸也可以存在一定的误差范围，即第一图像和第二图像的图像尺寸也可以有所区别。其中，两张图像的图像内容相同可以理解为仅指两张图像中各自所包含的物体(比如图像中的人、物、背景)相同，但并不表示各个物体分别在这两张图像中的画质是相同的，即图像内容相同可以说明两张图像中各自所包含的物体(比如图像中的人、物、背景)相同，但不能说明各个物体分别在这两张图像中的画质是相同的。
例如，屏上成像系统为具有屏上摄像头的终端设备，所述终端设备的遮光结构(如，信号线路)如图19所示，假设屏下成像系统配置于所述遮光结构对应的显示面板下方，那么，可以根据所述遮光结构(例如包括所述信号线路)对应的灰度图，图20所述的 灰度图可以为遮光结构对应的灰度图的局部区域，其中，第一黑线71可以对应信号线路中的数据线，第二黑线72可以对应信号线路中的扫描线。当然，在实际应用中，可以直接获取显示面板对应的全部遮光结构，也可以获取显示面板对应的遮光结构的部分区域，再通过所述部分区域模拟得到显示面板的全部遮光结构，只要可以获取到避光结构即可。
进一步，在所述第一图像为通过对第二图像进行预处理得到的图像时，所述训练图像组的获取过程可以为：首先获取终端显示面板的信号线路，并在所述信号线路中选取信号线路区域，其次确定所述信号线路区域对应的灰度图，根据所述灰度图以及夫琅禾费衍射公式生成点扩散函数；再通过所述屏上成像系统拍摄第二图像，并将第二图像与点扩散函数做卷积以得到第二图像对应的第一图像；最后将第二图像与根据第二图像生成的第一图像进行关联，得到训练图像组。当然，在实际应用中，在通过屏上成像系统拍摄第二图像时，可以获取多张第二图像，之后再将每张第二图像依次与点扩散函数做卷积，依次得到每张第二图像对应的第一图像，从而得到多组训练图像组，这样可以在拍摄得到所有训练图像组需要的第二图像后，再通过计算得到每张第二图像对应的第一图像，以提高训练图像组的获取速度。
此外，在本实施例的一个实现方式中，由于不同的显示面板对应的信号线路包含的网格尺寸可以不同，从而在生成灰度图时，可以获取多个显示面板的信号线路，并根据每个信号线路生成一张灰度图，对于获取第二图像可以在生成的多个灰度图中随机选取一张灰度图，并通过选取到的灰度图对应的点扩散函数对第二图像进行处理以得到该第二图像对应的第一图像，这样可以提高图像处理模型训练的去重影的效果。
进一步，在本实施例的一个实现方式中，如图21和22所示，所述预设网络模型包括编码器和解码器；所述预设网络模型根据训练图像集中第一图像，生成所述第一图像对应的生成图像具体包括：
N11、将所述训练图像集中第一图像输入所述编码器，通过所述编码器得到所述第一图像的特征图像，其中，所述特征图像的图像尺寸小于所述第一图像的图像尺寸；
N12、将所述特征图像输入所述解码器，通过所述解码器输出所述生成图像，其中，所述生成图像的图像尺寸等于第一图像的图像尺寸。
具体地，所述预设网络模型采用解码-编码结构，所述解码-编码结构为卷积神经网络CNN结构，其中，所述编码器100用于将输入图像转换为图像空间尺寸小于输入图像并且通道数多于输入图像的特征图像，所述解码器200用于将特征图像转换为与输入 图像的图像尺寸相同的生成图像。在本实施例中，所述编码器包括依次布置的第一冗余学习层101以及下采样层102，训练图像组中第一图像输入至第一冗余学习层101，通过第一冗余学习层101输出与第一图像的图像尺寸相同的第一特征图；第一特征图像作为下采样层102的输入项输入下采样层102，通过下采样层102对第一特征图像进行下采样以输出所述第一图像对应的第二特征图像(第二特征图像为通过编码器生成的第一图像的特征图像)，其中，第二特征图像的图像尺寸小于第一图像的图像尺寸。所述解码器200包括依次布置的上采样层201和第二冗余学习层202，所述编码器100输出的特征图像输入至上采样层201，通过上采样层201进行上采样后输出第三特征图像，第三特征图像输入至第二冗余学习层202，经过第二冗余学习层202后输出生成图像，其中，所述生成图像的图像尺寸与第一图像的图像尺寸相同。本实施通过采用编码器-解码器的结构，可以对预设网络模型进行多尺度的训练，从而可以提高训练得到的图像处理模型的去重影效果。
进一步，如图22所示，所述第一冗余学习层101包括第一卷积层11以及以第一冗余学习模块12，所述下采样层102包括第一编码冗余学习模块110和第二编码冗余学习模块120，第一编码冗余学习模块110包括第一下采样卷积层13和第二冗余学习模块14，第二编码冗余学习模块120包括第二下采样卷积层15和第三冗余学习模块16。其中，所述第一卷积层11的输入项为第一图像，并对第一图像进行采样以得到第第一特征图像，并将所述第一特征图像输入至第一冗余学习模块12进行特征提取，经过第一冗余学习模块12的第一特征图像依次通过第一下采样卷积层、第二冗余学习模块14、第二下采样卷积层15和第三冗余学习模块16进行下采样，以得到第二特征图像。由此可知，所述第一卷积层11对第一图像进行采样，所述第一下采样卷积层13和第二下采样卷积层15均用于对输入其的特征图像进行下采样，所述第一冗余学习模块12、第二冗余学习模块14和第三冗余学习模块16用于提取图像特征。此外，在本实施例的一种可能的实现方式中，所述第一下采样卷积层13和第二下采样卷积层15可以均为采用步长为2的卷积层，所述第一冗余学习模块12、第二冗余学习模块14和第三冗余学习模块16均包括依次布置的三个冗余学习块，所述三个冗余学习块依次提取输入图像的图像特征。
举例说明：假设第一图像为256*256的图像，第一图像通过输入层输入第一冗余学习层101，经过第一冗余学习层101后输出256*256的第一特征图像；第一特征图像输入第一编码冗余学习模块110的第一下采样卷积层13，经过第一下采样卷积层13输 出图像大小为128*128的第四特征图像，第四特征图像经过第一编码冗余学习模块110的第一冗余学习模块12进行特征提取；经过第一冗余学习模块12的第四特征图像输入第二编码冗余学习模块120的第二下采样卷积层15，经过第二下采样卷积层15输出图像大小为64*64的第二特征图像，第二特征图像经过第二编码冗余学习模块120的第二冗余学习模块16进行特征提取。
进一步，如图22所示，所述上采样层201包括第一解码冗余学习模块210和第二解码冗余学习模块220，第一解码冗余学习模块210包括第四冗余学习模块21和第一上采样卷积层22，第二解码冗余学习模块220包括第五冗余学习模块23和第二上采样卷积层24，所述第二冗余学习层202包括第六冗余学习模块25和第二卷积层26。其中，所述第一上采样卷积层22的输入项为第一特征图像，输入第一特征图像依次通过第四冗余学习模块21、第一上采样卷积层22、第五冗余学习模块23和第二上采样卷积层24进行上采样以得到第三特征图像，并将所述第三特征图像输入至第六冗余学习模块25，通过第六冗余学习模块25进行特征提取后的第三特征图像输入至第二卷积层26，通过第二卷积层26得到生成图像。由此可知，所述第一上采样卷积层22和第二上采样卷积层24用于对输入其的特征图像进行上采样，所述第四冗余学习模块21、第五冗余学习模块23以及第六冗余学习模块25均用于提取图像特征，所述第二卷积层26用于对输入其内的特征图像进行采样。在本实施例的一个可能的实现方式中，所述第一上采样卷积层22和第二上采样卷积层24均为步长为2的反卷积层，所述第四冗余学习模块21、第五冗余学习模块23以及第六冗余学习模块25均包括三个冗余学习块，所述三个冗余学习块依次提取输入图像的图像特征。此外，所述第一冗余学习层101中冗余学习模块的第三个冗余学习块与所述第二冗余学习层202中冗余学习模块的第一个冗余学习块跳跃连接，所述第一编码冗余学习模块110中冗余学习模块的第三个冗余学习块与所述第二解码冗余学习模块220中冗余学习模块的第一个冗余学习块跳跃连接。
举例说明：假设第一图像为256*256的图像经过上述编码器100得到64*64的第二特征图像，64*64的第二特征图像输入经过第一解码冗余学习模块210的第四冗余学习模块21进行特征提取，经过特征提取的64*64的第二特征图像输入第一解码冗余学习模块210的第一上采样卷积层22，经过第一上采样卷积层22输出的图像大小为128*128的第五特征图像，第五特征图像经过第二解码冗余学习模块220的第五冗余学习模块23进行特征提取；经过第五冗余学习模块23的第五特征图像输出第二解码冗余学习模块220的第二上采样卷积层24，经过第二上采样卷积层24输出图像大小为 256*256的第三特征图像，第三特征图像输入第二冗余学习层202，经过第二冗余学习层202后输出256*256的生成图像。
进一步，所述编码器和解码器包括的第一卷积层、第二卷积层、第一上采样卷积层、第二上采样卷积层、第一下采用卷积层和第二下采用卷积层以及所有冗余学习模块中的卷积层均使用线性整流函数作为激活函数且卷积核均为5*5，这样可以提高各层的梯度传递效率，并且经过多次的反向传播，梯度幅度变化小，提高了训练的生成器的准确性，同时还可以增大网络的感受野。
N20、所述预设网络模型根据所述第一图像对应的第二图像和所述第一图像对应的生成图像，对所述预设网络模型的模型参数进行修正，并继续执行根据训练图像集中的下一训练图像组中的第一图像，生成所述第一图像对应的生成图像的步骤，直至所述预设网络模型的训练情况满足预设条件，以得到已训练的图像处理模型。
具体地，所述对所述预设网络模型进行修正指的是对所述预设网络模型的模型参数进行修正，直至所述模型参数满足预设条件。所述预设条件包括损失函数值满足预设要求或者训练次数达到预设次数。所述预设要求可以是根据图像处理模型的精度来确定，这里不做详细说明，所述预设次数可以为预设网络模型的最大训练次数，例如，4000次等。由此，在预设网络模型输出生成图像，根据所述生成图像以及所述第二图像来计算预设网络模型的损失函数值，在计算得到损失函数值后，判断所述损失函数值是否满足预设要求；若损失函数值满足预设要求，则结束训练；若损失函数值不满足预设要求，则判断所述预设网络模型的训练次数是否达到预测次数，若未达到预设次数，则根据所述损失函数值对所述预设网络模型的网络参数进行修正；若达到预设次数，则结束训练。这样通过损失函数值和训练次数来判断预设网络模型训练是否结束，可以避免因损失函数值无法达到预设要求而造成预设网络模型的训练进入死循环。
进一步，由于对预设网络模型的网络参数进行修改是在预设网络模型的训练情况未满足预设条件(例如，损失函数值未满足预设要求并且训练次数未达到预设次数)，从而在根据损失函数值对所述预设网络模型的网络参数进行修正后，需要继续对网络模型进行训练，即继续执行将训练图像集中的第一图像输入预设网络模型的步骤。其中，继续执行将训练图像集中第一图像输入预设网络模型中的第一图像可以为未作为输入项输入过预设网络模型的第一图像。例如，训练图像集中所有第一图像具有唯一图像标识(例如，图像编号)，第一次训练输入预设网络模型的第一图像的图像标识与第二次训练输入预设网络模型的第一图像的图像标识不同，如，第一次训练输入预设网络模型 的第一图像的图像编号为1，第二次训练输入预设网络模型的第一图像的图像编号为2，第N次训练输入预设网络模型的第一图像的图像编号为N。当然，在实际应用中，由于训练图像集中的第一图像的数量有限，为了提高图像处理模型的训练效果，可以依次将训练图像集中的第一图像输入至预设网络模型以对预设网络模型进行训练，当训练图像集中的所有第一图像均输入过预设网络模型后，可以继续执行依次将训练图像集中的第一图像输入至预设网络模型的操作，以使得训练图像集中的训练图像组按循环输入至预设网络模型。需要说明的是，在将第一图像输入预设网络模型训练的过程中，可以按照各个第一图像的图像编号顺序输入，也可以不按照各个第一图像的图像编号顺序输入，当然，可以重复使用同一张第一图像对预设网络模型进行训练，也可以不重复使用同一张第一图像对预设网络模型进行训练，在本实施例中，不对“继续执行将训练图像集中的第一图像输入预设网络模型的步骤”的具体实现方式进行限定。
进一步，在本实施例的一个实现方式中，所述损失函数值为结构相似性损失函数和内容双向损失函数计算得到的。相应的，如图23所示，所述预设网络模型根据所述第一图像对应的第二图像和所述第一图像对应的生成图像，对所述预设网络模型的模型参数进行修正，并继续执行根据训练图像集中的下一训练图像组中的第一图像，生成所述第一图像对应的生成图像的步骤，直至所述预设网络模型的训练情况满足预设条件，以得到已训练的图像处理模型具体包括：
N21、根据所述第一图像对应的第二图像和所述第一图像对应的生成图像计算所述预设网络模型对应的结构相似性损失函数值和内容双向损失函数值；
N22、根据所述结构相似性损失函数值和所述内容双向损失函数值得到所述预设网络模型的总损失函数值；
N23、基于所述总损失函数值训练所述预设网络模型，并继续执行根据训练图像集中的下一训练图像组中的第一图像，生成所述第一图像对应的生成图像的步骤，直至所述预设网络模型的训练情况满足预设条件，以得到已训练的图像处理模型。
具体地，所述预设网络模型采用结构相似性指数(Structural similarity index，SSIM)损失函数和基于VGG(Visual Geometry Group Network，VGG网络)提取特征的内容双向(Contextual bilateral loss,CoBi)损失函数的结合作为损失函数。那么，在计算所述预设网络模型的损失函数值时，可以分别计算结构相似性损失函数值和内容双向损失函数值，再根据所述结构相似性损失函数值和内容双向损失函数值计算预设网络模型中损失函数值。在本实施例中，所述预设网络模型的总损失函数值＝a*结构相似 性损失函数值+b*内容双向损失函数值，所述a和b为权重系数。例如，所述权重系数a和权重系数b均为1，那么所述预设网络模型的总损失函数值＝结构相似性损失函数值+内容双向损失函数值。此外，在本实施例中，在采用总损失函数值对预设网络模型进行训练时采用随机梯度下降法对预设网络模型进行训练，其中，训练的初始网络参数设为0.0001，并且网络参数在修正时采用指数衰减的方式进行修正。
进一步，所述结构相似性损失函数值用于衡量生成图像与第二图像之间结构的相似性，所述结构相似性损失函数值越大，生成图像与第二图像的相似性越高，反之，所述结构相似性损失函数值越小，生成图像与第二图像的相似性越低。因此，结构相似性损失函数值对局部结构变化较敏感，更接近于人眼的感知系统，从而可以提高预设网络模型的精确性。在本实施例中，所述结构相似性损失函数值对应的结构相似性损失函数的表达式可以为：
Figure PCTCN2020141932-appb-000001
其中，μ x为生成图像中所有像素点的像素值的平均值，μ y为第二图像中所有像素点的像素值的平均值，σ x为生成图像中所有像素点的像素值的方差，σ y为第二图像中所有像素点的像素值的方差，σ xy为生成图像与第二图像的协方差。
进一步，所述内容双向损失函数值为通过基于VGG特征的CoBi损失函数计算得到，所述基于VGG特征的CoBi损失函数通过分别提取生成图像与第二图像的若干组VGG特征，并且针对生成图像的每个第一VGG特征，在第二图像的第二VGG特征中搜索与该第一VGG特征接近的第二VGG特征匹配，最后计算每个第一VGG特征与其匹配的第二VGG特征的距离和，以得到内容双向损失函数值，这样通过内容双向损失函数对双边距离进行搜索，考虑了第一VGG特征与其匹配的第二VGG特征在空间上的损失，从而可以避免第一图像和第二图像未完全对齐产生的影响，提高了预设网络模型训练的速度以及准确性。此外，在搜索所述第一VGG特征匹配的第二VGG特征时，根据第一VGG特征和第二VGG特征的距离和位置关系两个方面确定内容双向损失函数值，提高了匹配的精确性，从而进一步降低第一图像和第二图像不对齐对预设网络模型训练的影响。在本实施例中，所述内容双向损失函数的表达式可以为：
Figure PCTCN2020141932-appb-000002
其中,D为生成图像的VVG特征与第二图像的VVG特征之间的余弦距离，D′为生 成图像的VVG特征与第二图像的VVG特征之间的空间位置距离，N为生成图像的VVG特征的特征数量，ω s为权重系数。
基于上述图像处理模型的生成方法，本发明还提供了一种图像处理方法，所述方法应用如上述实施例所述的图像处理模型的生成方法训练得到图像处理模型，如图24所示，所述图像处理方法包括：
E100、获取待处理图像，并将所述待处理图像输入至所述图像处理模型。
具体地，所述待处理图像可以为通过屏下成像系统拍摄得到的图像，也可以为预先设置的图像，还可以为根据接收到的选取操作而确定的图像。在本实施例中，所述待处理图像优选为通过屏下成像系统拍摄得到的图像，例如，所述待处理图像为通过配置有屏下成像系统的手机拍摄得到的人物图像。
E200、通过所述图像处理模型对所述待处理图像进行去重影处理，以得到所述待处理图像对应的输出图像。
具体地，所述通过所述图像处理模型对所述待处理图像进行去重影指的是将所述待处理图像作为所述图像处理模型的输入项输入至所述图像处理模型中，通过所述图像处理模型去除所述待处理图像的重影，以得到输出图像，其中，所述输出图像为所述待处理图像经过去重影处理所得到图像。可以理解的是，待处理图像为输出图像对应的具有重影的图像，即输出图像与待处理图像相对应，它们呈现的是同一图像场景，输出图像为正常显示的图像，待处理图像的图像内容与输出图像对应，但待处理图像内容中的物体出现重影或者与重影类似的模糊效果。例如，如图25所示的待处理图像通过所述去重影处理后得到如图26所示的输出图像。
进一步，由所述图像处理模型的训练过程可以知道，所述图像处理模型包括编码器和解码器，从而在通过图像处理模型对应待处理图像进行处理时，需要分别通过编码器和解码器进行处理。相应的，所述通过所述图像处理模型对所述待处理图像进行去重影，以得到所述待处理图像对应的输出图像具体包括：
E201、将所述待处理图像输入所述编码器，通过所述编码器得到所述待处理图像的特征图像，其中，所述特征图像的图像尺寸小于所述待处理图像的图像尺寸；
E202、将所述特征图像输入所述解码器，通过所述解码器输出所述待处理图像对应的输出图像，其中，所述输出图像的图像尺寸等于所述待处理图像的图像尺寸。
具体地，所述编码器将输入的待处理图像转换为图像空间尺寸小于输入图像并且通道数多于输入图像的特征图像，并将所述特征图像输入至解码器，所述解码器将输入 的特征图像转换为与待处理图像的图像尺寸相同的生成图像。其中，所述编码器的结构与预设网络模型中的编码器的结构相同，具体可以参照预设网络模型中的编码器的结构的说明。所述图像处理模型的编码器的对待处理图像的处理与预设网络模型中的编码器对第一图像的处理过程相同，从而所述步骤E201的具体执行过程可以参照步骤N11。同样的，所述解码器的结构与预设网络模型中的解码器的结构相同，具体可以参照预设网络模型中的解码器的结构的说明。所述图像处理模型的解码器的对待处理图像对应的特征图像的处理与预设网络模型中的解码器对第一图像对应的特征图像的处理过程相同，从而所述步骤E202的具体执行过程可以参照步骤N12。
可以理解的是，图像处理模型在训练过程中对应的网络结构，与在应用过程(去除输出图像携带的重影)中所对应的网络结构相同。例如，在训练的过程中，图像处理模型包括编码器和编码器，那么相应地，在通过图像处理模型去除输出图像携带的重影时，图像处理模型也包括编码器和编码器。
进一步地，例如，在训练过程中，图像处理模型的编码器包括所述编码器包括第一冗余学习层和下采样层，解码器包括上采样层和第二冗余学习层；相应地，在通过图像处理模型去除输出图像携带的重影时，编码器也可以包括第一冗余学习层和下采样层，解码器包括上采样层和第二冗余学习层；并且在应用过程中，每一层的工作原理与在训练过程中每一层的工作原理相同，因此，图像处理模型应用过程中的每一层神经网络的输入输出情况可以参见图像处理模型的训练过程中的相关介绍，这里不再赘述。
进一步，为了进一步提高输出图像的图像质量，在获取到图像处理模型输出的输出图像后，还可以对所述输出图像进行后处理，其中，所述后处理可以包括锐化处理以及降噪处理等。相应的，所述通过所述图像处理模型对所述待处理图像进行去重影处理，以得到所待处理图像对应的输出图像之后还包括：
对所述输出图像进行锐化以及降噪处理，并将锐化以及降噪处理后的输出图像作为所述待处理图像对应的输出图像。
具体地，所述锐化处理指的是补偿输出图像的轮廓、增强输出图像的边缘及灰度跳变的部分，以提高输出图像的图像质量。其中，所述锐化处理可以采用现有的锐化处理方法，例如，高通滤波方法等。所述降噪处理指的是去除图像中的噪声，提高图像的信噪比。其中，所述降噪处理可以采用现有的降噪算法或已训练的降噪网络模型等，例如，所述降噪处理采用高斯低通滤波方法等。
基于上述图像处理模型的生成方法，如图36所示，本实施例提供了一种图像处理 模型的生成装置，其中，所述图像处理模型的生成装置包括：
第二生成模块301，用于利用预设网络模型根据训练图像集中的第一图像生成所述第一图像对应的生成图像；其中，所述训练图像集包括多组训练图像组，每一组训练图像组包括第一图像和第二图像，第一图像为第二图像对应的具有重影的图像；
第二修正模块302，用于利用所述预设网络模型根据所述第一图像对应的第二图像和所述第一图像对应的生成图像，对所述预设网络模型的模型参数进行修正，并继续执行根据训练图像集中的下一训练图像组中的第一图像，生成所述第一图像对应的生成图像的步骤，直至所述预设网络模型的训练情况满足预设条件，以得到已训练的图像处理模型。
在一个实施例中，所述预设网络模型包括编码器和解码器；所述第二生成模块具体用于：
将所述训练图像集中第一图像输入所述编码器，通过所述编码器得到所述第一图像的特征图像；以及将所述特征图像输入所述解码器，通过所述解码器输出所述生成图像，其中，所述特征图像的图像尺寸小于所述第一图像的图像尺寸；所述生成图像的图像尺寸等于第一图像的图像尺寸。
在一个实施例中，所述第二修正模块具体用于：
根据所述第一图像对应的第二图像和所述第一图像对应的生成图像分别计算所述预设网络模型对应的结构相似性损失函数值和内容双向损失函数值；根据所述结构相似性损失函数值和所述内容双向损失函数值得到所述预设网络模型的总损失函数值；以及基于所述总损失函数值训练所述预设网络模型，并继续执行根据训练图像集中的下一训练图像组中的第一图像，生成所述第一图像对应的生成图像的步骤，直至所述预设网络模型的训练情况满足预设条件，以得到已训练的图像处理模型。
在一个实施例中，所述第一图像为根据第二图像和点扩散函数生成的，其中，所述点扩散函数为根据屏下成像系统中的遮光结构生成的灰度图生成的。
在一个实施例中，所述第一图像为通过屏下成像系统拍摄得到的图像。
在一个实施例中，所述屏下成像系统为屏下摄像头。
在一个实施例中，所述图像处理模型的生成装置还包括：
第二对齐模块，用于针对所述训练图像集中每组训练图像组，将该组训练图像组中的第一图像与所述第一图像对应的第二图像进行对齐处理，得到与所述第二图像对齐的对齐图像，并将所述对齐图像作为第一图像。
在一个实施例中，所述第二对齐模块具体用于：
针对所述训练图像集中每组训练图像组，获取该组训练图像组中的第一图像与所述第一图像对应的第二图像之间的像素偏差量；根据所述像素偏差量确定所述第一图像对应的对齐方式，并采用所述对齐方式将所述第一图像与所述第二图像进行对齐处理，得到与所述第二图像对齐的对齐图像；以及将所述对齐图像作为第一图像。
在一个实施例中，所述第二对齐模块具体用于：
当所述像素偏差量小于或等于预设偏差量阈值时，根据所述第一图像与所述第二图像的互信息，以所述第二图像为基准对所述第一图像进行对齐处理；
当所述像素偏差量大于所述预设偏差量阈值时，提取所述第一图像的第一像素点集和所述第二图像的第二像素点集，所述第一像素点集包含所述第一图像中的若干第一像素点，所述第二像素点集包括所述第二图像中的若干第二像素点，所述第二像素点集中的第二像素点与所述第一像素点集中的第一像素点一一对应；针对所述第一像素点集中每个第一像素点，计算该第一像素点与其对应的第二像素点的坐标差值，并根据该第一像素点对应的坐标差值对该第一像素点进行位置调整，以将该第一像素点与该第一像素点对应的第二像素点对齐。
基于上述一种图像处理方法，如图37所示，本实施例提供了一种图像处理装置，所述图像处理装置包括：
第二获取模块401，用于获取待处理图像，并将所述待处理图像输入至所述图像处理模型；
第二处理模块402，用于通过所述图像处理模型对所述待处理图像进行去重影处理，以得到所述待处理图像对应的输出图像。
在一个实施例中，所述图像处理模型包括编码器和解码器；所述第二处理模块具体包括：
将所述待处理图像输入所述编码器，通过所述编码器得到所述待处理图像的特征图像；以及将所述特征图像输入所述解码器，通过所述解码器输出所述待处理图像对应的输出图像，其中，所述特征图像的图像尺寸小于所述待处理图像的图像尺寸；所述输出图像的图像尺寸等于所述待处理图像的图像尺寸。
在一个实施例中，所述图像处理装置还包括：
锐化模块，用于对所述输出图像进行锐化以及降噪处理，并将锐化以及降噪处理后的输出图像作为所述待处理图像对应的输出图像。
实施例三
本实施提供了一种图像处理方法，如图27所示，所述方法包括：
H10、获取待处理图像集。
具体地，所述待处理图像集包括至少两张图像，所述待处理图像集中的各张图像的获取方式可以包括：通过成像系统(如，屏下摄像头等)拍摄得到的、外部设备(如，智能手机等)发送得到的，以及通过网络(如，百度等)。在本实施例的中，所述待处理图像集包含的各张图像均为低曝光图像，并且所述待处理图像集中各去噪图像均是通过成像系统(如，照相机、摄像机、屏下摄像头等)拍摄的得到，并且各去噪图像属于相同的色彩空间(如，RGB色彩空间以及YUV色彩空间等)相同。例如，所述各去噪图像均是通过屏下摄像头拍摄得到，并且基础图像和各张临近图像均属于RGB色彩空间。
进一步，所述待处理图像集中的各待处理图像对应的拍摄场景均相同，并且各待处理图像的拍摄参数也可以相同，其中，所述拍摄参数可以包括环境照度以及曝光参数，其中，所述曝光参数可以包括光圈、开门速度、感光度、对焦以及白平衡等。当然，在实际应用中，所述拍摄参数还可以拍摄角度以及拍摄范围等。
进一步，由于在不同的环境照度下，成像系统拍摄的图像的噪声程度不同，例如，当环境照度低时，成像系统拍摄得到的图像携带的噪声较多，当环境照度高时，成像系统拍摄得到的图像携带的噪声较少。特别是对于屏下成像系统，由于显示面板对不同光线强度的吸收强度不同，并且显示面板对光线的吸收程度与光线强度为非线性光线(例如，当环境照度低时，光线强度低，显示面板吸收光线的比例高，当环境照度高时，光线强度高，显示面板吸收光线的比例低)，这使得屏下成像系统拍摄得到图像A的噪声强度高于图像B的噪声强度，其中，图像A对应的环境光强度小于图像A对应的环境光强度。由此，对于噪声强度不同的图像，可以采用不同数量的图像进行合成，例如，噪声强度高图像需要的图像数量大于噪声强度低图像需要的图像数量。相应的，所述待处理图像集包含的去噪图像的图像数量可以为根据所述待处理图像集对应的拍摄参数确定的，其中，所述拍摄参数至少包括包括环境照度。
此外，为了根据环境照度确定待处理图像的图像数量，可以预先设定环境照度区间与待处理图像的图像数量的对应关系。在获取到环境照度后，首先确定环境照度所处环境照度区间，在根据该对应关系确定该环境照度区间对应的待处理图像的图像数量，以得到待处理图像的图像数量。例如，所述环境照度区间与待处理图像的图像数量的对应关系为：当环境照度区间为[0.5,1)时，待处理图像的图像数量对应8；当环境照度为 [1,3)时，待处理图像的图像数量对应7；当环境照度为[3,10)时，待处理图像的图像数量对应6；当环境照度区间为[10,75)时，待处理图像的图像数量对应5；当环境照度区间为[75,300)时，待处理图像的图像数量对应4，当环境照度区间为[300,1000)时，待处理图像的图像数量对应3；当环境照度为[1000,5000)时，待处理图像的图像数量对应2。
进一步，在本实施例的一个实现方式中，所述待处理图像集为通过屏下成像系统拍摄得到，所述待处理图像集包含的待处理图像的图像数量是根据所述屏下成像系统拍摄图像时的环境照度确定。其中，所述环境照度可以在屏下成像系统启动时获取到的，也可以是根据拍摄得到的第一帧图像获取到，还可以是通过预先拍摄得到的预设数量的图像，再根据拍摄得到的预设数量的图像中任一张图像确定。
在本实施例的一个实现方式中，所述环境照度为在屏下成像系统启动时获取到的。相应的，所述待处理图像集的获取过程可以为：当屏下成像系统启动时，获取环境照度，并根据获取到环境照度确定待处理图像集包含的图像的第一图像数量，并通过屏下成像系统连续获取第一图像数量的图像，以得到所述待处理图像集。其中，所述第一图像数量可以根据预设设置的环境照度与待处理图像的图像数量的对应关系确定的。
在本实施例的一个实现方式中，所述环境照度是根据拍摄得到的第一帧图像获取到。相应的，所述待处理图像集的获取过程可以为：首先通过屏下成像系统获取第一帧图像，再获取第一帧图像的ISO值，并根据所述ISO值确定第一帧图像对应的环境照度，最后根据获取到环境照度确定待处理图像集包含的图像的第二预设图像数量，并通过屏下成像系统再连续获取第二图像数量减一张图像，以得到所述待处理图像集。
在本实施例的一个实现方式中，所述环境照度通过预先拍摄得到的预设数量的图像，再根据拍摄得到的预设数量的图像中任一张图像确定。所述待处理图像集的获取过程可以为：预先首先通过屏下成像系统获取预设数量的图像，并在获取到图像中随机选取一张第三预设图像，获取第三预设图像的ISO值，并根据所述ISO值确定第三预设图像对应的环境照度，最后根据获取到环境照度确定待处理图像集所包含图像的图像数量(即第三图像数量)。此外，由于已经获取到预设数量的图像，从而可以将预设数量与第三图像数量进行比较，若所述预设数量小于第三图像数量，则通过屏下成像系统再连续获取第四图像数量张图像，其中，第四图像数量等于第三图像数量减预设数量；若所述预设数量等于所述第三图像数量，则完成所述待处理图像集获取操作；若所述预设数量大于第三图像数量，则在获取到预设数量的图像中随机选取第三图像数量张图像，以 得到所述待处理图像集。
进一步，在本实施例的一个实现方式，在所述预设数量大于第三图像数量，为了使得待处理图像集包括第三预设图像，可以将所述第三预设图像添加至所述待处理图像集，之后再在获取到的图像中选取第三图像数量减一张图像。同时为了使得待处理图像集中的图像与第三预设图像连续，可以按照拍照顺序选取待处理图像集包含的图像。
举例说明：假设预设数量为5，5张图像按照拍摄顺序分别记为图像A、图像B、图像C、图像D以及图像E，第三图像数量为3，第三预设图像为按照拍摄时间顺序为3的图像C，那么按照拍摄顺序选取到的图像分别为图像B和图像D，从而所述待处理图像集包括图像B、图像C以及图像D。当然，在实际应用中，可以优先选取按照拍摄顺序从第三预设图像依次向前选取，当位于第三预设图像前的图像数量不够时，按照拍摄顺序从第三预设图像依次向后选取；也可以先向后选取，在向后的图像数量不够时，向前选取；还可以是其他选取方法，这里不做具体限制，只要是可以选取到第四图像数量的图像的方式均可以。
H20、根据所述待处理图像集，生成所述待处理图像集对应的去噪图像。
具体地，所述待处理图像集包括一基础图像和至少一张临近图像，其中，所述基础图像为待处理图像集中各待处理图像的图像基准，各临近图像可以以所述基础图像为参考基准与所述基础图像进行合成。由此，在根据所述待处理图像集生成去噪图像之前，需要在待处理图像集中选取一张图像作为基础图像，并将待处理图像集中除基础图像外的所有图像作为基础图像的临近图像。
进一步，由于待处理图像集包括一张基础图像和至少一张临近图像，从而需要在获取到图像中选取基础图像。其中，所述基础图像可以为按照获取顺序位于第一的图像，也可以是待处理图像集中任一张图像，还可以是待处理图像集中清晰度最高的一张图像。在本实施例中，所述基础图像为处理图像集中清晰度最高的一张图，即所述基础图像的清晰度大于或者等于任意一张临近图像的清晰度。
进一步，在本实施例的一个实现方式中，所述基础图像的确定过程可以：在获取到待处理图像集包含的所有图像后，获取各图像的清晰度，并将获取到各清晰度进行比较，以选取清晰度最大的图像，并将选取到的图像作为基础图像。其中，所述图像的清晰度可以理解为图像中地物边界(或物体边界)上的像素点的像素值和与该地物边界(或物体边界)相邻的像素点的像素值之间的差值；可以理解的是，若图像中地物边界(或物体边界)上的像素点的像素值和与该地物边界(或物体边界)相邻的像素点的像素值 之间的差值越大，则说明该图像的清晰度越高，反之，若图像中地物边界(或物体边界)上的像素点的像素值和与该地物边界(或物体边界)相邻的像素点的像素值之间的差值越小，则说明该图像的清晰度越低。也就是说，基础图像的清晰度高于各临近图像的清晰度，可以理解为，针对于每一张临近图像，基础图像中地物边界(或物体边界)上的像素点的像素值和与该地物边界(或物体边界)相邻的像素点的像素值之间的差值，大于该临近图像中地物边界(或物体边界)上的像素点的像素值和与该地物边界(或物体边界)相邻的像素点的像素值之间的差值。
为便于理解，接下来针对基础图像的清晰度高于临近图像的清晰度进行举例说明。假设待处理图像集包括图像A和一张图像B，且该图像A和该图像B中的图像内容是完全一样的，其中，该图像A和该图像B中均包括像素点a和像素点b，像素点a为图像中地物边界(或物体边界)上的像素点，而像素点b为与该地物边界(或物体边界)相邻的像素点；若图像A中像素点a的像素值与像素点b的像素值之间的差值为10，而该图像B中像素点a的像素值与像素点b的像素值之间的差值为30，则可以认为该图像B的清晰度高于该训练图像A的清晰度，因此，可以将该图像A作为该待处理图像集中的基础图像，将该图像B作为该待处理图像集中的临近图像。
进一步，在本实施例的一个实现方式中，在根据清晰度在待处理图像集中选取基础图像时，待处理图像集中存在多张清晰度相同的图像(记为图像C)，并且每张图像C的清晰度均不小于待处理图像集中的任一张图像的清晰度，那么所述多张图像C均可以作为基础图像。此时，可以在获取到多张图像C中随机选取一张图像C作为基础图像，也可以按照拍摄顺序，在多张图像C中选取位于第一位的图像C作为基础图像，还可以按照拍摄顺序，在多张图像C中选取位于最后一位的图像C作为基础图像。
进一步，在本实施例的一个实现方式中，如图28所示，所述根据所述待处理图像集，生成所述待处理图像集对应的去噪图像具体包括：
H21、将所述基础图像划分为若干基础图像块,分别确定各基础图像在各临近图像中对应的临近图像块；
H22、确定各个基础图像块分别对应的权重参数集；其中，基础图像块对应的权重参数集包括第一权重参数和第二权重参数，第一权重参数为基础图像块的权重参数，第二权重参数为临近图像中与基础图像块对应的临近图像块的权重参数；
H23、根据所述待处理图像集以及各个基础图像块分别对应的权重参数集，确定去噪图像。
具体地，在所述步骤H21中，所述基础图像块为所述基础图像的部分图像区域，并且所述若干基础图像块拼接后会形成所述基础图像。所述将基础图像划分为若干基础图像块指的是将所述基础图像作为一个区域，将所述区域划分为若干个子区域，每个子区域对应的图像区域为一个基础图像块，其中，将所述区域划分为若干个子区域可以为将所述区域等分为若干区域。例如，可以将8*8的基础图像分割成4个4*4的基础图像块。当然，在实际应用中，本实施例中的将基础图像划分为若干基础图像块的方法，可以根据具体的场景灵活选择，只要可以划分得到若干基础图像块的方法均可以。所述临近图像块为基础图像在临近图像中对应的图像块，所述临近图像块的图像块大小与所述临近图像块所对应的基础图像块的大小相同，并且所述临近图像块携带图像内容与所述基础图像块携带的图像内容相同。所述确定该基础图像块分别在每张临近图像中对应的临近图像块指的是在临近图像块的指定区域内选取与基础图像块相似度最高的图像块，其中，所述指定区域为根据基础图像块在基础图像中所处区域确定的。
进一步，在本实施例的一个实现方式中，如图29所示，所述分别确定各基础图像在各临近图像中对应的临近图像块具体包括：
A10、确定该基础图像块在基础图像中的区域范围，并根据所述区域范围确定临近图像中的指定区域；
A20、根据该基础图像块在指定区域内选取临近图像块，其中，所述临近图像块为指定区域内与基础图像块相似性最高的图像块，并且临近图像块的图像尺寸与基础图像块的图像尺寸相等。
具体地，所述区域范围指的是基础图像块在基础图像中所处的区域边界像素点的像素坐标形成的坐标点集合，例如，所述基础图像块为基础图像中的正方形区域，并且基础图像块的四个顶点的坐标点分别为(10,10)、(10,20)、(20,10)以及(20,20)，那么所述基础图像块对应的区域范围可以为{(10,10)、(10,20)、(20,10)以及(20,20)}。
所述指定区域为临近图像中的图像区域，基础图像块对应的区域范围可以与指定区域对应的区域范围相对应，即当临近图像与基础图像建立映射时，指定区域对应的区域范围与基础图像块对应的区域范围相对应。例如，在基础图像中，基础图像块对应的区域范围可以为{(10,10)、(10,20)、(20,10)以及(20,20)}，在临近图像中，指定区域对应的区域范围可以为{(10,10)、(10,20)、(20,10)以及(20,20)}，那么基础图像块对应的区域范围可以与指定区域对应的区域范围相对应。此外，基础图像块对应的区域范围也可以与指定区域的子区域对应的区域范围相对应，即当临近图像与基础图 像建立映射时，指定区域对应的区域范围中存在一个子区域，该子区域的区域范围与基础图像块对应的区域范围相对应。例如，基础图像块在基础图像中所占的图像区域对应的区域范围可以为{(10,10)、(10,20)、(20,10)以及(20,20)}，如图30所示，指定区域在临近图像中所占的图像区域12对应的区域范围可以为{(9,9)、(9,21)、(21,9)以及(21,21)}，那么指定区域包含子区域11，该子区域11的区域范围为{(10,10)、(10,20)、(20,10)以及(20,20)}，该子区域11对应的区域范围与基础图像块对应的区域范围相对应。
进一步，在本实施例的一个实现方式中，所述基础图像块对应的区域范围也可以与指定区域的子区域对应的区域范围相对应，并且所述指定区域为通过将区域范围对应的坐标点集合中各坐标点沿横纵或纵轴向远离区域范围的方向平移预设数值得到，其中，区域范围为基础图像块对应的区域范围。例如，基础图像块对应的区域范围可以为{(10,10)、(10,20)、(20,10)以及(20,20)}，预设数值为5，那么指定区域的区域范围为{(5,5)、(5,25)、(25,5)以及(25,25)}。此外，不同的临近图像对应的预设数值可以不同，并且各临近图像对应的预设数值可以根据该临近图像相对于基础图像的位移确定。其中，所述预设数值的确定过程可以为：针对于每一张临近图像，分别计算基础图像和该临近图像在行列方向上的投影，根据该临近图像对应的投影以及基础图像对应的投影，确定该临近图像相对于基础图像在行列上的位移，并根据该位移作为该临近图像对应的预设数值，其中，该位移可以采用SAD算法计算得到。
进一步，在所述步骤H22中，所述权重参数集中第二权重参数的数量与待处理图像集中的临近图像的数量相同，并且权重参数集中第二权重参数与待处理图像集中的临近图像一一对应。每张临近图像中均至少包括一个与基础图像块对应的临近图像块，并且每个临近图像块对应基础图像块均分别存在一个第二权重参数。由此，权重参数集包括一个第一权重参数以及至少一个第二权重参数，并且每个第二权重参数对应一个临近图像中与基础图像块对应的临近图像块。其中，所述第一权重参数可以为预先设置，用于表示基础图像块与其本身的相似程度；所述第二权重参数为根据基础图像块与其对应的临近图像进行得到。
由此，在本实施例的一个可能实现方式中，所述确定各个基础图像块分别对应的权重参数集具体包括：
针对每个基础图像块，确定该基础图像块对应的各临近图像块的第二权重参数，以及，获取该基础图像块对应的第一权重参数，以得到该基础图像块对应的权重参数集。
具体地，对于每个基础图像块，均对应至少一个临近图像块，其中，该基础图像对应的临近图像块的数量以基础图像对应的临近图像的数量相等。而对于该基础图像对应的每个临近图像块，该临近图像块均对应一第二权重参数，从而基础图像块对应的第二权重参数的数量与待处理图像集中的临近图像的图像数量相等。此外，所述第二权重参数为根据基础图像块与临近图像块的相似度计算得到。相应的，在本实施例的一个实现方式中，如图31所示，所述确定该基础图像块对应的各临近图像块的第二权重参数具体包括：
B10、针对每个临近图像块，计算该基础图像块与该临近图像块的相似度值；
B20、根据所述相似度值计算该临近图像块的第二权重参数。
具体地，所述相似度指的是所述基础图像块与临近图像块的相似度，临近图像块是根据基础图像块在临近图像中确定，并且临近图像块的图像大小与基础图像块的大小相同，从而基础图像块中包含的各像素点与临近图像块包含的各像素点为一一对应的，对于基础图像块中的每个像素点，在临近图像块中均可以找到一个与该像素点对应的像素点。由此，所述相似度可以根据基础图像块包含的各像素点的像素值与临近图像块包含的各像素点的像素值计算得到的。
其中，根据基础图像块包含的各像素点的像素值与临近图像块包含的各像素点的像素值计算相似度的具体过程可以为：读取基础图像块包含的各像素点分别对应的第一像素值，以及临近图像块包含的各像素点分别对应的第二像素值；针对于每一个第一像素值，计算该第一像素值与其对应的第二像素值的差值；再根据计算得到的所有差值来计算基础图像块和临近图像块的相似度值，其中，相似度值可以为计算得到的各差值的绝对值的均值，例如，计算得到第一像素值A和第二像素值A的差值，以及，第一像素值B和第二像素值B的差值，那么，可以根据第一像素值A和第二像素值A的差值，以及，第一像素值B和第二像素值B的差值确定基础图像块和临近图像块的相似度，具体地，该相似度值可以为第一像素值A和第二像素值A的差值的绝对值，与第一像素值B和第二像素值B的差值的绝对值之间的均值；因此，所述相似度值越大，说明基础图像块与临近图像块的相似性越低，反之，所述相似度值越小，说明基础图像块与临近图像块的相似性越高。
同时在本实施例中，对于基础图像的每个基础图像块
Figure PCTCN2020141932-appb-000003
和其对应的临近图像块
Figure PCTCN2020141932-appb-000004
Figure PCTCN2020141932-appb-000005
和
Figure PCTCN2020141932-appb-000006
的相似度d i的计算公式可以为：
Figure PCTCN2020141932-appb-000007
其中，j为像素索引，j＝1,2，...，M，M为基础图像块包含的像素点的数量(临近图像块与基础图像块包括的像素点数量相同)，
Figure PCTCN2020141932-appb-000008
为基础图像块中第j个像素点的像素值；
Figure PCTCN2020141932-appb-000009
为临近图像块中与基础图像块中第j个像素点对应的像素点的像素值，i表示第i个基础图像块，i＝1,2，...，N，N为基础图像块的数量。
进一步，根据所述相似度值的计算公式可以知道，所述相似度值与待处理图像集中图像的图像噪声强度，以及基础图像块的图像内容与临近图像块的图像内容的差异相关，具体地，当图像噪声强度高或者础图像块的图像内容与临近图像块的图像内容的差异大时，相似度值大；反之，当图像噪声强度低且础图像块的图像内容与临近图像块的图像内容的差异小时，相似度值小。那么，采用相似度值大的临近图像块进行后续合成操作，会使得合成效果差，从而在获取到基础图像块与各临近图像块的相似度值，可以根据各临近图像块对应的相似度值为各临近图像块配置第二权重参数，所述第二权重参数与相似度值负相关，即相似度值越大，第二权重参数越小；反之，相似度值越小，第二权重参数越大。这样通过为相似度低的临近图像分配较低的权重值，防止融合后出现拖影等失真问题。
示例性地，在本实施例的一个实现方式中，所述根据所述相似度值计算该临近图像块的第二权重参数具体包括：
C10、当所述相似度值小于或等于第一阈值时，将第一预设参数作为该临近图像块的第二权重参数；
C20、当所述相似度值大于第一阈值，且小于或等于第二阈值时，根据所述相似度值、所述第一阈值及所述第二阈值计算该临近图像块的第二权重参数；
C30、当所述相似度值大于第二阈值时，将预设第二预设参数作为该临近图像块的第二权重参数。
需要说明的是，本实施例中，B20可以仅包括C10、C20和C30中的任意一个步骤、任意两个步骤或者全部步骤，即在本实施例中，B20可以包括C10和/或C20和/或C30。
具体地，所述第一阈值和第二阈值均是用于衡量基础图像块与临近图像块的相似度，所述第二阈值大于第一阈值，那么，当相似度值小于第一阈值时，根据相似度值与相似度的关系可以得知，基础图像块与临近图像块的相似度性高，从而临近图像块对应 的第二权重参数值大，当相似度值大于第二阈值时，根据相似度值与相似度的关系可以得知，基础图像块与临近图像块的相似度性低，从而临近图像块对应的第二权重参数值小。由此可知，第一预设参数大于第二预设参数，并且根据所述相似度值、所述第一阈值及所述第二阈值计算该临近图像块的计算得到第三参数位于第一预设参数和第二预设参数之间。
进一步，在实施例的一个实现方式中，所述第三参数的计算过程可以为：首先计算相似度值与第二阈值的第一差值，再计算第一阈值与第二阈值的第二差值，然后再计算第一差值和第二差值的比值，并将所述比值作为临近图像块的第二权重参数。此外，由第三参数的计算过程可以得到，第三参数的取值范围为0-1，而第一预设参数大于第三参数，第二预设参数小于第三参数，从而可以将第一预设参数设置为1，第二预设参数设置为0。从而，所述第二权重参数与相似度值的对应关系的表达至可以为：
Figure PCTCN2020141932-appb-000010
其中，w i为第二权重参数，t 1为第一阈值，t 2为第二阈值，d i为相似度值i表示第i个基础图像块，i＝1,2，...，N，N为基础图像块的数量。
当然，值得说明的，相似度与权重系数呈正相关，即基础图像与临近图像的相似度越高，临近图像块对应的权重系数越大，反之，基础图像与临近图像的相似度越低，临近图像块对应的权重系数越低。而对于基础图像块而言，基础图像块确定相似度的比较对象为基础图像块本身，那么基础图像块与其自身的相似度大于或者等于临近图像块与基础图像块的相似度，相应的，第一权重参数大于或者等于第二权重系数。同时，由于第二权重参数的计算公式可以看出，第二权重系数最大为1，那么在本实施例的一个实现方式中，所述基础图像块对应的第一权重系数可以等于第二权重参数的最大值，即第一权重参数为1。
进一步，所述第一阈值和第二阈值可以是预先设置，也可以是根据该基础图像块对应临近图像块的相似度值确定的。在本实施中，所述第一阈值和第二阈值是根据该基础图像块对应各临近图像块的相似度值确定的。所述第一阈值和第二阈值的确定过程可以为：分别获取各临近图像块的相似度值，分别计算各相似度值的均值以及标准差，再根据均值以及标准差计算第一阈值和第二阈值，这样通过个临近图像块的相似度值确定第一阈值和第二阈值，可以使得第一阈值和第二阈值根据各临近图像的相似度值来自适 应调整，从而可以使得第一阈值和第二阈值根据各临近图像的噪声强度来自适应调整，避免了因第一阈值和第二阈值过大而造成图像去噪效果差，以及因第一阈值和第二阈值过小而造成的图像模糊，从而在保证图像去噪效果基础上，提高了图像的清晰度。
进一步，在本实施例的一个实现方式中，所述第一阈值t 1和第二阈值t 2的计算公式分别为：
t 1＝μ+s min×σ
t 2＝μ+s max×σ
Figure PCTCN2020141932-appb-000011
Figure PCTCN2020141932-appb-000012
其中，S min和S max为常数，d max为常数，L表示d i＜d max的临近图像块的数量，i＝1,2，...，L。
此外，待处理图像集中图像的图像噪声强度以及临近图像块的选取的准确性对相似度值的影响中，临近图像块的选取的准确性可以引起相似度值的大幅度变化，从而当临近图像块与基础图像块的相似度值大于预设值d max时，默认为基础图像块的图像内容与临近图像块的图像内容差异过大，将该临近图像块作为无效临近图像块(即，丢弃该无效临近图像块，不将该无效临近图像块作为基础图像块的临近图像块)。从而，对于d i≥d max的临近图像块，则可以认为基础图像块的图像内容与临近图像块的图像内容差异过大，从而无需确定该临近图像块对应的第一阈值和第二阈值，提高了基础图像块对应的权重参数集的计算速度。同时，与基础图像块的图像内容差异大的临近图像块，可以避免图像内容差大的临近图像块在图像融合时产生拖影，而造成输出图像失真的问题。
进一步，在所述步骤S23中，所述输出图像是由若干输出图像块拼接而成的，输出图像块是根据基础图像块、基础图像块对应的临近图像块以及基础图像块对应的权重参数集计算得到，例如，对于基础图像块中每一像素点，获取该像素点的第一像素值，以及各临近图像块中与该像素点对应的像素点的第二像素值，然后将各临近图像块对应的第二权重参数以及基础图像对应的第一权重参数作为加权系数，将第一像素值和各第二像素值进行加权处理，以得到输出图像块中的每个像素点的像素值。由此，在根据待处理图像集和各基础图像块对应的权重参数集确定输出图像时，可以针对于每一个基础 图像块，将该基础图像块与其对应的各临近图像块进行加权以得到该基础图像块对应的输出图像块，其中，在基础图像块与各临近图像块加权过程中，基础图像块的加权系数为权重参数集中的第一权重系统，各临近图像块的加权系数为权重参数集中各临近图像块分别对应的第二权重参数。此外，在计算得到各输出图像块后，根据计算得到的各输出图像块生成所述输出图像，其中，根据输出图像块生成输出图像可以是采用各输出图像块替换基础图像中与其对应的基础图像块的方式，或者，可以是将各输出图像块进行拼接得到输出图像的。
举例说明：假设待处理图像集包括一张基础图像和4张临近图像，那么对于每个基础图像块，该基础图像块对应四个临近图像块，分别记为第一临近图像块、第二临近图像块、第三临近图像块和第四临近图像块，并且按照拍摄顺序的排序为：基础图像块、第一临近图像块、第二临近图像块、第三临近图像块和第四临近图像块；那么在确定基础图像块对应的输出图像块时，对于基础图像块中的每个像素点，将该像素点的像素值为A、第一临近图像块中与该像素点对应的像素点的像素值为B、第二临近图像块中与该像素点对应的像素点的像素值为C、第三临近图像块中与该像素点对应的像素点的像素值为D，第四临近图像块中与该像素点对应的像素点的像素值为E，并且基础图像块对应的第一权重参数为a，第一临近图像块对应的第二权重参数为b，第二临近图像块对应的第二权重参数为c，第三临近图像块对应的第二权重参数为d，第四临近图像块对应的第二权重参数为e，那么该像素点对应的输出像素点的像素值＝(A*a+B*b+C*c+D*d+E*e)/5。
H30、将所述去噪图像输入至以已训练的第一图像处理模型，通过所述图像处理模型对所述去噪图像进行去偏色处理，得到所述去噪图像对应的处理后图像。
具体地，所述去噪图像为根据待处理图像集生成的，所述第一图像处理模型可是处理所述去噪图像的图像设备(例如，配置屏下摄像头的手机)预先训练的，也可以是由其他训练好后将第一图像处理模型对应的文件移植到图像设备中。此外，图像设备可以将所述第一图像处理模型可作为一个图像处理功能模块，当图像设备获取到去噪图像时，启动所述图像处理功能模块，将去噪图像输入至第一图像处理模型。
进一步，所述第一图像处理模型为基于训练图像集训练得到的，所述第一图像处理模型的训练过程可以为：
Q10、第一预设网络模型根据训练图像集中第一图像，生成所述第一图像对应的生成图像，其中，所述训练图像集包括多组训练图像组，每一组训练图像组包括第一图像 和第二图像，第一图像为对应第二图像的偏色图像；
Q20、所述第一预设网络模型根据所述第一图像对应的第二图像和所述第一图像对应的生成图像，对模型参数进行修正，并继续执行根据所述训练图像集中的下一训练图像组中的第一图像，生成所述第一图像对应的生成图像的步骤，直至所述第一预设网络模型的训练情况满足预设条件，以得到所述图像处理模型。
具体地，在所述步骤Q10中，所述第一预设网络模型可以采用深度网络学习模型，所述训练图像集包括多组具有不同图像内容的训练图像组，每一组训练图像组均包括第一图像和第二图像，第一图像为对应第二图像的偏色图像。其中，所述第一图像为对应第二图像的偏色图像指的是第一图像与第二图像相对应，第一图像和第二图像呈现同一图像场景，并且所述第一图像中满足预设偏色条件的第一目标像素点的数量满足预设数量条件。可以理解的是，第二图像为正常显示图像，第一图像中存在若干满足预设偏色条件的第一目标像素点，并且若干第一目标像素点的数量满足预设条件。例如，第二图像为如图6所示的图像，第一图像为如图5所示的图像，其中，第一图像的图像内容与第二图像的图像内容相同，但在第一图像中苹果对应的呈现的色彩与第二图像中苹果呈现的色彩不同，例如，在图8中，第一图像中苹果在第一图像中呈现的色彩为绿色偏蓝色；在图9中，第二图像中苹果在第二图像中呈现的色彩为深绿色。
进一步，所述预设偏色条件为第一图像中第一目标像素点的显示参数与第二图像中第二目标像素点的显示参数之间的误差满足预设误差条件，所述第一目标像素点与所述第二目标像素点之间具有一一对应关系。其中，所述显示参数为用于反映像素点对应的色彩的参数，例如，所述显示参数可以为像素点的RGB值，其中，R值为红色通道值、G值为绿色通道值、B值为蓝色通道值；也可以为像素点的hsl值，其中，h值为色相值，l为亮度值，s为饱和度值。此外，当显示参数为像素点的RGB值时，第一图像和第二图像中任一像素点的显示参数均包括R值、G值和B值三个显示参量；当显示显示为像素点的hls值，第一图像和第二图像中任一像素点的显示参数均包括h值、l值和s值三个显示参量。
所述预设误差条件用于衡量第一目标像素点是否为满足预设偏色条件的像素点，其中，所述预设误差条件为预设误差阈值，误差满足预设误差条件为误差大于或等于预设误差阈值。此外，所述显示参数包括若干显示参数，例如显示参数为像素点的RGB值，显示参数包括R值、G值和B值三个显示参，当显示参数为像素点的hsl值时，显示参数包括h值、l值和s值三个显示参量。由此，所述误差可以为显示参数中各显示参量 的误差最大值，也可以为显示参数中各显示参量的误差的最小值，还可以是所有显示参量的误差平均值。例如，这里以显示参数为像素点的RGB值进行说明，第一目标像素点的显示参数为(55,86,108)，第二目标像素点的显示参数为(58,95,120)，那么各显示参量的误差值分为3,9以及12；由此，当第一目标像素点与第二目标像素点的误差为各显示参量的误差最大值时，该误差为12；当第一目标像素点与第二目标像素点的误差为各显示参量的误差最小值时，该误差为3；当第一目标像素点与第二目标像素点的误差为所有显示参量的误差平均值时，该误差为8；需要说明的是，在一种可能的实现方式中，也可以仅参考RGB中一个参数(例如R、G或B)或任意两个参数的误差，当显示参数为像素点的hsl值时，同理。
进一步，用于与第一目标像素点计算误差的第二目标像素点与第一目标显示点之间存在一一对应关系。可以理解的是，对于第一目标像素点，第二图像中存在唯一的第二目标像素点与第一目标像素点对应，其中，第一目标像素点与第二目标像素点对应指的是第一目标像素点在第一图像中的像素位置，与第二目标像素点在第二图像中的像素位置相对应。例如，第一目标像素点在第一图像中的像素位置为(5,6)，第二目标像素点在第二图像中的像素位置为(5,6)。此外，所述第一目标像素点可以为第一图像中任一像素点，也可以是第一图像中目标区域中任一像素点，其中，所述目标区域可以为第一图像中物品所处区域，其中，所述物品所处区域可以为人或物在图像中对应的区域。例如，如图5所示，所述目标区域为第一图像中苹果所处区域。也就是说，第一图像中可以全部像素点与第二图像相比较出现偏色，即第一图像中全部像素点均为第一目标像素点，也可以只有一部分像素点与第二图像相比较出现偏色，即第一图像中部分像素点为第一目标像素点，例如，当一图像中只有部分区域(例如图中苹果对应的区域)中的像素点与第二图像相比较出现偏色时，该图像也可以理解为对应第二图像的偏色图像，即第一图像。
进一步，所述第一图像和第二图像相对应指的是第一图像的图像尺寸与第二图像的图像尺寸相等，并且所述第一图像和第二图像对应相同的图像场景。所述第一图像和第二图像对应相同的图像场景指的是第一图像携带的图像内容与第二图像携带的图像内容的相似度达到预设阈值，所述第一图像的图像尺寸与第二图像的图像尺寸相同，以使得当第一图像和第二图像重合时，第一图像携带的物体对第二图像中与其对应的物体的覆盖率达到预设条件。其中，所述预设阈值可以为99％，所述预设条件可以为99.5％等。在实际应用中，所述第一图像可以是通过屏下成像系统拍摄得到；所述第二图像可 以是通过正常屏上成像系统(如，屏上摄像头)拍摄得到，也可以是通过网络(如，百度)获取到，还可以是通过其他外部设备(如，智能手机)发送的。
在本实施例的一个可能实现方式中，所述第二图像为通过正常屏上成像系统拍摄得到，所述第二图像和第一图像的拍摄参数相同。其中，所述拍摄参数可以包括成像系统的曝光参数，所述曝光参数可以包括光圈、快门速度、感光度、对焦以及白平衡等。当然，在实际应用中，所述拍摄参数还可以包括环境光、拍摄角度以及拍摄范围等。例如，所述第一图像为如图5所示的通过屏下摄像头拍摄一场景得到的图像，第二图像为如图6所示的通过屏上摄像头拍摄该场景得到的图像。
进一步，在本实施例的一个实现方式中，为了减少第一图像和第二图像的图像差异对第一预设网络模型训练的影响，所述第一图像的图像内容和第二图像的图像内容可以完全相同。即所述第一图像和第二图像具有相同图像内容指的是第一图像具有的物体内容与第二图像具有的物体内容相同，所述第一图像的图像尺寸与第二图像的图像尺寸相同，并且当第一图像和第二图像重合时，第一图像具有的物体可以覆盖第二图像中与其对应的物体。
举例说明：所述第一图像的图像尺寸为400*400，第一图像的图像内容为一个圆，并且第一图像中圆的圆心在第一图像中的位置为(200，200)、半径长度为50像素。那么，所述第二图像的图像尺寸为400*400，第二图像的图像内容也为一个圆，第二图像中圆的圆心在第二图像中的位置为(200，200)，半径为50像素；当第一图像放置于第二图像上并与第二图像重合时，所述第一图像覆盖所述第二图像，并且第一图像中的圆与第二图像的圆上下重叠。
进一步，当第二图像为通过正常屏上成像系统拍摄得到时，由于第一图像和第二图像是通过两个不同的成像系统拍摄得到，而在更换成像系统时，可能会造成屏上成像系统和屏下成像系统的拍摄角度和/或拍摄位置的变化，使得第一图像和第二图像在空间上存在不对齐的问题。由此，在本实施例的一个可能实现方式中，在通过屏上成像系统拍摄第二图像以及通过屏下成像系统拍摄第一图像时，可以将屏上成像系统和屏下成像系统设置于同一固定架上，将屏上成像系统和屏下成像系统并排布置在固定架上，并保持屏上成像系统和屏下成像系统相接触。同时，将屏上成像系统和屏下成像系统分别与无线设置(如，蓝牙手表等)相连接，通过无线设置触发屏上成像系统和屏下成像系统的快门，这样可以减少拍摄过程中屏上成像系统和屏下成像系统的位置变化，提高第一图像和第二图像在空间上的对齐性。当然，屏上成像系统和屏下成像系统的拍摄时间 和拍摄范围均相同。
此外，虽然在第一图像和第二图像的拍摄过程中，可以通过固定屏下成像系统和屏上成像系统的拍摄位置、拍摄角度、拍摄时间以及曝光系数等。但是，由于环境参数(如，光线强度、风吹动成像系统等)，屏下成像系统拍摄得到的第一图像和屏上成像系统拍摄得到的第二图像在空间上还可能存在不对齐的问题。由此，在将训练图像集中第一图像输入第一预设网络模型之前，可以训练图像集中的各训练图像组中的第一图像和第二图像进行对齐处理，从而在本实施例的一个实现方式中，所述第一预设网络模型根据训练图像集中第一图像，生成所述第一图像对应的生成图像之前还包括：
F100、针对所述训练图像集中每组训练图像组，将该组训练图像组中的第一图像与所述第一图像对应的第二图像进行对齐处理，得到与所述第二图像对齐的对齐图像，并将所述对齐图像作为第一图像。
具体地，所述针对所述训练图像集中每组训练图像组指的是对训练图像集中每一组训练图像组均执行对齐处理，所述对齐处理可以是在获取到训练图像集之后，分别对每一组训练图像组进行对齐处理，以得到对齐后的训练图像组，并在所有组训练图像组对齐后执行将每一组训练图像组中的第一图像输入第一预设网络模型的步骤；当然也可以是在将每一组训练图像组中的第一图像输入第一预设网络模型之前，对该组训练图像组进行对齐处理，以得到该组训练图像对应的对齐后的训练图像组，之后将对齐后的训练图像组中的第一图像输入第一预设网络模型。在本实施例中，所述对齐处理是在获取到训练图像集后，分别对每一组训练图像组进行，并在所有训练图像组完成对齐处理后，在执行将训练图像集中第一图像输入第一预设网络模型的操作。
进一步，在将第一图像与第二图像进行对应处理的过程中，将第一图像作为参考图像，第二图像作为基准图像，其中，对齐处理是以基准图像为基准。可以理解的是，所述将该组训练图像组中的参考图像与所述参考图像对应的基准图像进行对齐处理为所述将该组训练图像组中的参考图像与所述参考图像对应的基准图像进行对齐处理，并且所述将该组训练图像组中的参考图像与所述参考图像对应的基准图像进行对齐处理可以为以基础图像为基准，将参考图像中像素点与基础图像中与其对应的像素点对齐，以使得参考图像中像素点与基准图像中像素点的对齐率可以达到预设值，例如，99％等。其中，所述参考图像中像素点与基准图像中与其对应的像素点对齐指的是：对于参考图像中的参考像素点和基准图像中与参考像素点相对应的基准像素点，若参考像素点对应的像素坐标与基准像素点对应的像素坐标相同，那么参考像素点与基准像素点对齐；若 参考像素点对应的像素坐标与基准像素点对应的像素坐标不相同，那么参考像素点与基准像素点对齐。所述对齐图像指的通过对参考图像进行对齐处理得到图像，并且对齐图像中每个像素点与基准图像中其对应的像素点的像素坐标相同。此外，在得到对齐图像后，采用所述对齐图像替换其对应的参考图像以更新训练图像组，以使得更新后的训练图像组中的参考图像和基准图像在空间上对齐。
进一步，由于不同组训练图像组中的参考图像和基准图像的对齐程度不同，从而可以在实现对齐的基础上，针对不同对齐程度的参考图像和基准图像可以采用不同的对齐方式，以使得各组训练图像组均可以采用复杂度低的对齐方式进行对齐处理。由此，在本实施例的一个实现方式中，所述将该组训练图像组中的参考图像与所述参考图像对应的基准图像进行对齐处理具体包括：
F11、获取该组训练图像组中的参考图像与所述参考图像对应的基准图像之间的像素偏差量；
F12、根据所述像素偏差量确定所述参考图像对应的对齐方式，并采用所述对齐方式将所述参考图像与所述基准图像进行对齐处理。
具体地，所述像素偏差量指的是参考图像中参考像素点与基准图像中与该参考像素点对应的基准像素点不对齐的参考像素点的总数量。所述像素偏差量可以通过获取参考图像中各参考像素点的参考坐标，以及基准图像中各基准像素点的基准坐标，然后将参考像素点的参考坐标与其对应的基准像素点的基准坐标进行比较，若参考坐标与基准坐标相同，则判定参考像素点与其对应的基准像素点对齐；若参考坐标与基准坐标不相同，则判定参考像素点与其对应的基准像素点不对齐，最后获取所有不对齐的参考像素点的总数量，以得到所述像素偏差量。例如，当所述参考图像中的参考像素点的参考坐标为(200,200)，基准图像中与所述参考像素点对应的基准像素点的基准坐标为(201,200)时，所述参考像素点与基准像素点不对齐，不对齐的参考像素点的总数量加一；当所述参考图像中的参考像素点的参考坐标为(200,200)，基准图像中与所述参考像素点对应的基准像素点的基准坐标为(200,200)时，所述参考像素点与基准像素点对齐，不对齐的参考像素点的总数量不变。
进一步，为了确定像素偏差量与对齐方式的对应关系，可以需要设置偏差量阈值，在获取到参考图像的像素偏差量时，可以通过将获取到的像素偏差量与预设偏差量阈值进行比较，以确定像素偏差量对应的对齐方式。由此，在本实施例的一个实现方式中，所述根据所述像素偏差量确定所述述参考图像对应的对齐方式，并采用所述对齐方式将 所述参考图像与所述基准图像进行对齐处理具体包括：
F121、当所述像素偏差量小于或等于预设偏差量阈值时，根据所述参考图像与所述基准图像的互信息，以所述基准图像为基准对所述参考图像进行对齐处理；
F122、当所述像素偏差量大于所述预设偏差量阈值时，提取所述参考图像的参考像素点集和所述基准图像的基准像素点集，所述参考像素点集包含所述参考图像中的若干参考像素点，所述基准像素点集包括所述基准图像中的若干基准像素点，所述基准像素点集中的基准像素点与所述参考像素点集中的参考像素点一一对应；针对所述参考像素点集中每个参考像素点，计算该参考像素点与其对应的基准像素点的坐标差值，并根据该参考像素点对应的坐标差值对该参考像素点进行位置调整，以将该参考像素点与该参考像素点对应的基准像素点对齐。
具体地，所述预设偏差量阈值为预先设置，例如，预设偏差量阈值为20。所述当所述像素偏差量小于或等于预设偏差量阈值时指的是当将所述像素偏差量与所述预设偏差量阈值时，所述像素偏差量小于等于预设偏差量阈值。而当所述像素偏差量小于等于预设偏差量阈值时，说明参考图像和基准图像在空间上的偏差较小，此时可以采用根据所述参考图像与所述基准图像的互信息对参考图像和基准图像进行对齐。在本实施例中，以所述参考图像和其对应的基准图像之间互信息对参考图像和基准图像进行对齐的过程可以采用图像配准方法，所述图像配准方法中以互信息作为度量准则，通过优化器对度量准则迭代进行优化以得到对齐参数，通过所述配准所述对齐参数的配准器将参考图像与基准图像进行对齐，这保证参考图像与基准图像的对齐效果的基础，降低了参考图像与基准图像对齐的复杂性，从而提高了对齐效率。在本实施例中，所述优化器主要采用平移和旋转变换，以通过所述平移和旋转变换来优化度量准则。
进一步，所述像素偏差量大于所述预设偏差量阈值，说明参考图像和基准图像在空间上不对齐程度较高，此时需要着重考虑对齐效果。从而此时可以采用通过选取参考图像中的参考像素点集和基准图像中基准像素点集的方式对参考图像和基准图像进行对齐。所述参考像素点集的参考像素点与基准像素点集中基准像素点一一对应，以使得对于参考像素点集中的任一参考像素点，在基准像素点集中均可以找到一个基准像素点，所述基准像素点在基准图像中的位置与参考像素点在参考图像中的位置相对应。此外，所述参考像素点集和基准像素点集可以是在获取到参考像素点集/基准像素点集后，根据参考像素点与基准像素点的对应关系确定基准像素点集/参考像素点集，例如，所述参考像素点集通过在参考图像中随机选取多个参考像素点的方式生成，基准像素点则是 根据参考像素点集包含的各参考像素点确定的。
同时在本实施例中，所述参考像素点集和基准像素点集均是通过尺度不变特征变换(Scale-invariant feature transform，sift)的方式获取得到，即所述参考像素点集中参考像素点为参考图像中第一sift特征点，所述基准像素点集中基准像素点为基准图像的第二sift特征点。相应的，所述计算该参考像素点与其对应的基准像素点的坐标差值为将参考像素点中第一sift特征点与基准像素点集中第二sift特征点进行点对点匹配，以得到各第一sift特征点与其对应的各第二sift特征点的坐标差值，并根据该第一sift特征点对应的坐标差值对该第一sift特征点进行位置变换，以将该参考像素点与该第一sift特征点对应的第二sift特征点对齐，从而使得参考图像中第一sift特征点与基准图像中第二sift特征点位置相同，从而实现了参考图像与基准图像的对齐。
进一步，在本实施例的一个实现方式中，如图3、图4和图8所示，所述第一预设网络模型包括下采样模块100以及变换模块200，相应的，所述第一预设网络模型根据训练图像集中第一图像，生成所述第一图像对应的生成图像可以具体包括：
Q11、将所述训练图像集中第一图像输入所述下采样模块，通过所述下采样模块得到所述第一图像对应的双边网格以及所述第一图像对应的指导图像，其中，所述指导图像的分辨率与所述第一图像的分辨率相同；
Q12、将所述指导图像、所述双边网格以及所述第一图像输入所述变换模块，通过变换模块生成所述第一图像对应的生成图像。
具体地，所述双边网格10为在二维图像的像素坐标中增加一维代表像素强度的维度而得到的三维双边网格，其中，所述三维双边网络的三维分别为二维图像的像素坐标中横轴和纵轴，以及增加的代表像素强度的维度。所述指导图像为通过对第一图像进行像素级操作得到的，所述指导图像50的分辨率与所述第一图像的分辨率相同，例如，所述指导图像50为所述第一图像对应的灰阶图像。
进一步，由于所述下采样模块100用于输出第一图像对应的双边网格10和指导图像50，从而所述下采样模块100包括下采样单元70和卷积单元30，所述下采样单元70用于输出所述第一图像对应的双边网格10，所述卷积单元30用于输出所述第一图像对应的指导图像50。相应的，如图3、图4和图9所示，所述将所述训练图像集中第一图像输入所述下采样模块，通过所述下采样模块得到所述第一图像对应的双边网格参数以及所述第一图像对应的指导图像具体包括：
Q111、将所述训练图像集中第一图像分别输入所述下采样单元以及所述卷积单元；
Q112、通过所述下采样单元得到所述第一图像对应的双边网格，并通过所述卷积单元得到所述第一图像对应的指导图像。
具体地，所述下采样单元70用于对第一图像进行下采样，以得到第一图像对应的特征图像，并根据所述特征图像生成所述第一图像对应的双边网格，特征图像的空间通道数大于第一图像的空间通道数。所述双边网格是根据所述特征图像的局部特征和全局特征生成的，其中，所述局部特征为从图像局部区域中抽取的特征，例如，边缘、角点、线、曲线和属性区域等，在本实施例中，所述局部特征可以为区域颜色特征。所述全局特征指的是表示整幅图像属性的特征，例如，颜色特征、纹理特征和形状特征。在本实施例中，所述全局特征可以为整幅图像的颜色特征。
进一步，在本实施例的一个可能实现方式中，所述下采样单元70包括下采样层、局部特征提取层、全局特征提取层以及全连接层，所述局部特征提取层连接于所述下采样层与全连接层之间，所述全局特征提取层连接于下采样层与全连接层之间，并且所述全局特征提取层与所述局部特征提取层并联。由此可知，第一图像作为输入项输入下采样层，经过下采样层输出特征图像；下采样层的特征图像分别输入至局部特征提取层和全局特征提取层，局部特征提取层提取特征图像的局部特征，全局特征提取层提取特征图像的全局特征；局部特征提取层输出的局部特征和全局特征提取层输出的全局特征分别输入全连接层，以通过全连接层输出第一图像对应的双边网格。此外，在本实施例的一个可能实现方式中，所述下采样层包括下采样卷积层和四个第一卷积层，第一卷积层的卷积核为1*1，步长为1；所述局部特征提取层可以包括两个第二卷积层，两个第二卷积层的卷积核均为3*3，步长均为1；所述全局特征提取层可以包括两个第三卷积层和三个全连接层，两个第三卷积层的卷积核均为3*3，步长均为2。
进一步，所述卷积单元30包括第四卷积层，第一图像输入第四卷积层，经过第四卷积层输入指导图像，其中，所述指导图像与第一图像的分辨率相同。例如，第一图像为彩色图像，所述第四卷积层对第一图像进行像素级操作，以使得指导图像为第一图像的灰阶图像。
举例说明：第一图像I输入下采样卷积层，经过下采样卷积层输出256x256大小的三通道低分辨率图像，256x256大小的三通道低分辨率图像依次经过四个第一卷积层，得到16x16大小的64通道特征图像；16x16大小的64通道特征图像输入局部特征提取层得到局部特征L，16x16大小的64通道特征图像输入全局特征提取层得到全局特征； 局部特征和全局特征输入全连接层，经过全连接层输出双边网格。此外，将第一图像输入至卷积单元，经过卷积单元输入第一图像对应的指导图像。
进一步，所述在本实施例的一个实现方式中，所述变换模块200包括切分单元40以及变换单元60，相应的，如图3、图4和10所示，所述将所述指导图像、所述双边网格以及所述第一图像输入所述变换模块，通过变换模块生成所述第一图像对应的生成图像具体包括：
Q121、将所述指导图像输入所述切分单元，通过所述切分单元对所述双边网格进行切分，以得到所述第一图像中各像素点的颜色变换矩阵；
Q122、将所述第一图像以及所述第一图像中各像素点的颜色变换矩阵输入所述变换单元，通过所述变换单元生成所述第一图像对应的生成图像。
具体地，所述切分单元40包括上采样层，所述上采样层的输入项为指导图像和双边网格，通过所述指导图像对双边网格进行上采样，以得到第一图像中各像素点的颜色变换矩阵。其中，所述上采样层的上采样过程可以为将所述双边网格参考指导图进行上采样，以得到第一图像中各像素点的颜色变换矩阵。此外，所述变换单元60的输入项为各像素点的颜色变换矩阵以及第一图像，通过各像素点的颜色变换矩阵对第一图像中其对应的像素点的颜色进行变换，以得到所述第一图像对应的生成图像。
进一步，在所述步骤Q20中，所述预设条件包括损失函数值满足预设要求或者训练次数达到预设次数。所述预设要求可以是根据第一图像处理模型的精度来确定，这里不做详细说明，所述预设次数可以为第一预设网络模型的最大训练次数，例如，5000次等。由此，在第一预设网络模型输出生成图像，根据所述生成图像以及所述第二图像来计算第一预设网络模型的损失函数值，在计算得到损失函数值后，判断所述损失函数值是否满足预设要求；若损失函数值满足预设要求，则结束训练；若损失函数值不满足预设要求，则判断所述第一预设网络模型的训练次数是否达到预测次数，若未达到预设次数，则根据所述损失函数值对所述第一预设网络模型的网络参数进行修正；若达到预设次数，则结束训练。这样通过损失函数值和训练次数来判断第一预设网络模型训练是否结束，可以避免因损失函数值无法达到预设要求而造成第一预设网络模型的训练进入死循环。
进一步，由于对第一预设网络模型的网络参数进行修改是在第一预设网络模型的训练情况未满足预设条件(即，损失函数值未满足预设要求并且训练次数未达到预设次数)，从而在根据损失函数值对所述第一预设网络模型的网络参数进行修正后，需要继 续对网络模型进行训练，即继续执行将训练图像集中的第一图像输入第一预设网络模型的步骤。其中，继续执行将训练图像集中第一图像输入第一预设网络模型中的第一图像为未作为输入项输入过第一预设网络模型的第一图像。例如，训练图像集中所有第一图像具有唯一图像标识(例如，图像编号)，第一次训练输入的第一图像为图像标识与第二次训练输入的第一图像的图像标识不同，如，第一次训练输入的第一图像的图像编号为1，第二次训练输入的第一图像的图像编号为2，第N次训练输入的第一图像的图像编号为N。当然，在实际应用中，由于训练图像集中的第一图像的数量有限，为了提高第一图像处理模型的训练效果，可以依次将训练图像集中的第一图像输入至第一预设网络模型以对第一预设网络模型进行训练，当训练图像集中的所有第一图像均输入过第一预设网络模型后，可以继续执行依次将训练图像集中的第一图像输入至第一预设网络模型的操作，以使得训练图像集中的训练图像组按循环输入至第一预设网络模型。
此外，对于不同曝光度下拍摄的图像的高光部分的扩散程度不同，从而屏下成像系统在不同光线强度下拍摄的图像的高光部分的扩散程度不同，从而使得屏下成像系统拍摄得到的图像质量不同。由此，在对第一图像处理模型进行训练时，可以获取多个训练图像集，每个训练图像集对应不同的曝光度，并采用每个训练图像集对第一预设网络模型进行训练，以得到每个训练图像集对应的模型参数。这样采用具有相同曝光度的第一图像作为训练样本图像，可以提高网络模型的训练速度，同时使得不同曝光度对应不同的模型参数，在采用第一图像处理模型对具有偏色的待处理图像进行处理时，可以根据去噪图像对应的曝光度选取相应的模型参数，抑制各曝光度下图像高光部分的扩散，以提高去噪图像对应的处理后图像的图像质量。
进一步，在本实施例的一个实现方式中，所述训练图像集包括若干训练子图像集，每个训练子图像集包括若干组训练样本图像组，若干训组训练图像组中的任意两组训练样本图像组中的第一图像的曝光度相同(即对于每组训练图像组而言，该组中的每组训练样本图像组中的第一图像的曝光度均相同)，若干组训练图像组中的每组训练样本图像组中的第二图像的曝光度均处于预设范围内，任意两个训练子图像集中的第一图像的曝光度不相同。其中，所述第二图像的曝光度的预设范围可以根据曝光时间和ISO(现有的手机的光圈为固定值)确定，所述曝光度的预设范围表示在无需曝光补偿下拍摄图像的曝光度，屏上摄像头在曝光度的预设范围内的第一曝光度下拍摄得到的第二图像为正常曝光图像，这通过采用正常曝光图像作为第二图像，可以使得根据训练图像集训练得到的第一图像处理模型输出的图像具有正常曝光度，从而使得第一图像处理模型具有 提亮的功能。例如，当输入图像处理模的图像A为低曝光度的图像，那么图像A通过所述第一图像处理模型处理后，可以使得输出图像A的曝光度为正常曝光度，从而提高了图像A的图像亮度。
举例说明：假设图像的曝光度包括5个等级，分别记为0,-1,-2,-3和-4，其中，所述曝光度随着曝光度等级的降低而增强，例如，曝光等级0对应的曝光度低于曝光等级-4对应的曝光度。所述训练图像集可以包括5个训练子图像集，分别记为第一训练子图像集、第二训练子图像集、第三训练子图像集、第四训练子图像集以及第五训练子图像集，所述第一训练子图像集包含的每组训练图像组中第一图像的曝光度对应0等级，第二图像为曝光度在预设范围内的图像；所述第二训练子图像集包含的每组训练图像组中第一图像的曝光度对应-1等级，第二图像为曝光度在预设范围内的图像；所述第三训练子图像集包含的每组训练图像组中第一图像的曝光度对应-2等级，第二图像为曝光度在预设范围内的图像；所述第四训练子图像集包含的每组训练图像组中第一图像的曝光度对应-3等级，第二图像为曝光度在预设范围内的图像；所述第五训练子图像集包含的每组训练图像组中第一图像的曝光度对应-4等级，第二图像为曝光度在预设范围内的图像。当然，值得说明的，所述第一训练子图像集、第二训练子图像集、第三训练子图像集、第四训练子图像集以及第五训练子图像集包含的训练图像组的数量可以相同，也可以不同。例如，所述第一训练子图像集、第二训练子图像集、第三训练子图像集、第四训练子图像集以及第五训练子图像集均包括5000组训练图像组。
此外，针对于每个训练子图像集，该训练子图像集为第一预设网络模型的一个训练图像集，通过该训练子图像集对第一预设网络模型进行训练，以得到该训练子图像集对应的模型参数。其中，该训练子图像集作为训练图像集对第一预设网络模型进行训练的过程包括：所述第一预设网络模型根据训练子图像集中第一图像，生成第一图像对应的生成图像；所述第一预设网络模型根据所述第一图像对应的第二图像和所述第一图像对应的生成图像，对模型参数进行修正，并且第一预设网络模型继续执行根据训练子图像集中第一图像，生成第一图像对应的生成图像的步骤，直至所述第一预设网络模型的训练情况满足预设条件，以得到该训练子图像对应的模型参数，具体地可以参数步骤M10和步骤Q20，这里就不再赘述。
进一步，每个训练子图像集对所述第一预设网络模型的训练过程为相互独立，即分别采用每个训练子图像集对所述第一预设网络模型进行训练。同时，分别采用个训练子图像集对所述第一预设网络模型进行训练可以得到若干模型参数，每个模型参数均为 根据一个训练子图像集训练得到，并且任意两个模型参数各自对应的训练子图像集互不相同。由此可知，第一图像处理模型对应若干模型参数，若干模型参数与若干训练子图像集一一对应。
举例说明：以上述训练样本图像包括第一训练子图像集、第二训练子图像集、第三训练子图像集、第四训练子图像集以及第五训练子图像集为例，那么第一图像处理模型包括5个模型参数，分别记为第一模型参数、第二模型参数、第三模型参数、第四模型参数以及第五模型参数，其中，第一模型参数对应第一训练子图像集，第二模型参数对应第二训练子图像集，第三模型参数对应第三训练子图像集，第四模型参数对应第四训练子图像集，第五模型参数对应第五训练子图像集。
进一步，当训练图像集包括若干训练子图像集时，第一预设网络模型根据每个训练子图像集进行训练。这里以训练图像集包括5个训练子图像集为例加以说明。采用第一训练子图像集、第二训练子图像集、第三训练子图像集、第四训练子图像集以及第五训练子图像集分别对第一预设网络模型进行训练的过程可以为：首先采用第一训练子图像集对第一预设网络模型进行训练，得到第一训练子图像集对应的第一模型参数，之后再采用第二训练子图像集对第一预设网络模型进行训练，得到第二训练子图像集对应的第二模型参数，依次类推得到第五训练子图像集对应的第五模型参数。
此外，当使用同一个第一预设网络模型对多个训练子图像集分别进行训练时，会存在各个训练子图像集对于第一预设网络模型的模型参数产生影响的问题，举例来说，假设训练子图像集A包括1000组训练图像组，训练子图像集B包括200组训练图像组，那么，先用训练子图像集A对第一预设网络模型进行训练，再紧接着用训练子图像集B对第一预设网络模型进行训练所得到的该训练子图像集B所对应的模型参数，与仅用训练子图像集B对第一预设网络模型进行训练所得到的该训练子图像集B所对应的模型参数，是不同的。
故此，在本实施例的一种实现方式中，第一预设网络模型在训练完一训练子图像集之后，可以先对该第一预设网络模型进行初始化，再使用该初始化后的第一预设网络模型对下一训练子图像集进行训练。举例来说，第一预设网络模型根据第一训练子图像集进行训练，得到第一训练子图像集对应的第一模型参数后，所述第一预设网络模型可以进行初始化，以使得用于训练第二模型参数的第一预设网络模型的初始模型参数以及模型结构均与用于训练第一模型参数的第一预设网络模型相同，当然，在训练第三模型参数、第四模型参数和第五模型参数之前，均可以对第一预设网络模型进行初始化，以 使得每个训练子图像集对应的第一预设网络模型的初始模型参数以及模型结构均相同。当然，在实际应用中，第一预设网络模型根据第一训练子图像集进行训练，得到第一训练子图像集对应的第一模型参数后，也可以直接采用基于第一训练子图像集训练后的第一预设网络模型(配置第一模型参数)对第二训练子图像集进行训练，以得到第二训练子图像集对应的第二模型参数，继续执行第一预设网络模型(配置第二模型参数)根据第三训练子图像集进行训练的步骤，直至第五训练子图像集训练完毕，得到第五训练子图像集对应的第五模型参数。
此外，第一训练子图像集、第二训练子图像集、第三训练子图像集、第四训练子图像集以及第五训练子图像集均包括一定数量的训练图像组，以使得每组训练子图像均可以满足第一预设网络模型的训练需求。当然，在实际应用中，在基于每一训练子图像集对第一预设网络模型进行训练时，可以循环将该训练子图像集中的训练图像组输入至第一预设网络模型，以对所述第一预设网络模型进行训练，使得第一预设网络模型满足预设要求。
进一步，在本实施例的一个实现按时，所述获取包含各个训练子图像集的训练样本的获取过程可以为：首先将屏下成像系统设置为第一曝光度，通过屏下成像系统获取第一训练子图像集中的第一图像，以及通过屏上成像系统获取第一训练子图像集中和第一图像对应的第二图像；在第一训练子图像集获取完成后，将屏下成像系统设置为第二曝光度，通过屏下成像系统和屏上成像系统获取第二训练子图像集中第一图像和第一图像对应的第二图像；在第二训练子图像集获取完成后；继续执行设置屏下成像系统的曝光度以及获取训练子图像集的步骤，直至获取到训练图像集包含的所有训练子图像集。其中，训练图像集包含的每个训练子图像集包含的训练图像组的数量可以相同，也可以不相同。在本实施例的一个实现方式中，所述训练图像集包含的每个训练子图像集包含的训练图像组的数量可以相同，例如，每个训练子图像集包含的训练图像组的数量为5000。
进一步，由于各训练子图像集均对应不同的曝光度，从而在获取到每个训练子图像集对应的模型参数后，针对于每个训练子图像集，可以将该训练子图像集对应的模型参数与该训练子图像集对应的曝光度相关联，以建立曝光度与模型参数的对应关系。这样在采用第一图像处理模型对去噪图像进行处理时，可以先获取去噪图像的曝光度，再根据曝光度确定去噪图像对应的模型参数，然后将去噪图像对应的模型参数配置于第一预设网络模型，以得到去噪图像对应的第一图像处理模型，以便于采用该第一图像处理 模型对去噪图像进行处理。这样对于不同曝光度的去噪图像可以确定配置不同网络参数的第一图像处理模型，并采用去噪图像对应的第一图像处理模型对去噪图像进行处理，避免曝光度对偏色的影响，从而可以提高去除去噪图像的偏色的效果。此外，所述第二图像可以为采用正常曝光度，使得所述第一图像处理模型输出的输出图像为正常曝光度，对去噪图像起到提亮的效果。
进一步，由第一图像处理模型的生成过程可以知道，在本实施例的一种可能实现方式中，所述第一图像处理模型包括可以若干模型参数，并且每个模型参数均对应一个曝光度。因此，在该实现方式中，在获取到去噪图像后，可以先检测所述第一图像处理模型包括的模型参数的数量，当模型参数的数量为一个时，直接将所述去噪图像输入到所述第一图像处理模型内，以通过所述图像处理对去噪图像进行处理；当模型参数为多个时，可以先获取去噪图像的曝光度，再根据曝光度确定该去噪图像对应的模型参数，将该去噪图像对应的模型参数配置于所述第一图像处理模型，以对图像处理参数配置的模型参数进行更新，并将去噪图像输入更新后第一图像处理模型。
进一步，在本实施例的一个实现方式中，所述第一图像处理模型对应若干模型参数，每个模型参数均为根据一个训练子图像集训练得到的，并且任意两个模型参数各自分别对应的训练子图像集互不相同(例如，模型参数A对应的训练子图像集与模型参数B对应的训练子图像集是不同的)。相应的，所述将所述去噪图像输入至以已训练的第一图像处理模型具体包括：
F101、提取所述去噪图像的曝光度。
具体地，所述曝光度为图像采集装置的感光元件被光线照射的程度，用于反映成像时的曝光程度。所述去噪图像可以为RGB三通道图像，所述去噪图像的曝光度为根据去噪图像的高光区域确定的，所述高光区域包含的各像素点的R(即红色通道)值、G(即绿色通道)值以及B(即蓝色通道)值中至少存在一个值大于预设阈值。当然，在实际应用中，所述去噪图像还可以是Y通道图像或者贝尔格式图像，而当所述去噪图像为Y通道图像或者贝尔格式图像(Raw格式)时，在提取所述去噪图像之前，需要将所述Y通道图像或者贝尔格式图像转换为RGB三通道图像，以便于根据去噪图像的红色通道R值、绿色通道G值以及蓝色通道B值确定去噪图像的高光区域。
进一步，在本实施例的一个实现方式中，所述提取所述去噪图像的曝光度具体包括：
G10、根据所述去噪图像中各像素点的红色通道R值、绿色通道G值以及蓝色通道 B值确定满足预设条件的第一像素点，其中，所述预设条件为R值、G值以及B值中至少一个值大于预设阈值；
G20、根据满足预设条件的所有第一像素点确定所述去噪图像的高光区域，并根据所述高光区域确定所述去噪图像的曝光度。
具体地，所述去噪图像为RGB三通道图像，从而对于去噪图像中的每个像素点，该像素点均包括红色通道R值、绿色通道G值和蓝色通道B值，即对于去噪图像中的每个像素点，均可以获取到该像素点的红色通道R值、绿色通道G值以及蓝色通道B值。由此，在提取所述去噪图像的曝光的过程中，首先针对于每个去噪图像的每个像素点，获取该像素点的红色通道R值、绿色通道G值以及蓝色通道B值，之后再分别将各像素点的R值、G值以及B值分别与预设阈值进行比较，以获取去噪图像中满足预设条件的第一像素点。所述预设条件为预设条件为R值、G值以及B值中至少一个值大于预设阈值，第一像素点满足预设条件指的是第一像素点的R值大于预设阈值，第一像素点的G值大于预设阈值，第一像素点的B值大于预设阈值，第一像素点的R值和G值均大于预设阈值，第一像素点的R值和B值均大于预设阈值，第一像素点的G值和B值均大于预设阈值，或者第一像素点的R值、B值和G值均大于预设阈值。
进一步，在获取到满足预设条件的所有第一像素点后，将获取到所有第一像素点记为第一像素点集，第一像素点集中存在相邻的像素点，也存在不相邻的像素点，其中，像素点相邻指的是像素点在去噪图像中的位置相邻，所述不相邻指的是像素点在去噪图像中的位置不相邻，所述位置相邻指的在待处理的像素坐标中，相邻两个像素点的横坐标和纵坐标中存在一个相同。例如，第一像素点集中包括像素点(100,101)、像素点(100,100)，像素点(101,101)以及像素点(200,200)，那么像素点(100,101)、像素点(100,100)为相邻像素点，并且像素点(100,101)、像素点(101,101)为相邻像素点，而像素点(100,101)、像素点(100,100)和像素点(101,101)和像素点(200,200)均为不相邻像素点。
进一步，所述高光区域根据第一像素点集中相邻像素点构成的连通区域，即高光区域包含的每个第一像素点的像素值均满足预设条件。由此，在本实施例一个实现方式中，所述根据满足预设条件的所有第一像素点确定所述去噪图像的高光区域具体包括：
L10、获取所述满足预设条件的所有第一像素点所形成的连通区域，并在获取到的所有连通区域进行选取满足预设规则的目标区域，其中，所述预设规则为目标区域中的第一像素点的R值、G值和B值中大于预设阈值的R值、G值和/或B值的类型相同；
L20、计算筛选得到的各目标区域分别对应的面积，并选取面积最大的目标区域作为高光区域。
具体地，所述连通区域是第一像素点集中所有相邻第一像素点形成的闭合区域，所述连通区域包含的每个像素点均为第一像素点，并且对于连通区域内的每个第一像素点A，连通区域内至少一个第一像素点B与该第一像素点A相邻。同时，针对于第一像素点集中去除该连通区域包含的第一像素点外的每个第一像素点C，该第一像素点C与连通区域内的任一第一像素点A均不相邻。例如，第一像素点集中包括像素点(100,101)、像素点(100,100)、像素点(101,100)、像素点(101,101)、像素点(100,102)以及像素点(200,200)，那么，像素点(100,101)、像素点(100,100)、像素点(101,100)、像素点(101,101)、像素点(100,102)形成一个连通区域。
此外，由于去噪图像的连通区域是有光源形成，并且光源会产生光线颜色相同。从而在获取到去噪图像包含的所有连通区域后，可以根据各连通区域对应的区域颜色对连通区域进行选取。由此，在获取到去噪图像的连通区域后，判断连通区域内各第一像素点的R值、G值和B值中第一像素点的R值、G值和B值中大于预设阈值的R值、G值和/或B值的类型是否相同，以判断连通区域是否满足预设规则。所述类型相同指的是对于两个第一像素点，分别记为像素点A和像素点B，若像素点A为R值大于预设阈值，那么像素点B也只有R值大于预设阈值；若像素点A的R值和G值均大于预设阈值，那么像素点B也只有R值和G值大于预设阈值；若像素点A的R值、G值和B值均大于预设阈值，那么像素点B的R值、G值和B值均大于预设阈值。所述类型不同指的是，对于两个第一像素点，分别记为像素点C和像素点D,若像素点C为V值(V值可以为R值，G值，B值中一种)大于预设阈值，那么像素点D中V值小于或等于预设阈值，或者像素点D中V值大于预设阈值且至少存在一个M值(M值为R值，G值和B值中去除V值外的两个值中一个)大于预设阈值。例如，像素点C的R值大于预设阈值，像素点D的R值小于等于预设阈值，那么像素点C和像素点D的类型不同；再如，像素点C的R值大于预设阈值，像素点D的R值大于预设阈值，并且像素点D的G值大于预设阈值，那么像素点C和像素点D的类型不同。本实施例中，所述预设规则为各连通区域中的第一像素点的R值、G值和B值中大于预设阈值的R值、G值和/或B值的类型相同。
进一步，由于去噪图像中可能包括多个目标区域，从而在获取到目标区域后，可以根据目标区域的面积对目标区域进行筛选以得到高光区域。其中，所述目标区域的面积指的是目标区域在去噪图像中所在区域的面积，所述面积是在去噪图像的像素坐标系 内计算的。在获取到各目标区域的面积后，可以将各目标区域的面积进行比较，并选取面积最大的目标区域，将所述目标区域作为高光区域，这样将面积最大的目标区域作为高光区域，可以获取到去噪图像中亮度面积最大的区域，根据亮度面积最大的区域确定曝光度，可以提高曝光度的准确性。
进一步，在本实施例的一个实现方式中，所述根据所述高光区域确定所述去噪图像的曝光度具体包括：
J10、计算所述高光区域的第一面积以及去噪图像的第二面积；
J20、根据所述第一面积和第二面积的比值确定所述去噪图像对应的曝光度。
具体地，所述去噪图像的第二面积指的是根据去噪图像的图像尺寸计算得到，例如，去噪图像的图像尺寸为400*400，那么去噪图像的图像面积为400*400＝160000。所述高光区域的第一面积为高光区域在去噪图像的像素坐标系中区域面积，例如，高光区域为边长为20的正方形区域，那么高光区域的第一面积为20*20＝400。
进一步，为了根据第一面积和第二面积的比值确定曝光度，预先设定了比值区间与曝光度的对应关系，在获取到比值后，首先取得比值所处比值区域，在根据该对应关系确定该比值区间对应曝光度，以得到去噪图像的曝光度。例如，所述比值区间与曝光度的对应关系为：当区间为[0,1/100)时，曝光度对应0等级；当区间为[1/100,1/50)时，曝光度对应-1等级；当区间为[1/50,1/20)时，曝光度对应-2等级；当区间为[1/20,1/50)时，曝光度对应-3等级；当区间为[1/20,1]时，曝光度对应-4等级。那么当第一面积与第二面积的比值为1/10时，该比值处于区间[1/20,1]，从而该去噪图像对应的曝光度为-4等级。
F102、根据所述曝光度确定所述去噪图像对应的模型参数，并采用所述模型参数更新所述第一图像处理模型的模型参数。
具体地，在第一图像处理模型训练时建立了曝光度与模型参数的对应关系，从而在获取到去噪图像的曝光度后，可以根据曝光度与模型参数的对应关系确定该曝光度对应的模型参数，其中，所述曝光度指的是曝光度等级，即所述曝光度与模型参数的对应关系为曝光度等级与模型参数的对应关系。此外，由上述可以知道，每个曝光等级对应一个比值区间，那么在获取到去噪图像后，可以获取去噪图像中高光区域的区域面积与图像面积的比值，并确定所述比值所处的比值区间，再根据比值区域确定去噪图像对应的曝光等级，最后根据曝光等级确定去噪图像对应的模型参数，从而得到去噪图像对应的模型参数。此外，在获取到曝光度对应的模型参数后，采用获取到的模型参数更新第 一图像处理模型配置的模型参数，以更新第一图像处理模型，即获取到的模型参数所对应的第一图像处理模型。
F103、将所述去噪图像输入至更新后的第一图像处理模型。
具体地，将去噪图像作为更新后的第一图像处理模型的输入项，并将去噪图像输出至更新后的第一图像处理模型对去噪图像进行处理。可以理解的是，所述待处理图像对应的图像处理模型的模型参数为根据所述待处理图像的曝光度确定模型参数，并且该模型参数为通过对预设网络模型进行训练得到的模型参数，这样可以保证更新后的图像处理模型对待处理图像处理的精确度。
进一步，在本实施例的一个实现方式中，所述通过所述第一图像处理模型生成所述去噪图像对应的输出图像指的是将所述去噪图像作为所述第一图像处理模型的输入项输入至所述第一图像处理模型中，通过所述第一图像处理模型调整所述去噪图像的图像色彩，以得到输出图像，其中，所述输出图像为所述待去噪图像对应的去偏色处理后的图像。例如，如图13所示的去噪图像通过所述图像处理图像后得到如图14所示的输出图像。
进一步，由所述第一图像处理模型的训练过程可以知道，所述第一图像处理模型包括下采样模块以及变换模块，从而在通过第一图像处理模型对应待处理图像进行处理时，需要依次通过下采样模块以及变换模块进行处理。相应的，所述第一图像处理模型包括；所述通过所述第一图像处理模型生成所述去噪图像对应的输出图像具体包括：
F201、将所述去噪图像输入所述下采样模块，通过所述下采样模块得到所述待处理图像对应的双边网格以及所述待处理图像对应的指导图像，其中，所述指导图像的分辨率与所述待处理图像的分辨率相同；
F202、将所述指导图像、所述双边网格以及所述去噪图像输入所述变换模块，通过变换模块生成所述去噪图像对应的输出图像。
具体地，所述下采样模块的输入项为去噪图像，输出项为待去噪图像对应的双边网格以及指导图像，所述变换模块的输入项为指导图像、双边网格以及待处理图像，输出项为输出图像。其中，所述下采样模块的结构与第一预设网络模型中的下采样模块的结构相同，具体可以参照第一预设网络模型中的下采样模块的结构的说明。所述第一图像处理模型的下采样模块的对待处理图像的处理与第一预设网络模型中的下采样模块对第一图像的处理过程相同，从而所述步骤F201的具体执行过程可以参照步骤Q11。同样的，所述变换模块的结构与第一预设网络模型中的变换模块的结构相同，具体可以参 照第一预设网络模型中的变换模块的结构的说明。所述第一图像处理模型的变换模块的对待处理图像的处理与第一预设网络模型中的变换模块对第一图像的处理过程相同，从而所述步骤F202的具体执行过程可以参照步骤Q12。
进一步，在本实施例的一个实现方式中，所述下采样模块包括下采样单元以及卷积单元。相应的，所述将所述去噪图像输入所述下采样模块，通过所述下采样模块得到所述去噪图像对应的双边网格以及所述待处理图像对应的指导图像具体包括：
F2011、将所述去噪图像分别输入所述下采样单元以及所述卷积单元；
F2012、通过所述下采样单元得到所述去噪图像对应的双边网格，并通过所述卷积单元得到所述待处理图像对应的指导图像。
具体地，所述下采样单元的输入项为去噪图像，输出项为双边网格，所述卷积单元的输入项为去噪图像，输出项为指导图像。其中，其中，所述下采样单元的结构与第一预设网络模型中的下采样单元的结构相同，具体可以参照第一预设网络模型中的下采样单元的结构的说明。所述第一图像处理模型的下采样单元的对待处理图像的处理与第一预设网络模型中的下采样单元对第一图像的处理过程相同，从而所述步骤F2011的具体执行过程可以参照步骤Q111。同样的，所述卷积单元的结构与第一预设网络模型中的卷积单元的结构相同，具体可以参照第一预设网络模型中的卷积单元的结构的说明。所述第一图像处理模型的卷积单元的对去噪图像的处理与第一预设网络模型中的卷积单元对第一图像的处理过程相同，从而所述步骤F2012的具体执行过程可以参照步骤Q112。
进一步，在本实施例的一个实现方式中，所述变换模块包括切分单元以及变换单元。相应的，所述将所述指导图像、所述双边网格以及所述待处理图像输入所述变换模块，通过变换模块生成所述去噪图像对应的输出图像具体包括：
F2021、将所述指导图像输入所述切分单元，通过所述切分单元对所述双边网格进行切分，以得到所述待处理图像中各像素点的颜色变换矩阵；
F2022、将所述去噪图像以及所述待处理图像中各像素点的颜色变换矩阵输入所述变换单元，通过所述变换单元生成所述去噪图像对应的输出图像。
具体地，所述切分单元的输入项为指导图像和双边网格，输出项为待处理图像中各像素点的颜色变换矩阵，所述变换单元的输入项为去噪图像和去噪图像中各像素点的颜色变换矩阵，输出项为输出图像。其中，其中，所述切分单元的结构与第一预设网络模型中的切分单元的结构相同，具体可以参照第一预设网络模型中的切分单元的结构的说明。所述第一图像处理模型的切分单元对待处理图像对应的双边网格以及指导图像的 处理，与第一预设网络模型中的下采样单元对第一图像对应的双边网格以及指导图像的处理过程相同，从而所述步骤F2021的具体执行过程可以参照步骤Q121。同样的，所述变换单元的结构与第一预设网络模型中的变换单元的结构相同，具体可以参照第一预设网络模型中的变换单元的结构的说明。所述第一图像处理模型的变换单元基于待处理图像中各像素点的颜色变换矩阵对待处理图像的处理与第一预设网络模型中的变换单元基于第一图像中各像素点的颜色变换矩阵对第一图像的处理过程相同，从而所述步骤F2022的具体执行过程可以参照步骤Q122。
可以理解的是，第一图像处理模型在训练过程中对应的网络结构，与在应用过程(去除待处理图像携带的偏色)中所对应的网络结构相同。例如，在训练的过程中，第一图像处理模型包括下采样模块和变换模块，那么相应地，在通过第一图像处理模型对去噪图像进行去偏色处理时，第一图像处理模型也包括下采样模块和变换模块。
例如，在训练过程中，第一图像处理模型的下采样模块包括下采样单元以及卷积单元，变换模块包括切分单元和变换单元；相应地，在通过第一图像处理模型对去噪图像进行去偏色处理时，下采样模块也可以包括下采样单元以及卷积单元，变换模块包括切分单元和变换单元；并且在应用过程中，每一层的工作原理与在训练过程中每一层的工作原理相同，因此，第一图像处理模型应用过程中的每一层神经网络的输入输出情况可以参见第一图像处理模型的训练过程中的相关介绍，这里不再赘述。
与现有技术相比，本公开提供了一种图像处理方法、存储介质以及终端设备，所述图像处理方法包括获取待处理图像集，根据所述待处理图像集，生成所述待处理图像集对应的去噪图像；将所述去噪图像输入至以已训练的第一图像处理模型，通过所述第一图像处理模型生成所述去噪图像对应的输出图像。本公开是首先获取多张图像，并根据多张图像来生成一张去噪图像，在采用基于训练图像集进行深度学习得到已训练的第一图像处理模型来调整去噪图像的图像颜色，这样使得可以提高输出图像的色彩质量以及噪声质量，从而提高图像质量。
H40、将所述处理后图像输入至以已训练的第二图像处理模型，通过所述第二图像处理模型对所述处理后图像进行去重影处理，以得到输出图像。
具体地，所述第二图像处理模型可是处理所述待处理图像集的图像设备(例如，配置屏下摄像头的手机)预先训练的，也可以是由其他训练后将第二图像处理模型对应的文件移植到图像设备中。此外，图像设备可以将所述第二图像处理模型可作为一个去重影功能模块，当图像设备获取到处理后图像时，启动所述去重影功能模块，将待处理 图像输出至第二图像处理模型。
进一步，在本实施例的一个实现方式中，如图32所示，所述第二图像处理模型的训练过程可以包括：
L100、第二预设网络模型根据所述训练图像集中第三图像，生成所述第三图像对应的生成图像。
具体地，所述第二预设网络模型可以采用深度学习网络模型，所述训练图像集包括多组具有不同图像内容的训练图像组，每一组训练图像组均包括第三图像和第四图像，第三图像与第四图像相对应，它们呈现的是同一图像场景，第四图像为正常显示的图像(即原始图像)，第三图像的图像内容与第四图像对应但图像内容中的物体出现重影或者与重影类似的模糊效果。其中，所述重影指的是图像中的物体周围形成了虚像，例如，可以包括图像中物体的边缘出现一重或多重轮廓或虚像的情况，举例来说，当图像中的物体出现了双重影像(即出现物体边缘出现一重轮廓或虚像)时，其中，像素值较小的一列影像可以理解为物体的实像，像素值较大的另一列影像可以理解为物体的轮廓或虚像。
进一步所述第三图像和第四图像对应相同的图像场景。所述第三图像和第四图像对应相同的图像场景指的是第三图像携带的图像内容与第四图像携带的图像内容的相似度达到预设阈值，所述第三图像的图像尺寸与第四图像的图像尺寸相同，以使得当第三图像和第四图像重合时，第三图像携带的物体对第四图像中与其对应的物体的覆盖率达到预设条件。其中，所述预设阈值可以为99％，所述预设条件可以为99.5％等。
此外，在本实施例的一个实现方式中，为了减少第三图像和第四图像的图像差异对第二预设网络模型训练的影响，所述第三图像的图像内容和第四图像的图像内容可以完全相同。例如，所述第三图像为图像尺寸为600*800的具有重影的图像，第三图像的图像内容为一个正方形，并且第三图像中正方形的四个顶点在第三图像中的位置分别为(200，300)、(200,400)，(300,400)以及(300,300)。那么，所述第四图像的图像尺寸为600*800的未具有重影的图像，第四图像的图像内容为一个正方形，第四图像中正方形的四个顶点在第四图像中的位置分别为(200，300)、(200,400)，(300,400)以及(300,300)，当第三图像放置于第四图像上并与第四图像重合时，所述第三图像覆盖所述第四图像，并且第三图像中的正方形与第四图像的正方形上下重叠。
进一步，所述第四图像可以是通过正常拍摄得到的图像，例如将屏下成像系统中的显示面板移除后由屏下摄像头拍摄的图像，或者通过制作不带数据线和扫描线等遮光 结构的实验性质的显示面板替代实际的显示面板，然后利用其作为屏下成像系统的显示面板而由屏下摄像头拍摄的图像，也可以是通过网络(如，百度)获取的图像，还可以是通过其他外部设备(如，智能手机)发送的图像。所述第三图像可以为通过屏下成像系统(例如，屏下摄像头)拍摄得到，也可以是通过对第四图像进行处理得到。所述对第四图像进行处理指的是在第四图像上形成重影，在一种可能的实现方式中，在处理的过程中可以同时保持第四图像的图像尺寸以及图像内容不变。
在本实施例的一个实现方式中，所述第三图像为通过屏下成像系统拍摄得到，所述第三图像和第四图像的拍摄参数相同，并且所述第三图像对应的拍摄场景与第四图像的拍摄场景相同。例如，所述第三图像为如图25所示的通过屏下摄像头拍摄一场景得到的图像，其因显示面板内遮光结构的影响导致图像内容较为模糊，第四图像为如图26所示的正常显示的图像。同时在实施例的一个可能实现方式中，所述拍摄参数可以包括成像系统的曝光参数，其中，所述曝光参数可以包括光圈、开门速度、感光度、对焦以及白平衡等。当然，在实际应用中，所述拍摄参数还可以包括环境光、拍摄角度以及拍摄范围等。
进一步，当所述第三图像为通过屏下成像系统拍摄得到的图像时，由于第三图像和第四图像可以是通过两个不同的成像系统拍摄，而在更换成像系统时，可能会造成拍摄位置或拍摄角度的变化，使得所述第三图像和第四图像在空间上存在不对齐的问题。由此，在所述第二预设网络模型根据训练图像集中第三图像，生成所述第三图像对应的生成图像之前还包括：
M10、针对所述训练图像集中每组训练图像组，将该组训练图像组中的第三图像与所述第三图像对应的第四图像进行对齐处理，得到与所述第四图像对齐的对齐图像，并将所述对齐图像作为第三图像。
具体地，所述针对所述训练图像集中每组训练图像组指的是对训练图像集中每一组训练图像组均执行对齐处理，所述对齐处理可以是在获取到训练图像集之后，分别对每一组训练图像组进行对齐处理，以得到对齐后的训练图像组，并在所有组训练图像组对齐后执行将每一组训练图像组中的第三图像输入第二预设网络模型的步骤；当然也可以是在将每一组训练图像组中的第三图像输入第二预设网络模型之前，对该组训练图像组进行对齐处理，以得到该组训练图像对应的对齐后的训练图像组，之后将对齐后的训练图像组中的第三图像输入第二预设网络模型。在本实施例中，所述对齐处理是在获取到训练图像集后，分别对每一组训练图像组进行，并在所有训练图像组完成对齐处理后， 在执行将训练图像集中第三图像输入第二预设网络模型的操作。此外，值得说明，将第三图像与所述第三图像对应的第四图像进行对齐处理的处理过程与将参考图像与所述基准图像进行对齐处理的过程相同，这里就是将第三图像作为参考图像，将第四图像作为基准图像，因而这里就不对将第三图像与第四图像的进行对齐的过程做详细说明，具体可以参照上述将参考图像与基准图像进行对齐的过程的说明。
进一步，在本实施例的一个实现方式中，如图22所示，所述第二预设网络模型包括编码器和解码器；所述第二预设网络模型根据训练图像集中第三图像，生成所述第三图像对应的生成图像具体包括：
L101、将所述训练图像集中第三图像输入所述编码器，通过所述编码器得到所述第三图像的特征图像，其中，所述特征图像的图像尺寸小于所述第三图像的图像尺寸；
L102、将所述特征图像输入所述解码器，通过所述解码器输出所述生成图像，其中，所述生成图像的图像尺寸等于第三图像的图像尺寸。
具体地，所述第二预设网络模型采用解码-编码结构，所述解码-编码结构为卷积神经网络CNN结构，其中，所述编码器1000用于将输入图像转换为图像空间尺寸小于输入图像并且通道数多于输入图像的特征图像，所述解码器2000用于将特征图像转换为与输入图像的图像尺寸相同的生成图像。在本实施例中，所述编码器包括依次布置的第一冗余学习层101以及下采样层102，训练图像组中第三图像输入至第一冗余学习层101，通过第一冗余学习层101输出与第三图像的图像尺寸相同的第一特征图；第一特征图像作为下采样层102的输入项输入下采样层102，通过下采样层102对第一特征图像进行下采样以输出所述第三图像对应的第二特征图像(第二特征图像为通过编码器生成的第三图像的特征图像)，其中，第二特征图像的图像尺寸小于第三图像的图像尺寸。所述解码器2000包括依次布置的上采样层201和第二冗余学习层202，所述编码器1000输出的特征图像输入至上采样层201，通过上采样层201进行上采样后输出第三特征图像，第三特征图像输入至第二冗余学习层202，经过第二冗余学习层202后输出生成图像，其中，所述生成图像的图像尺寸与第三图像的图像尺寸相同。本实施通过采用编码器-解码器的结构，可以对第二预设网络模型进行多尺度的训练，从而可以提高训练得到的第二图像处理模型的去重影效果。
进一步，如图22所示，所述第一冗余学习层101包括第一卷积层11以及以第一冗余学习模块12，所述下采样层102包括第一编码冗余学习模块110和第二编码冗余学习模块120，第一编码冗余学习模块110包括第一下采样卷积层13和第二冗余学习模块 14，第二编码冗余学习模块120包括第二下采样卷积层15和第三冗余学习模块16。其中，所述第一卷积层11的输入项为第三图像，并对第三图像进行采样以得到第第一特征图像，并将所述第一特征图像输入至第一冗余学习模块12进行特征提取，经过第一冗余学习模块12的第一特征图像依次通过第一下采样卷积层、第二冗余学习模块14、第二下采样卷积层15和第三冗余学习模块16进行下采样，以得到第二特征图像。由此可知，所述第一卷积层11对第三图像进行采样，所述第一下采样卷积层13和第二下采样卷积层15均用于对输入其的特征图像进行下采样，所述第一冗余学习模块12、第二冗余学习模块14和第三冗余学习模块16用于提取图像特征。此外，在本实施例的一种可能的实现方式中，所述第一下采样卷积层13和第二下采样卷积层15可以均为采用步长为2的卷积层，所述第一冗余学习模块12、第二冗余学习模块14和第三冗余学习模块16均包括依次布置的三个冗余学习块，所述三个冗余学习块依次提取输入图像的图像特征。
举例说明：假设第三图像为256*256的图像，第三图像通过输入层输入第一冗余学习层101，经过第一冗余学习层101后输出256*256的第一特征图像；第一特征图像输入第一编码冗余学习模块110的第一下采样卷积层13，经过第一下采样卷积层13处理后图像大小为128*128的第四特征图像，第四特征图像经过第一编码冗余学习模块110的第一冗余学习模块12进行特征提取；经过第一冗余学习模块12的第四特征图像输入第二编码冗余学习模块120的第二下采样卷积层15，经过第二下采样卷积层15处理后图像大小为64*64的第二特征图像，第二特征图像经过第二编码冗余学习模块120的第二冗余学习模块16进行特征提取。
进一步，如图19所示，所述上采样层201包括第一解码冗余学习模块210和第二解码冗余学习模块220，第一解码冗余学习模块210包括第四冗余学习模块21和第一上采样卷积层22，第二解码冗余学习模块220包括第五冗余学习模块23和第二上采样卷积层24，所述第二冗余学习层202包括第六冗余学习模块25和第二卷积层26。其中，所述第一上采样卷积层22的输入项为第一特征图像，输入第一特征图像依次通过第四冗余学习模块21、第一上采样卷积层22、第五冗余学习模块23和第二上采样卷积层24进行上采样以得到第三特征图像，并将所述第三特征图像输入至第六冗余学习模块25，通过第六冗余学习模块25进行特征提取后的第三特征图像输入至第二卷积层26，通过第二卷积层26得到生成图像。由此可知，所述第一上采样卷积层22和第二上采样卷积层24用于对输入其的特征图像进行上采样，所述第四冗余学习模块21、第五冗余学习 模块23以及第六冗余学习模块25均用于提取图像特征，所述第二卷积层26用于对输入其内的特征图像进行采样。在本实施例的一个可能的实现方式中，所述第一上采样卷积层22和第二上采样卷积层24均为步长为2的反卷积层，所述第四冗余学习模块21、第五冗余学习模块23以及第六冗余学习模块25均包括三个冗余学习块，所述三个冗余学习块依次提取输入图像的图像特征。此外，所述第一冗余学习层101中冗余学习模块的第三个冗余学习块与所述第二冗余学习层202中冗余学习模块的第一个冗余学习块跳跃连接，所述第一编码冗余学习模块110中冗余学习模块的第三个冗余学习块与所述第二解码冗余学习模块220中冗余学习模块的第一个冗余学习块跳跃连接。
举例说明：假设第三图像为256*256的图像经过上述编码器1000得到64*64的第二特征图像，64*64的第二特征图像输入经过第一解码冗余学习模块210的第四冗余学习模块21进行特征提取，经过特征提取的64*64的第二特征图像输入第一解码冗余学习模块210的第一上采样卷积层22，经过第一上采样卷积层22输出的图像大小为128*128的第五特征图像，第五特征图像经过第二解码冗余学习模块220的第五冗余学习模块23进行特征提取；经过第五冗余学习模块23的第五特征图像输出第二解码冗余学习模块220的第二上采样卷积层24，经过第二上采样卷积层24处理后图像大小为256*256的第三特征图像，第三特征图像输入第二冗余学习层202，经过第二冗余学习层202后输出256*256的生成图像。
进一步，所述编码器和解码器包括的第一卷积层、第二卷积层、第一上采样卷积层、第二上采样卷积层、第一下采用卷积层和第二下采用卷积层均使用线性整流函数作为激活函数且卷积核均为5*5，这样可以提高各层的梯度传递效率，并且经过多次的反向传播，梯度幅度变化小，提高了训练的生成器的准确性，同时还可以增大网络的感受野。
L200、所述第二预设网络模型根据所述第三图像对应的第四图像和所述第三图像对应的生成图像，对模型参数进行修正，并继续执行根据训练图像集中的下一训练图像组中的第三图像，生成所述第三图像对应的生成图像的步骤，直至所述第二预设网络模型的训练情况满足预设条件，以得到所述第二图像处理模型。
具体地，所述对所述第二预设网络模型进行修正指的是对所述第二预设网络模型的模型参数进行修正，直至所述模型参数满足预设条件。所述预设条件包括损失函数值满足预设要求或者训练次数达到预设次数。所述预设要求可以是根据第二图像处理模型的精度来确定，这里不做详细说明，所述预设次数可以为第二预设网络模型的最大训练 次数，例如，4000次等。由此，在第二预设网络模型输出生成图像，根据所述生成图像以及所述第四图像来计算第二预设网络模型的损失函数值，在计算得到损失函数值后，判断所述损失函数值是否满足预设要求；若损失函数值满足预设要求，则结束训练；若损失函数值不满足预设要求，则判断所述第二预设网络模型的训练次数是否达到预测次数，若未达到预设次数，则根据所述损失函数值对所述第二预设网络模型的网络参数进行修正；若达到预设次数，则结束训练。这样通过损失函数值和训练次数来判断第二预设网络模型训练是否结束，可以避免因损失函数值无法达到预设要求而造成第二预设网络模型的训练进入死循环。
进一步，由于对第二预设网络模型的网络参数进行修改是在第二预设网络模型的训练情况未满足预设条件(例如，损失函数值未满足预设要求并且训练次数未达到预设次数)，从而在根据损失函数值对所述第二预设网络模型的网络参数进行修正后，需要继续对网络模型进行训练，即继续执行将训练图像集中的第三图像输入第二预设网络模型的步骤。其中，继续执行将训练图像集中第三图像输入第二预设网络模型中的第三图像可以为未作为输入项输入过第二预设网络模型的第三图像。例如，训练图像集中所有第三图像具有唯一图像标识(例如，图像编号)，第一次训练输入第二预设网络模型的第三图像的图像标识与第二次训练输入第二预设网络模型的第三图像的图像标识不同，如，第一次训练输入第二预设网络模型的第三图像的图像编号为1，第二次训练输入第二预设网络模型的第三图像的图像编号为2，第N次训练输入第二预设网络模型的第三图像的图像编号为N。当然，在实际应用中，由于训练图像集中的第三图像的数量有限，为了提高第二图像处理模型的训练效果，可以依次将训练图像集中的第三图像输入至第二预设网络模型以对第二预设网络模型进行训练，当训练图像集中的所有第三图像均输入过第二预设网络模型后，可以继续执行依次将训练图像集中的第三图像输入至第二预设网络模型的操作，以使得训练图像集中的训练图像组按循环输入至第二预设网络模型。需要说明的是，在将第三图像输入第二预设网络模型训练的过程中，可以按照各个第三图像的图像编号顺序输入，也可以不按照各个第三图像的图像编号顺序输入，当然，可以重复使用同一张第三图像对第二预设网络模型进行训练，也可以不重复使用同一张第三图像对第二预设网络模型进行训练，在本实施例中，不对“继续执行将训练图像集中的第三图像输入第二预设网络模型的步骤”的具体实现方式进行限定。
进一步，在本实施例的一个实现方式中，所述损失函数值为结构相似性损失函数和内容双向损失函数计算得到的。相应的，如图33所示，所述第二预设网络模型根据 所述第三图像对应的第四图像和所述第三图像对应的生成图像，对所述第二预设网络模型的模型参数进行修正，并继续执行根据训练图像集中第三图像，生成所述第三图像对应的生成图像的步骤，直至所述第二预设网络模型的训练情况满足预设条件，以得到已训练的第二图像处理模型具体包括：
L201、根据所述第三图像对应的第四图像和所述第三图像对应的生成图像计算所述第二预设网络模型对应的结构相似性损失函数值和内容双向损失函数值；
L202、根据所述结构相似性损失函数值和所述内容双向损失函数值得到所述第二预设网络模型的总损失函数值；
L203、基于所述总损失函数值训练所述第二预设网络模型，并继续执行将第二预设训练图像集中的下一训练图像组中的第三图像输入第二预设网络模型的步骤，直至所述第二预设网络模型的训练情况满足预设条件，以得到已训练的第二图像处理模型。
具体地，所述第二预设网络模型采用结构相似性指数(Structural similarity index，SSIM)损失函数和基于VGG(Visual Geometry Group Network，VGG网络)提取特征的内容双向(Contextual bilateral loss,CoBi)损失函数的结合作为损失函数。那么，在计算所述第二预设网络模型的损失函数值时，可以分别计算结构相似性损失函数值和内容双向损失函数值，再根据所述结构相似性损失函数值和内容双向损失函数值计算第二预设网络模型中损失函数值。在本实施例中，所述第二预设网络模型的总损失函数值＝a*结构相似性损失函数值+b*内容双向损失函数值，所述a和b为权重系数。例如，所述权重系数a和权重系数b均为1，那么所述第二预设网络模型的总损失函数值＝结构相似性损失函数值+内容双向损失函数值。此外，在本实施例中，在采用总损失函数值对第二预设网络模型进行训练时采用随机梯度下降法对第二预设网络模型进行训练，其中，训练的初始网络参数设为0.0001，并且网络参数在修正时采用指数衰减的方式进行修正。
进一步，所述结构相似性损失函数值用于衡量生成图像与第四图像之间结构的相似性，所述结构相似性损失函数值越大，生成图像与第四图像的相似性越高，反之，所述结构相似性损失函数值越小，生成图像与第四图像的相似性越低。因此，结构相似性损失函数值对局部结构变化较敏感，更接近于人眼的感知系统，从而可以提高第二预设网络模型的精确性。在本实施例中，所述结构相似性损失函数值对应的结构相似性损失函数的表达式可以为：
Figure PCTCN2020141932-appb-000013
其中，μ x为生成图像中所有像素点的像素值的平均值，μ y为第二图像中所有像素点的像素值的平均值，σ x为生成图像中所有像素点的像素值的方差，σ y为第二图像中所有像素点的像素值的方差，σ xy为生成图像与第二图像的协方差。
进一步，所述内容双向损失函数值为通过基于VGG特征的CoBi损失函数计算得到，所述基于VGG特征的CoBi损失函数通过分别提取生成图像与第四图像的若干组VGG特征，并且针对生成图像的每个第一VGG特征，在第四图像的第二VGG特征中搜索与该第一VGG特征接近的第二VGG特征匹配，最后计算每个第一VGG特征与其匹配的第二VGG特征的距离和，以得到内容双向损失函数值，这样通过内容双向损失函数对双边距离进行搜索，考虑了第一VGG特征与其匹配的第二VGG特征在空间上的损失，从而可以避免第三图像和第四图像未完全对齐产生的影响，提高了第二预设网络模型训练的速度以及准确性。此外，在搜索所述第一VGG特征匹配的第二VGG特征时，根据第一VGG特征和第二VGG特征的距离和位置关系两个方面确定内容双向损失函数值，提高了匹配的精确性，从而进一步降低第三图像和第四图像不对齐对第二预设网络模型训练的影响。在本实施例中，所述内容双向损失函数的表达式可以为：
Figure PCTCN2020141932-appb-000014
其中,D为生成图像的VVG特征与第二图像的VVG特征之间的余弦距离，D′为生成图像的VVG特征与第二图像的VVG特征之间的空间位置距离，N为生成图像的VVG特征的特征数量，ω s为权重系数。
N200、通过所述第二图像处理模型对所述处理后图像进行去重影处理，并将去重影处理后图像作为处理后图像。
具体地，所述通过所述第二图像处理模型对所述待处理图像进行去重影指的是将所述待处理图像作为所述第二图像处理模型的输入项输入至所述第二图像处理模型中，通过所述第二图像处理模型去除所述待处理图像的重影，以得到处理后图像，其中，所述处理后图像为所述待处理图像对应的通过第二图像处理模型处理得到的图像。可以理解的是，待处理图像为输出图像对应的具有重影的图像，即输出图像与待处理图像相对应，它们呈现的是同一图像场景，输出图像为正常显示的图像，待处理图像的图像内容与输出图像对应，但待处理图像内容中的物体出现重影或者与重影类似的模糊效果。例 如，如图25所示的待处理图像通过所述图像处理图像后得到如图26所示的输出图像。
进一步，由所述第二图像处理模型的训练过程可以知道，所述第二图像处理模型包括编码器和解码器，从而在通过第二图像处理模型对应待处理图像进行处理时，需要分别通过编码器和解码器进行处理。相应的，所述通过所述第二图像处理模型对所述待处理图像进行去重影，以得到所述待处理图像对应的处理后图像具体包括：
N201、将所述待处理图像输入所述编码器，通过所述编码器得到所述待处理图像的特征图像，其中，所述特征图像的图像尺寸小于所述待处理图像的图像尺寸；
N202、将所述特征图像输入所述解码器，通过所述解码器输出所述待处理图像对应的处理后图像，其中，所述处理后图像的图像尺寸等于所述待处理图像的图像尺寸。
具体地，所述编码器将输入的待处理图像转换为图像空间尺寸小于输入图像并且通道数多于输入图像的特征图像，并将所述特征图像输入至解码器，所述解码器将输入的特征图像转换为与待处理图像的图像尺寸相同的生成图像。其中，所述编码器的结构与第二预设网络模型中的编码器的结构相同，具体可以参照第二预设网络模型中的编码器的结构的说明。所述第二图像处理模型的编码器的对待处理图像的处理与第二预设网络模型中的编码器对第三图像的处理过程相同，从而所述步骤N201的具体执行过程可以参照步骤L101。同样的，所述解码器的结构与第二预设网络模型中的解码器的结构相同，具体可以参照第二预设网络模型中的解码器的结构的说明。所述第二图像处理模型的解码器的对待处理图像对应的特征图像的处理与第二预设网络模型中的解码器对第三图像对应的特征图像的处理过程相同，从而所述步骤N202的具体执行过程可以参照步骤L202。
可以理解的是，第二图像处理模型在训练过程中对应的网络结构，与在应用过程(去除处理后图像携带的重影)中所对应的网络结构相同。例如，在训练的过程中，第二图像处理模型包括编码器和编码器，那么相应地，在通过第二图像处理模型去除处理后图像携带的重影时，第二图像处理模型也包括编码器和编码器。
进一步地，例如，在训练过程中，第二图像处理模型的编码器包括所述编码器包括第一冗余学习层和下采样层，解码器包括上采样层和第二冗余学习层；相应地，在通过第二图像处理模型去除处理后图像携带的重影时，编码器也可以包括第一冗余学习层和下采样层，解码器包括上采样层和第二冗余学习层；并且在应用过程中，每一层的工作原理与在训练过程中每一层的工作原理相同，因此，第二图像处理模型应用过程中的每一层神经网络的输入输出情况可以参见第二图像处理模型的训练过程中的相关介绍， 这里不再赘述。
为了进一步提高图像处理方法得到的输出图像的图像质量，在获取到输出图像后，还可以对所述输出图像进行后处理，其中，所述后处理可以包括锐化处理以及降噪处理等。相应的，所述通过所述第二图像处理模型对所述处理后图像进行去重影处理，以得到输出图像之后，所述方法还包括：
对所述处理后图像进行锐化以及降噪处理，并将锐化以及降噪后的输出图像作为所述待处理图像对应的输出图像。
具体地，所述锐化处理指的是补偿输出图像的轮廓、增强输出图像的边缘及灰度跳变的部分，以提高处理后图像的图像质量。其中，所述锐化处理可以采用现有的锐化处理方法，例如，高通滤波方法等。所述降噪处理指的是去除图像中的噪声，提高图像的信噪比。其中，所述降噪处理可以采用现有的降噪算法或已训练的降噪网络模型等，例如，所述降噪处理采用高斯低通滤波方法等。
基于上述图像处理方法，如图38所示，本实施例提供了一种图像处理装置，所述图像处理装置包括：
第三获取模块501，用于获取待处理图像集，其中，所述待处理图像集包括多张图像；
第三生成模块502，用于根据所述待处理图像集，生成所述待处理图像集对应的去噪图像；
第三处理模块503，用于将所述去噪图像输入至以已训练的第一图像处理模型，通过所述图像处理模型对所述去噪图像进行去偏色处理，得到所述去噪图像对应的处理后图像；
第四处理模块504，用于将所述处理后图像输入至以已训练的第二图像处理模型，通过所述第二图像处理模型对所述处理后图像进行去重影处理，以得到输出图像。
在一个实施例中，所述待处理图像集包括的多张图像中一张图像为基础图像，其余图像为基础图像的临近图像,所述第三生成模块具体用于：
将所述基础图像划分为若干基础图像块，分别确定各基础图像在各临近图像中对应的临近图像块；
确定各个基础图像块分别对应的权重参数集；其中，基础图像块对应的权重参数集包括第一权重参数和第二权重参数，第一权重参数为基础图像块的权重参数，第二权重参数为临近图像中与基础图像块对应的临近图像块的权重参数；
根据所述待处理图像集以及各个基础图像块分别对应的权重参数集，确定去噪图像。
在一个实施例中，所述待处理图像集的图像数量为根据所述待处理图像集对应的拍摄参数确定的。
在一个实施例中，所述基础图像的图像清晰度大于或等于所述临近图像的图像清晰度。
在一个实施例中，所述第三生成模块具体用于：
针对每个基础图像块，确定该基础图像块对应的各临近图像块的第二权重参数，以及，获取该基础图像块对应的第一权重参数，以得到该基础图像块对应的权重参数集。
在一个实施例中，所述第三生成模块具体用于：
针对每个临近图像块，计算该基础图像块与该临近图像块的相似度值；
根据所述相似度值计算该临近图像块的第二权重参数。
在一个实施例中，所述第三生成模块具体用于：
当所述相似度值小于或等于第一阈值时，将第一预设参数作为该临近图像块的第二权重参数；
当所述相似度值大于第一阈值，且小于或等于第二阈值时，根据所述相似度值、所述第一阈值及所述第二阈值计算该临近图像块的第二权重参数；
当所述相似度值大于第二阈值时，将预设第二预设参数作为该临近图像块的第二权重参数。
在一个实施例中，所述第一阈值和第二阈值均为根据该基础图像块对应临近图像块的相似度值确定的。
在一个实施例中，所述图像处理装置还包括：
空域降噪模块，用于对所述去噪图像进行空域降噪，并将空域降噪后得到的图像作为去噪图像。
在一个实施例中，所述第一图像处理模型为基于第一训练图像集训练得到，所述第一训练图像集包括多组训练图像组，每一组训练图像组包括第一图像和第二图像，第一图像为对应第二图像的偏色图像。
在一个实施例中，所述第一图像为通过屏下成像系统拍摄得到的图像。
在一个实施例中，所述图像处理装置还包括：
第三对齐模块，用于针对所述第一训练样本集中每组训练图像组，将该组训练图 像组中的第一图像与所述第一图像对应的第二图像进行对齐处理，得到与所述第二图像对齐的对齐图像，并将所述对齐图像作为第一图像。
在一个实施例中，所述第一训练图像集包括若干训练子图像集，每个训练子图像集包括若干组训练样本图像组，若干训组训练图像组中的任意两组训练样本图像组中的第一图像的曝光度相同，若干组训练图像组中的每组训练样本图像组中的第二图像的曝光度均处于预设范围内，任意两个训练子图像集中的第一图像的曝光度不相同。
在一个实施例中，所述第一图像处理模型对应若干模型参数，每个模型参数均为根据所述第一训练图像集中一个训练子图像集训练得到的，并且任意两个模型参数各自分别对应的训练子图像集互不相同。
在一个实施例中，所述第三处理模块具体用于：
提取所述去噪图像的曝光度；根据所述曝光度确定所述去噪图像对应的模型参数，并采用所述模型参数更新所述第一图像处理模型的模型参数；以及将所述去噪图像输入至更新后的第一图像处理模型。
在一个实施例中，所述第三处理模块具体用于：
根据所述去噪图像中各像素点的R值、G值以及B值确定满足预设条件的第三像素点，其中，所述预设条件为R值、G值以及B值中至少一个值大于预设阈值；根据满足预设条件的所有第三像素点确定所述去噪图像的高光区域，并根据所述高光区域确定所述去噪图像的曝光度。
在一个实施例中，所述第三处理模块具体用于：
获取所述满足预设条件的所有第三像素点所形成的连通区域，并在获取到的所有连通区域进行选取满足预设规则的目标区域，计算筛选得到的各目标区域分别对应的面积，并选取面积最大的目标区域作为高光区域，其中，所述预设规则为目标区域中的第三像素点的R值、G值和B值中大于预设阈值的R值、G值和/或B值的类型相同。
在一个实施例中，所述第三处理模块具体用于：
计算所述高光区域的第一面积以及去噪图像的第二面积；以及根据所述第一面积和第二面积的比值确定所述去噪图像对应的曝光度。
在一个实施例中，所述第一图像处理模型包括下采样模块以及变换模块；所述第三处理模块具体用于：
将所述去噪图像输入所述下采样模块，通过所述下采样模块得到所述去噪图像对应的双边网格以及所述去噪图像对应的指导图像，其中，所述指导图像的分辨率与所述 去噪图像的分辨率相同；
将所述指导图像、所述双边网格以及所述去噪图像输入所述变换模块，通过变换模块生成所述去噪图像对应的处理后图像。
在一个实施例中，所述下采样模块包括下采样单元和卷积单元；所述第三处理模块具体用于：
将所述去噪图像分别输入所述下采样单元以及所述卷积单元；
通过所述下采样单元得到所述去噪图像对应的双边网格，并通过所述卷积单元得到所述去噪图像对应的指导图像。
在一个实施例中，所述变换模块包括切分单元以及变换单元，所述第三处理模块具体用于：
将所述指导图像输入所述切分单元，通过所述切分单元对所述双边网格进行切分，以得到所述去噪图像中各像素点的颜色变换矩阵；
将所述去噪图像以及所述去噪图像中各像素点的颜色变换矩阵输入所述变换单元，通过所述变换单元生成所述去噪图像对应的处理后图像。
在一个实施例中，所述第二图像处理模型为基于第二训练图像集训练得到，所述第二训练图像集包括多组训练图像组，每一组训练图像组包括第三图像和第四图像，第三图像为第四图像对应的具有重影的图像。
在一个实施例中，所述第三图像为根据第四图像和点扩散函数生成的，其中，所述点扩散函数为根据屏下成像系统中的遮光结构生成的灰度图生成的。
在一个实施例中，所述第三图像为通过屏下成像系统拍摄得到的图像。
在一个实施例中，所述屏下成像系统为屏下摄像头。
在一个实施例中，所述图像处理装置还包括：
第四对齐模块，用于针对所述第二训练图像集中每组训练图像组，将该组训练图像组中的第三图像与所述第三图像对应的第四图像进行对齐处理，得到与所述第四图像对齐的对齐图像，并将所述对齐图像作为第三图像。
在一个实施例中，所述第二图像处理模型包括编码器和解码器；所述第四处理模块具体用于：
将所述处理后图像输入所述编码器，通过所述编码器得到所述处理后图像的特征图像；以及将所述特征图像输入所述解码器，通过所述解码器输出所述处理后图像对应的输出图像，其中，所述特征图像的图像尺寸小于所述处理后图像的图像尺寸；所述输 出图像的图像尺寸等于所述处理后图像的图像尺寸。
在一个实施例中，所述第三对齐模块和/或所述第四对齐模块均具体用于：
获取训练图像组中的基准图像和参考图像，并计算所述参考图像与所述基准图像之间的像素偏差量；以及根据所述像素偏差量确定所述参考图像对应的对齐方式，并采用所述对齐方式将所述参考图像与所述基准图像进行对齐处理，其中，当基准图像为第二图像时，参考图像为第一图像；当基准图像为第四图像时，参考图像为第三图像。
在一个实施例中，所述第三对齐模块和/或所述第四对齐模块均具体用于：
当所述像素偏差量小于或等于预设偏差量阈值时，根据所述参考图像与所述基准图像的互信息，以所述基准图像为基准对所述参考图像进行对齐处理；
当所述像素偏差量大于所述预设偏差量阈值时，提取所述参考图像的基准像素点集和所述基准图像的参考像素点集，所述基准像素点集包含所述参考图像中的若干参考像素点，所述基准像素点集包括所述基图像中的若干基准像素点，所述参考像素点集中的参考像素点与所述基准像素点集中的基准像素点一一对应；针对所述基准像素点集中每个基准像素点，计算该基准像素点与其对应的参考像素点的坐标差值，并根据该参考像素点对应的坐标差值对该参考像素点进行位置调整，以将该参考准像素点与该参考像素点对应的基准像素点对齐。
在一个实施例中，所述图像处理装置还包括：
锐化降噪模块，用于对所述处理后图像进行锐化以及降噪处理，并将锐化以及降噪处理后的处理后图像作为所述输出图像。
基于上述图像处理方法，本实施例提供了一种计算机可读存储介质，所述计算机可读存储介质存储有一个或者多个程序，所述一个或者多个程序可被一个或者多个处理器执行，以实现如上述实施例所述的图像处理方法中的步骤。
基于上述图像处理方法，本公开还提供了一种终端设备，如图39所示，其包括至少一个处理器(processor)30；显示面板31；以及存储器(memory)32，还可以包括通信接口(Communications Interface)33和总线34。其中，处理器30、显示面板31、存储器32和通信接口33可以通过总线34完成相互间的通信。显示面板31设置为显示初始设置模式中预设的用户引导界面。通信接口33可以传输信息。处理器30可以调用存储器32中的逻辑指令，以执行上述实施例中的方法。
此外，上述的存储器32中的逻辑指令可以通过软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。

Claims (53)

  1. 一种图像处理模型的生成方法，其中，所述图像处理模型的生成方法具体包括：
    预设网络模型根据训练图像集中的第一图像，生成所述第一图像对应的生成图像，其中，所述训练图像集包括多组训练图像组，每一组训练图像组包括第一图像和第二图像，第一图像为对应第二图像的偏色图像；
    所述预设网络模型根据所述第一图像对应的第二图像和所述第一图像对应的生成图像，对模型参数进行修正，并继续执行根据所述训练图像集中的下一训练图像组中的第一图像，生成所述第一图像对应的生成图像的步骤，直至所述预设网络模型的训练情况满足预设条件，以得到所述图像处理模型。
  2. 根据权利要求1所述图像处理模型的生成方法，其中，所述第一图像中满足预设偏色条件的第一目标像素点的数量满足预设数量条件；所述预设偏色条件为第一图像中第一目标像素点的显示参数与第二图像中第二目标像素点的显示参数之间的误差满足预设误差条件，其中，所述第一目标像素点与所述第二目标像素点之间具有一一对应关系。
  3. 根据权利要求2所述图像处理模型的生成方法，其中，所述第一目标像素点为所述第一图像中任意一个像素点或者所述第一图像的目标区域中任意一个像素点。
  4. 根据权利要求1所述图像处理模型的生成方法，其中，所述训练图像集包括若干训练子图像集，每个训练子图像集包括若干组训练样本图像组，若干训组训练图像组中的任意两组训练样本图像组中的第一图像的曝光度相同，若干组训练图像组中的每组训练样本图像组中的第二图像的曝光度均处于预设范围内，任意两个训练子图像集中的第一图像的曝光度不相同。
  5. 根据权利要求4所述图像处理模型的生成方法，其中，所述图像处理模型对应若干模型参数，每个模型参数均为根据所述训练图像集中的一个训练子图像集训练得到的，并且任意两个模型参数各自分别对应的训练子图像集互不相同。
  6. 根据权利要求1所述的图像处理模型的生成方法，其中，所述预设网络模型包括下采样模块以及变换模块；所述预设网络模型根据训练图像集中第一图像，生成所述第一图像对应的生成图像具体包括：
    将所述训练图像集中的第一图像输入所述下采样模块，通过所述下采样模块得到所述第一图像对应的双边网格以及所述第一图像对应的指导图像，其中，所述指导图像的分辨率与所述第一图像的分辨率相同；
    将所述指导图像、所述双边网格以及所述第一图像输入所述变换模块，通过变换 模块生成所述第一图像对应的生成图像。
  7. 根据权利要求6所述的图像处理模型的生成方法，其中，所述下采样模块包括下采样单元和卷积单元；所述将所述训练图像集中第一图像输入所述下采样模块，通过所述下采样模块得到所述第一图像对应的双边网格参数以及所述第一图像对应的指导图像具体包括：
    将所述训练图像集中第一图像分别输入所述下采样单元以及所述卷积单元；
    通过所述下采样单元得到所述第一图像对应的双边网格，并通过所述卷积单元得到所述第一图像对应的指导图像。
  8. 根据权利要求6所述的图像处理模型的生成方法，其中，所述变换模块包括切分单元以及变换单元，所述将所述指导图像、所述双边网格以及所述第一图像输入所述变换模块，通过变换模块生成所述第一图像对应的生成图像具体包括：
    将所述指导图像输入所述切分单元，通过所述切分单元对所述双边网格进行切分，以得到所述第一图像中各像素点的颜色变换矩阵；
    将所述第一图像以及所述第一图像中各像素点的颜色变换矩阵输入所述变换单元，通过所述变换单元生成所述第一图像对应的生成图像。
  9. 根据权利要求1所述的图像处理模型的生成方法，其中，所述第一图像为通过屏下成像系统拍摄得到的图像。
  10. 根据权利要求9所述的图像处理模型的生成方法，其中，所述屏下成像系统为屏下摄像头。
  11. 根据权利要求1-10任一所述的图像处理模型的生成方法，其中，所述预设网络模型根据训练图像集中第一图像，生成所述第一图像对应的生成图像之前还包括：
    针对所述训练图像集中每组训练图像组，将该组训练图像组中的第一图像与所述第一图像对应的第二图像进行对齐处理，得到与所述第二图像对齐的对齐图像，并将所述对齐图像作为第一图像。
  12. 根据权利要求11所述的图像处理模型的生成方法，其中，所述将该组训练图像组中的第一图像与所述第一图像对应的第二图像进行对齐处理具体包括：
    获取该组训练图像组中的第一图像与所述第一图像对应的第二图像之间的像素偏差量；
    根据所述像素偏差量确定所述第一图像对应的对齐方式，并采用所述对齐方式将所述第一图像与第二图像进行对齐处理。
  13. 根据权利要求12所述的图像处理模型的生成方法，其中，所述根据所述像素偏差量确定所述述第一图像对应的对齐方式，并采用所述对齐方式将所述第一图像与第二图像进行对齐处理具体包括：
    当所述像素偏差量小于等于预设偏差量阈值时，根据所述第一图像与所述第二图像的互信息，以所述第二图像为基准对所述第一图像进行对齐处理；
    当所述像素偏差量大于所述预设偏差量阈值时，提取所述第一图像的第一像素点集和所述第二图像的第二像素点集，所述第一像素点集包含所述第一图像中的若干第一像素点，所述第二像素点集包括所述第二图像中的若干个第二像素点，所述第二像素点集中的第二像素点与所述第一像素点集中的第一像素点一一对应；针对第一像素点集中每个第一像素点，计算该第一像素点与其对应的第二像素点的坐标差值，并根据该第一像素点对应的坐标差值对该第一像素点进行位置变换，以将该第一像素点与该第一像素点对应的第二像素点对齐。
  14. 一种图像处理方法，其中，应用如权利要求1-13任意一项所述的图像处理模型的生成方法生成的图像处理模型，所述图像处理方法包括：
    获取待处理图像，并将所述待处理图像输入至所述图像处理模型；
    通过所述图像处理模型对所述待处理图像进行偏色处理，以得到所述待处理图像对应的处理后的图像。
  15. 根据权利要求14所述图像处理方法，其中，所述图像处理模型对应若干模型参数，每个模型参数均为根据一个训练子图像集训练得到的，并且任意两个模型参数各自分别对应的训练子图像集互不相同。
  16. 根据权利要求15所述图像处理方法，其中，所述获取待处理图像，并将所述待处理图像输入至所述图像处理模型具体包括：
    获取待处理图像，并提取所述待处理图像的曝光度；
    根据所述曝光度确定所述待处理图像对应的模型参数，并采用所述模型参数更新所述图像处理模型的模型参数；
    将所述待处理图像输入至更新后的图像处理模型。
  17. 根据权利要求16所述图像处理方法，其中，所述提取所述待处理图像的曝光度具体包括：
    根据所述待处理图像中各像素点的R值、G值以及B值确定满足预设条件的第三像素点，其中，所述预设条件为R值、G值以及B值中至少一个值大于预设阈值；
    根据满足预设条件的所有第三像素点确定所述待处理图像的高光区域，并根据所述高光区域确定所述待处理图像的曝光度。
  18. 根据权利要求17所述图像处理方法，其中，所述根据满足预设条件的所有第三像素点确定所述待处理图像的高光区域具体包括：
    获取所述满足预设条件的所有第三像素点所形成的连通区域，并在获取到的所有连通区域进行选取满足预设规则的目标区域，其中，所述预设规则为目标区域中的第三像素点的R值、G值和B值中大于预设阈值的R值、G值和/或B值的类型相同；
    计算筛选得到的各目标区域分别对应的面积，并选取面积最大的目标区域作为高光区域。
  19. 根据权利要求17所述图像处理方法，其中，所述根据所述高光区域确定所述待处理图像的曝光度具体包括：
    计算所述高光区域的第一面积以及待处理图像的第二面积；
    根据所述第一面积和第二面积的比值确定所述待处理图像对应的曝光度。
  20. 根据权利要求14-19任意一项所述的图像处理方法，其中，所述图像处理模型包括下采样模块以及变换模块；所述通过所述图像处理模型对所述待处理图像进行偏色处理，以得到所述待处理图像对应的处理后的图像具体包括：
    将所述待处理图像输入所述下采样模块，通过所述下采样模块得到所述待处理图像对应的双边网格以及所述待处理图像对应的指导图像，其中，所述指导图像的分辨率与所述待处理图像的分辨率相同；
    将所述指导图像、所述双边网格以及所述待处理图像输入所述变换模块，通过变换模块生成所述第一图像对应的处理后的图像。
  21. 根据权利要求20所述的图像处理方法，其中，所述下采样模块包括下采样单元和卷积单元；所述将所述待处理图像输入所述下采样模块，通过所述下采样模块得到所述待处理图像对应的双边网格以及所述待处理图像对应的指导图像具体包括：
    将所述待处理图像分别输入所述下采样单元以及所述卷积单元；
    通过所述下采样单元得到所述待处理图像对应的双边网格，并通过所述卷积单元得到所述待处理图像对应的指导图像。
  22. 根据权利要求21所述的图像处理方法，其中，所述变换模块包括切分单元以及变换单元，所述将所述指导图像、所述双边网格以及所述待处理图像输入所述变换模块，通过变换模块生成所述待处理图像对应的处理后的图像具体包括：
    将所述指导图像输入所述切分单元，通过所述切分单元对所述双边网格进行切分，以得到所述待处理图像中各像素点的颜色变换矩阵；
    将所述待处理图像以及所述待处理图像中各像素点的颜色变换矩阵输入所述变换单元，通过所述变换单元生成所述待处理图像对应的处理后的图像。
  23. 根据权利要求14-19任意一项所述的图像处理方法，其中，所述通过所述图像处理模型对所述待处理图像进行偏色处理，以得到所述待处理图像对应的处理后的图像之后还包括：
    对所述处理后的图像进行锐化以及降噪处理，并将锐化以及降噪处理后的图像作为所述待处理图像对应的处理后的图像。
  24. 一种图像处理模型的生成方法，其特征在于，其包括：
    预设网络模型根据训练图像集中的第一图像生成所述第一图像对应的生成图像；其中，所述训练图像集包括多组训练图像组，每一组训练图像组包括第一图像和第二图像，第一图像为第二图像对应的具有重影的图像；
    所述预设网络模型根据所述第一图像对应的第二图像和所述第一图像对应的生成图像，对所述预设网络模型的模型参数进行修正，并继续执行根据训练图像集中的下一训练图像组中的第一图像，生成所述第一图像对应的生成图像的步骤，直至所述预设网络模型的训练情况满足预设条件，以得到已训练的图像处理模型。
  25. 根据权利要求24所述的图像处理模型的生成方法，其特征在于，所述预设网络模型包括编码器和解码器；所述预设网络模型根据训练图像集中第一图像，生成所述第一图像对应的生成图像具体包括：
    将所述训练图像集中第一图像输入所述编码器，通过所述编码器得到所述第一图像的特征图像，其中，所述特征图像的图像尺寸小于所述第一图像的图像尺寸；
    将所述特征图像输入所述解码器，通过所述解码器输出所述生成图像，其中，所述生成图像的图像尺寸等于第一图像的图像尺寸。
  26. 根据权利要求24所述的图像处理模型的生成方法，其特征在于，所述预设网络模型根据所述第一图像对应的第二图像和所述第一图像对应的生成图像，对所述预设网络模型的模型参数进行修正，并继续执行根据训练图像集中的下一训练图像组中的第一图像，生成所述第一图像对应的生成图像的步骤，直至所述预设网络模型的训练情况满足预设条件，以得到已训练的图像处理模型具体包括：
    根据所述第一图像对应的第二图像和所述第一图像对应的生成图像分别计算所述 预设网络模型对应的结构相似性损失函数值和内容双向损失函数值；
    根据所述结构相似性损失函数值和所述内容双向损失函数值得到所述预设网络模型的总损失函数值；
    基于所述总损失函数值训练所述预设网络模型，并继续执行根据训练图像集中的下一训练图像组中的第一图像，生成所述第一图像对应的生成图像的步骤，直至所述预设网络模型的训练情况满足预设条件，以得到已训练的图像处理模型。
  27. 根据权利要求24-26任意一项所述图像处理模型的生成方法，其特征在于，所述第一图像为根据第二图像和点扩散函数生成的，其中，所述点扩散函数为根据屏下成像系统中的遮光结构生成的灰度图生成的。
  28. 根据权利要求24-26任意一项所述的图像处理模型的生成方法，其特征在于，所述第一图像为通过屏下成像系统拍摄得到的图像。
  29. 根据权利要求28所述的图像处理模型的生成方法，其特征在于，所述屏下成像系统为屏下摄像头。
  30. 根据权利要求28所述的图像处理模型的生成方法，其特征在于，所述预设网络模型根据训练图像集中第一图像，生成所述第一图像对应的生成图像之前还包括：
    针对所述训练图像集中每组训练图像组，将该组训练图像组中的第一图像与所述第一图像对应的第二图像进行对齐处理，得到与所述第二图像对齐的对齐图像，并将所述对齐图像作为第一图像。
  31. 根据权利要求30所述的图像处理模型的生成方法，其特征在于，所述将该组训练图像组中的第一图像与所述第一图像对应的第二图像进行对齐处理具体包括：
    获取该组训练图像组中的第一图像与所述第一图像对应的第二图像之间的像素偏差量；
    根据所述像素偏差量确定所述第一图像对应的对齐方式，并采用所述对齐方式将所述第一图像与所述第二图像进行对齐处理。
  32. 根据权利要求31所述的图像处理模型的生成方法，其特征在于，所述根据所述像素偏差量确定所述述第一图像对应的对齐方式，并采用所述对齐方式将所述第一图像与所述第二图像进行对齐处理具体包括：
    当所述像素偏差量小于或等于预设偏差量阈值时，根据所述第一图像与所述第二图像的互信息，以所述第二图像为基准对所述第一图像进行对齐处理；
    当所述像素偏差量大于所述预设偏差量阈值时，提取所述第一图像的第一像素点 集和所述第二图像的第二像素点集，所述第一像素点集包含所述第一图像中的若干第一像素点，所述第二像素点集包括所述第二图像中的若干第二像素点，所述第二像素点集中的第二像素点与所述第一像素点集中的第一像素点一一对应；针对所述第一像素点集中每个第一像素点，计算该第一像素点与其对应的第二像素点的坐标差值，并根据该第一像素点对应的坐标差值对该第一像素点进行位置调整，以将该第一像素点与该第一像素点对应的第二像素点对齐。
  33. 一种图像处理方法，其特征在于，应用如权利要求24-32任意一项所述的图像处理模型的生成方法训练得到的图像处理模型，所述图像处理方法包括：
    获取待处理图像，并将所述待处理图像输入至所述图像处理模型；
    通过所述图像处理模型对所述待处理图像进行去重影处理，以得到所述待处理图像对应的输出图像。
  34. 根据权利要求33所述的图像处理方法，其特征在于，所述图像处理模型包括编码器和解码器；所述通过所述图像处理模型对所述待处理图像进行去重影，以得到所述待处理图像对应的输出图像具体包括：
    将所述待处理图像输入所述编码器，通过所述编码器得到所述待处理图像的特征图像，其中，所述特征图像的图像尺寸小于所述待处理图像的图像尺寸；
    将所述特征图像输入所述解码器，通过所述解码器输出所述待处理图像对应的输出图像，其中，所述输出图像的图像尺寸等于所述待处理图像的图像尺寸。
  35. 根据权利要求33或34所述的图像处理方法，其特征在于，所述通过所述图像处理模型对所述待处理图像进行去重影处理，以得到所待处理图像对应的输出图像之后还包括：
    对所述输出图像进行锐化以及降噪处理，并将锐化以及降噪处理后的输出图像作为所述待处理图像对应的输出图像。
  36. 一种图像处理方法，其中，所述方法包括：
    获取待处理图像集，其中，所述待处理图像集包括多张图像；
    根据所述待处理图像集，生成所述待处理图像集对应的去噪图像；
    将所述去噪图像输入至以已训练的第一图像处理模型，通过所述图像处理模型对所述去噪图像进行去偏色处理，得到所述去噪图像对应的处理后图像；
    将所述处理后图像输入至以已训练的第二图像处理模型，通过所述第二图像处理模型对所述处理后图像进行去重影处理，以得到输出图像。
  37. 根据权利要求36所述图像处理方法，其中，所述待处理图像集包括的多张图像中一张图像为基础图像，其余图像为基础图像的临近图像,所述根据所述待处理图像集，生成所述待处理图像集对应的去噪图像具体包括：
    将所述基础图像划分为若干基础图像块，分别确定各基础图像在各临近图像中对应的临近图像块；
    确定各个基础图像块分别对应的权重参数集；其中，基础图像块对应的权重参数集包括第一权重参数和第二权重参数，第一权重参数为基础图像块的权重参数，第二权重参数为临近图像中与基础图像块对应的临近图像块的权重参数；
    根据所述待处理图像集以及各个基础图像块分别对应的权重参数集，确定去噪图像。
  38. 根据权利要求37所述图像处理方法，其中，所述待处理图像集的图像数量为根据所述待处理图像集对应的拍摄参数确定的。
  39. 根据权利要求37所述图像处理方法，其中，所述基础图像的图像清晰度大于或等于所述临近图像的图像清晰度。
  40. 根据权利要求37所述图像处理方法，其中，所述确定各个基础图像块分别对应的权重参数集具体包括：
    针对每个基础图像块，确定该基础图像块对应的各临近图像块的第二权重参数，以及，获取该基础图像块对应的第一权重参数，以得到该基础图像块对应的权重参数集。
  41. 根据权利要求40所述图像处理方法，其中，所述确定该基础图像块对应的各临近图像块的第二权重参数具体包括：
    针对每个临近图像块，计算该基础图像块与该临近图像块的相似度值；
    根据所述相似度值计算该临近图像块的第二权重参数。
  42. 根据权利要求41所述图像处理方法，其中，所述根据所述相似度值计算该临近图像块的第二权重参数具体包括：
    当所述相似度值小于或等于第一阈值时，将第一预设参数作为该临近图像块的第二权重参数；
    当所述相似度值大于第一阈值，且小于或等于第二阈值时，根据所述相似度值、所述第一阈值及所述第二阈值计算该临近图像块的第二权重参数；
    当所述相似度值大于第二阈值时，将预设第二预设参数作为该临近图像块的第二权重参数。
  43. 根据权利要求42所述图像处理方法，其中，所述第一阈值和第二阈值均为根据该基础图像块对应临近图像块的相似度值确定的。
  44. 根据权利要求36所述图像处理方法，其中，所述根据所述待处理图像集以及各个基础图像块分别对应的权重参数集，确定去噪图像之后还包括：
    对所述去噪图像进行空域降噪，并将空域降噪后得到的图像作为去噪图像。
  45. 根据权利要求36-44任意一项所述的图像处理方法，其中，所述第一图像处理模型为应用如权利要求1-13任意一项所述的图像处理模型的生成方法生成的图像处理模型。
  46. 根据权利要求36-44任意一项所述的图像处理方法，其中，所述第二图像处理模为应用如权利要求24-32任意一项所述的图像处理模型的生成方法训练得到的图像处理模型。
  47. 一种图像处理模型的生成装置，其中，所述图像处理模型的生成装置包括：
    第一生成模块，用于利用预设网络模型根据训练图像集中的第一图像，生成所述第一图像对应的生成图像，其中，所述训练图像集包括多组训练图像组，每一组训练图像组包括第一图像和第二图像，其中，第一图像为对应第二图像的偏色图像；
    第一修正模块，用于利用预设网络模型根据所述第一图像对应的第二图像和所述第一图像对应的生成图像，对模型参数进行修正，并继续执行根据所述训练图像集中的下一训练图像组中的第一图像，生成所述第一图像对应的生成图像的步骤，直至所述预设网络模型的训练情况满足预设条件，以得到所述图像处理模型。
  48. 一种图像处理装置，其中，应用如权利要求47所述的图像处理模型的生成装置生成的图像处理模型，所述图像处理装置包括：
    第一获取模块，用于获取待处理图像，并将所述待处理图像输入至所述图像处理模型；
    第一处理模块，用于通过所述图像处理模型对所述待处理图像进行偏色处理，以得到所述待处理图像对应的处理后的图像。
  49. 一种图像处理模型的生成装置，其中，所述图像处理模型的生成装置包括：
    第二生成模块，用于利用预设网络模型根据训练图像集中的第一图像生成所述第一图像对应的生成图像；其中，所述训练图像集包括多组训练图像组，每一组训练图像组包括第一图像和第二图像，第一图像为第二图像对应的具有重影的图像；
    第二修正模块，用于利用所述预设网络模型根据所述第一图像对应的第二图像和 所述第一图像对应的生成图像，对所述预设网络模型的模型参数进行修正，并继续执行根据训练图像集中的下一训练图像组中的第一图像，生成所述第一图像对应的生成图像的步骤，直至所述预设网络模型的训练情况满足预设条件，以得到已训练的图像处理模型。
  50. 一种图像处理装置，其中，应用如权利要求49所述的图像处理模型的生成装置生成的图像处理模型，所述图像处理装置包括：
    第二获取模块，用于获取待处理图像，并将所述待处理图像输入至所述图像处理模型；
    第二处理模块，用于通过所述图像处理模型对所述待处理图像进行去重影处理，以得到所述待处理图像对应的输出图像。
  51. 一种图像处理装置，其中，所述图像处理装置包括：
    第三获取模块，用于获取待处理图像集，其中，所述待处理图像集包括多张图像；
    第三生成模块，用于根据所述待处理图像集，生成所述待处理图像集对应的去噪图像；
    第三处理模块，用于将所述去噪图像输入至以已训练的第一图像处理模型，通过所述图像处理模型对所述去噪图像进行去偏色处理，得到所述去噪图像对应的处理后图像；
    第四处理模块，用于将所述处理后图像输入至以已训练的第二图像处理模型，通过所述第二图像处理模型对所述处理后图像进行去重影处理，以得到输出图像。
  52. 一种计算机可读存储介质，其中，所述计算机可读存储介质存储有一个或者多个程序，所述一个或者多个程序可被一个或者多个处理器执行，以实现如权利要求1～13任意一项所述的图像处理模型的生成方法，如权利要求14～23任意一项所述的图像处理方法，如权利要求24～32任意一项所述的图像处理模型的生成方法、如权利要求33～35任意一项所述的图像处理方法和/或如权利要求36～46任意一项所述的图像处理方法中的步骤。
  53. 一种终端，其中，包括：处理器和存储器；所述存储器上存储有可被所述处理器执行的计算机可读程序；所述处理器执行所述计算机可读程序时实现如权利要求1～13任意一项所述的图像处理模型的生成方法，如权利要求14～23任意一项所述的图像处理方法，如权利要求24～32任意一项所述的图像处理模型的生成方法、如权利要求33～35任意一项所述的图像处理方法和/或如权利要求36～46任意一项所述的图像处理方法中的步骤。
PCT/CN2020/141932 2020-03-10 2020-12-31 图像处理模型生成方法、处理方法、存储介质及终端 WO2021179764A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/865,340 US20220398698A1 (en) 2020-03-10 2022-07-14 Image processing model generation method, processing method, storage medium, and terminal

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
CN202010163472.4A CN113379611B (zh) 2020-03-10 2020-03-10 图像处理模型的生成方法、处理方法、存储介质及终端
CN202010163472.4 2020-03-10
CN202010162684.0A CN113379608A (zh) 2020-03-10 2020-03-10 一种图像处理方法、存储介质及终端设备
CN202010162709.7 2020-03-10
CN202010162709.7A CN113379610B (zh) 2020-03-10 2020-03-10 图像处理模型的训练方法、图像处理方法、介质及终端
CN202010162684.0 2020-03-10

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/865,340 Continuation US20220398698A1 (en) 2020-03-10 2022-07-14 Image processing model generation method, processing method, storage medium, and terminal

Publications (1)

Publication Number Publication Date
WO2021179764A1 true WO2021179764A1 (zh) 2021-09-16

Family

ID=77670441

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/141932 WO2021179764A1 (zh) 2020-03-10 2020-12-31 图像处理模型生成方法、处理方法、存储介质及终端

Country Status (2)

Country Link
US (1) US20220398698A1 (zh)
WO (1) WO2021179764A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024073203A1 (en) * 2022-09-26 2024-04-04 Sony Interactive Entertainment Inc. Using vector graphics to create 3d content

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114650361B (zh) * 2020-12-17 2023-06-06 北京字节跳动网络技术有限公司 拍摄模式确定方法、装置、电子设备和存储介质
CN116777796A (zh) * 2023-08-22 2023-09-19 深圳市小铭工业互联网有限公司 全自动摄像精准pcba电路板物料检测系统

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110097512A (zh) * 2019-04-16 2019-08-06 四川大学 基于Wasserstein生成对抗网络的三维MRI图像去噪模型的构建方法及应用
CN110223254A (zh) * 2019-06-10 2019-09-10 大连民族大学 一种基于对抗生成网络的图像去噪方法
CN110473154A (zh) * 2019-07-31 2019-11-19 西安理工大学 一种基于生成对抗网络的图像去噪方法
CN110602467A (zh) * 2019-09-09 2019-12-20 Oppo广东移动通信有限公司 图像降噪方法、装置、存储介质及电子设备

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110097512A (zh) * 2019-04-16 2019-08-06 四川大学 基于Wasserstein生成对抗网络的三维MRI图像去噪模型的构建方法及应用
CN110223254A (zh) * 2019-06-10 2019-09-10 大连民族大学 一种基于对抗生成网络的图像去噪方法
CN110473154A (zh) * 2019-07-31 2019-11-19 西安理工大学 一种基于生成对抗网络的图像去噪方法
CN110602467A (zh) * 2019-09-09 2019-12-20 Oppo广东移动通信有限公司 图像降噪方法、装置、存储介质及电子设备

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
MICHAËL GHARBI: "Deep Bilateral Learning for Real-Time Image Enhancement", 22 August 2017 (2017-08-22), pages 1 - 12, XP055533360, Retrieved from the Internet <URL:https://arxiv.org/pdf/1707.02880.pdf> DOI: 10.1145/3072959.3073592 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024073203A1 (en) * 2022-09-26 2024-04-04 Sony Interactive Entertainment Inc. Using vector graphics to create 3d content

Also Published As

Publication number Publication date
US20220398698A1 (en) 2022-12-15

Similar Documents

Publication Publication Date Title
US11882357B2 (en) Image display method and device
US11182877B2 (en) Techniques for controlled generation of training data for machine learning enabled image enhancement
WO2021179764A1 (zh) 图像处理模型生成方法、处理方法、存储介质及终端
US11037278B2 (en) Systems and methods for transforming raw sensor data captured in low-light conditions to well-exposed images using neural network architectures
CN111353948B (zh) 一种图像降噪方法、装置及设备
US20200068151A1 (en) Systems and methods for processing low light images
WO2021164234A1 (zh) 图像处理方法以及图像处理装置
CN113888437A (zh) 图像处理方法、装置、电子设备及计算机可读存储介质
CN113379609B (zh) 一种图像处理方法、存储介质及终端设备
Liu et al. Survey of natural image enhancement techniques: Classification, evaluation, challenges, and perspectives
KR102262671B1 (ko) 비디오 영상에 보케 효과를 적용하는 방법 및 기록매체
Liang et al. Deep bilateral retinex for low-light image enhancement
CN112651911A (zh) 一种基于偏振图像的高动态范围成像生成方法
CN114862698A (zh) 一种基于通道引导的真实过曝光图像校正方法与装置
CN114298942A (zh) 图像去模糊方法及装置、计算机可读介质和电子设备
US20210327026A1 (en) Methods and apparatus for blending unknown pixels in overlapping images
CN113379608A (zh) 一种图像处理方法、存储介质及终端设备
CN113658197B (zh) 图像处理方法、装置、电子设备及计算机可读存储介质
CN114511487A (zh) 图像融合方法及装置、计算机可读存储介质、终端
KR20220151713A (ko) 콘텐츠 기반 이미지 프로세싱
CN113379611B (zh) 图像处理模型的生成方法、处理方法、存储介质及终端
CN113379610A (zh) 图像处理模型的训练方法、图像处理方法、介质及终端
CN113379753B (zh) 一种图像处理方法、存储介质及终端设备
Liu et al. Learning to recover lost details from the dark
Jung et al. Deep low-contrast image enhancement using structure tensor representation

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20923862

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20923862

Country of ref document: EP

Kind code of ref document: A1