CN111988593B - Three-dimensional image color correction method and system based on depth residual optimization - Google Patents

Three-dimensional image color correction method and system based on depth residual optimization Download PDF

Info

Publication number
CN111988593B
CN111988593B CN202010893423.6A CN202010893423A CN111988593B CN 111988593 B CN111988593 B CN 111988593B CN 202010893423 A CN202010893423 A CN 202010893423A CN 111988593 B CN111988593 B CN 111988593B
Authority
CN
China
Prior art keywords
image
model
target image
stereo
map
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010893423.6A
Other languages
Chinese (zh)
Other versions
CN111988593A (en
Inventor
陈羽中
林冠妙
范媛媛
牛玉贞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fuzhou University
Original Assignee
Fuzhou University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fuzhou University filed Critical Fuzhou University
Priority to CN202010893423.6A priority Critical patent/CN111988593B/en
Publication of CN111988593A publication Critical patent/CN111988593A/en
Application granted granted Critical
Publication of CN111988593B publication Critical patent/CN111988593B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/64—Circuits for processing colour signals
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/22—Matching criteria, e.g. proximity measures
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/64—Circuits for processing colour signals
    • H04N9/646—Circuits for processing colour signals for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters

Landscapes

  • Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Theoretical Computer Science (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Artificial Intelligence (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Image Processing (AREA)

Abstract

The invention relates to a stereoscopic image color correction method based on depth residual optimization, which comprises the following steps: s1: carrying out color distortion processing on a left view or a right view of an undistorted stereo image in a data set to generate a distorted stereo image, and establishing a training set comprising the undistorted stereo image and the distorted stereo image; s2: calculating disparity maps of all distorted stereo images by using a stereo matching model, and generating an initialized matching image by using an image deformation technology; s3: constructing a color correction residual error map optimization model based on a neural network, taking a residual error map as the input of the model, and designing a loss function; s4: using a loss function training model, and obtaining a trained model by minimizing the optimal parameters of the loss function learning model; s5: and carrying out color correction on the distorted stereo image to be corrected by using the trained model. The method and the system are beneficial to improving the color consistency of the color correction result and the reference image and keeping the structural consistency with the target image.

Description

Three-dimensional image color correction method and system based on depth residual optimization
Technical Field
The invention relates to the field of image processing and computer vision, in particular to a method and a system for correcting colors of a three-dimensional image based on depth residual error optimization.
Background
With the continuous development of multimedia technology and computer vision, the visual effect brought by stereoscopic vision draws extensive attention and develops rapidly. Compared with the planar sense of the traditional two-dimensional image, the stereoscopic image brings more real and natural visual experience to people. In the case of some specific research, such as stereoscopic vision, panoramic image stitching, medical image analysis, and remote sensing image fusion, it is desirable that the image sequence obtained from the image acquisition device has good color consistency. However, due to the influence of factors such as image acquisition equipment, light source distribution change and diffuse reflection on the surface of an object, the brightness and color of the same object photographed by the camera at different viewpoints may be greatly different. The human visual system has color constancy, can eliminate the influence of various color differences to a certain extent, and correctly perceive the inherent color of an object. When the computer is used for image analysis, the difference not only affects post-production related to color consistency, but also affects depth information reconstruction in binocular stereo vision, and further makes audiences feel visual fatigue. The computer system also needs to have the ability to handle such differences between images that would otherwise affect the effectiveness of subsequent processing.
The application field of the image color correction algorithm is quite wide, and the image color correction algorithm is widely applied to the fields of color correction between three-dimensional videos/multi-view video views, panoramic image splicing, remote sensing image fusion and the like. In image stitching/fusion (including remote sensing images, panoramic images, multi-view videos and the like) applications, color correction ensures that images obtained from adjacent shooting visual angles do not have obvious stitching traces near stitching edges. In the color correction of the left and right views of the three-dimensional image, the color correction algorithm of the three-dimensional image enables the colors of the corresponding contents of the left and right eye views of the three-dimensional media to be consistent, reduces the color difference of the left and right views, enhances the reliability of depth information, reduces the burden of a visual system of a viewer and relieves visual fatigue. Although many color correction algorithms have been proposed, the existing stereo image color correction algorithms are all traditional methods based on calculation, and the existing image color correction algorithms still have many limitations. The conventional color correction algorithm may be classified into a global color correction algorithm and a local color correction algorithm, which are classified according to the number of mapping functions.
The global color correction algorithm uses the same mapping function for all pixels in the image, for example, a global color migration method first proposed by Reinhard performs color conversion on the target image channel by using the standard deviation and average value of each channel in a Lab color space without correlation. Xiao et al believe that the conversion of Lab space to RGB space introduces additional time overhead, and to eliminate this conversion process they implement global color migration using covariance matrix conversion in RGB color space. Yao et al propose a gradient preserving color migration algorithm based on the histogram, minimize both the histogram error and the gradient error by establishing a Laplacian pyramid, so as to achieve the purpose of mapping the color of the reference image to the source image while preserving the gradient of the source image. Although the global image color correction algorithm is fast, the global image color correction algorithm only utilizes limited statistical information, is not suitable for performing color correction on colorful images and has poor processing capability on local color differences appearing in the images. The local color correction algorithm uses different mapping functions for different regions in the image. For example, the primary region mapping method proposed by Zhang et al first performs overlapping region matching on an image by using a registration algorithm, finds out a matched primary region in the overlapping region through color histogram peak value pairing in HSV color space, and finally calculates an independent mapping function in each primary region. The matching and optimization-based color correction algorithm proposed by Zheng first initializes color values in the resulting image using the dense stereo matching map and the global color correction result, and then treats the color correction as a quadratic energy minimization problem to improve the local color smoothness and global color consistency of the result. When the left-right view parallax is large, the initial composite image and the target image have large structural deformation which is difficult to optimize. Although the local image color correction algorithm improves the correction capability of local color difference to a certain extent, and partial algorithms can obtain good correction results, the processing time is long due to the introduction of matching, segmentation and subsequent optimization, and the method is not beneficial to practical application.
Disclosure of Invention
The invention aims to provide a method and a system for correcting colors of a three-dimensional image based on depth residual optimization, which are beneficial to improving the color consistency of a color correction result and a reference image and keeping the structural consistency with a target image.
In order to realize the purpose, the invention adopts the technical scheme that: a stereoscopic image color correction method based on depth residual optimization comprises the following steps:
step S1: performing color distortion processing on a left view or a right view of an undistorted stereo image in a data set to generate a distorted stereo image with color difference, wherein the view subjected to the color distortion processing in the distorted stereo image is a target image, the other view is a reference image, and a training set comprising the undistorted stereo image and the distorted stereo image is established;
step S2: calculating disparity maps of all distorted stereo images in a training set by using a pre-trained stereo matching model, and then generating an initialized matching image by using an image deformation technology according to the disparity maps;
step S3: constructing a color correction residual image optimization model based on a neural network, taking a residual image obtained by calculation based on each initialized matching image and a corresponding target image as the input of the model, and designing a loss function suitable for color correction;
step S4: using the loss function training model to obtain a trained color correction residual error image optimization model by minimizing the optimal parameters of the loss function learning model;
step S5: and carrying out color correction on the distorted stereo image to be corrected by using the trained model.
Further, in step S1, performing color distortion processing on the left or right view of the undistorted stereo image in the data set to generate a distorted stereo image with color difference, and establishing a training set including the undistorted stereo image and the distorted stereo image, includes the following steps:
step S11: taking a left view of the undistorted stereo image as a reference image, and taking a right view as an ideal target image; carrying out multi-color distortion processing on the right view of each undistorted stereo image in the data set to obtain a plurality of target images, wherein each target image and a corresponding reference image form a reference-target image pair, and each reference image-target image pair forms a distorted stereo image, so that each undistorted stereo image obtains a plurality of distorted stereo images; all undistorted stereo images and distorted stereo images form a training set;
step S12: and (2) carrying out consistent size adjustment and clipping on the undistorted stereo images and the distorted stereo images in the training set, namely, the size adjustment and clipping operation of each distorted stereo image and the undistorted stereo image corresponding to the distorted stereo image are the same, so as to obtain more new undistorted stereo images and distorted stereo images, storing each distorted stereo image and the undistorted stereo image corresponding to the distorted stereo image according to the form of a reference image-target image-ideal target image pair, so that each undistorted stereo image in the step S11 obtains a plurality of reference image-target image-ideal target image pairs, and forming a new training set.
Further, in step S2, the method includes the following steps of calculating disparity maps of all distorted stereo images in the training set by using the pre-trained stereo matching model, and then generating an initialized matching image by using an image warping technique according to the disparity maps:
step S21: calculating a disparity map between a reference image and a target image in all distorted stereo images in a training set by using a pre-trained PSmNet stereo matching model; synthesizing a virtual viewpoint disparity map by using the disparity map, wherein the disparity map is set to be DrThe virtual viewpoint disparity map is DvThe synthesis process of the virtual viewpoint disparity map is represented as:
Figure BDA0002657633710000031
wherein, (x, y) represents the position of the pixel point on the image, d represents the parallax value, dr(x, y) denotes a parallax map DrThe disparity value of the pixel at (x, y) in (m), w represents the width of the image;
step S22: filling holes in cracks of the virtual viewpoint disparity map, determining positions of pixel points in the reference image corresponding to the target image according to the filled virtual viewpoint disparity map, determining color values of the initialized matched image by using linear interpolation, and initializing the matched image ImThe calculation formula of (a) is as follows:
Im(x,y)＝αIr(x,index(|y+dv(x,y)+1|,w))+(1-α)Ir(x,index(|y+dv(x,y)|,w))
wherein, IrFor reference picture, dv(x, y) represents the filled virtual viewpoint disparity map DvThe disparity value of the pixel at (x, y); the calculation process of α and index is as follows:
α＝y+dv(x,y)-|y+dv(x,y)|
Figure BDA0002657633710000041
Figure BDA0002657633710000042
step S23: calculating an initialization matching image by using each reference image-target image-ideal target image pair in the training set, forming a reference image-target image-ideal target image-initialization matching image pair, and then normalizing the images in all the reference image-target image-ideal target image-initialization matching image pairs to obtain a normalized reference image-target image-ideal target image-initialization matching image pair, wherein the normalized ideal target image is a training label, and other normalized images are input of a training model.
Further, in step S3, constructing a color correction residual map optimization model based on a neural network, taking a residual map calculated based on each initialized matching image and the corresponding target image as an input of the model, and designing a loss function suitable for color correction, includes the following steps:
step S31: constructing a color correction residual map optimization model based on a neural network, wherein the network structure adopts a basic structure of an enhanced deep super-resolution network (EDSR), the model adopts a residual training mode, and the input of the model is an initialized matching image ImWith the target image ItResidual error map R ofmI.e. the initial residual map, the calculation formula is as follows:
Figure BDA0002657633710000043
step S32: designing a loss function suitable for a color correction residual image optimization model, wherein the loss function consists of a perception loss, a structural loss and a pixel-by-pixel loss; the perception loss is calculated by using a feature map obtained by 5 activation layers of a pre-trained VGG-19 model; respectively inputting a correction result graph obtained by training a color correction residual image optimization model and an ideal target image into a pre-trained VGG-19 model, and taking out a feature graph corresponding to an activation layer to perform 1-norm distance measurement; the perceptual loss is calculated as follows:
Figure BDA0002657633710000044
wherein, | represents an absolute value, phi represents a pre-trained VGG-19 model, Cj、HjAnd WjRespectively representing the number of channels, the height and the width of a jth feature map, respectively, z, x and y representing the number of channels, the height and the width of the feature map, (z, x and y) representing the position on the feature map, i.e. a point with coordinates (x and y) in the jth channel, and IgtIs a right view of an ideal object image, i.e. a distortion-free stereo image, IresultIs a graph of the result of the correction,
Figure BDA0002657633710000051
a characteristic diagram of the j activation layer of the image I in the VGG-19 model is shown, the j values are from 1 to 5, the j values correspond to the 1 to 5 activation layers in the VGG-19 model,
Figure BDA0002657633710000052
and
Figure BDA0002657633710000053
respectively representing the values of the feature maps corresponding to the ideal target image and the correction result map at (z, x, y);
the structural loss adopts an SSIM structural similarity index, the SSIM index respectively calculates the brightness similarity, the contrast similarity and the structural similarity of the two images, and multiplies the brightness similarity, the contrast similarity and the structural similarity to obtain the similarity index of the two images, and the structural loss calculation formula is as follows:
Figure BDA0002657633710000054
Figure BDA0002657633710000055
wherein, muIWhich represents the average value of the image I,
Figure BDA0002657633710000056
which represents the variance of the image I and,
Figure BDA0002657633710000057
is the average value of the ideal target image,
Figure BDA0002657633710000058
in order to correct the average value of the result graph,
Figure BDA0002657633710000059
the variance of the ideal target image is represented,
Figure BDA00026576337100000510
a variance of the correction result graph is represented,
Figure BDA00026576337100000511
is the covariance of the ideal target image and the corrected result image; c1And C2Is a constant for maintaining stability;
the pixel-by-pixel loss is the L1 loss between the correction result map and the ideal target image and target image residual map, and the formula is as follows:
Figure BDA00026576337100000512
wherein W and H represent the width and height of the image, respectively, and (x, y) represents the pixel in the figureCoordinates on the image, RresultIs a correction result chart IresultWith the target image ItI.e. corrected residual map, RgtIs an ideal target image IgtWith the target image ItThe residual map of the two images is normalized by calculating the difference value of the two images, and the residual map R is correctedresultAnd an ideal residual map RgtThe calculation formula of (a) is as follows:
Figure BDA0002657633710000061
Figure BDA0002657633710000062
the final loss function is a weighted sum of the perceptual, structural and pixel-wise losses:
Figure BDA0002657633710000063
wherein λ is1、λ2、λ3Representing the weight of the perceptual, structural and pixel-wise losses, respectively.
Further, in step S4, the method for obtaining a trained color correction residual error map optimization model by using the loss function training model and minimizing the optimal parameters of the loss function learning model includes the following steps:
step S41: calculating each initialized matching image ImWith the corresponding target image ItResidual error map R ofm；
Step S42: the residual error map RmThe output of the network is the resulting residual map R as input to the modelresultAnd carrying out reverse normalization on the result residual image and adding the target image to obtain a corrected result image, wherein the calculation formula is as follows:
Iresult＝Rresult×2-1+It
step S43: and calculating a loss function according to the loss function formula of the step S32 and carrying out back propagation, wherein the network minimizes the loss function through multiple iterations, the training set is divided into multiple batches for batch optimization in each iteration, and the batch optimization learning rate of each parameter is adaptively controlled by adopting an ADAM method based on gradient variance.
Further, in step S5, performing color correction on the distorted stereoscopic image to be corrected by using the trained model, includes the following steps:
step S51: calculating by using a pre-trained stereo matching model to obtain a disparity map of a distorted stereo image to be corrected, and generating a corresponding initialized matching image by using an image deformation technology according to the disparity map;
step S52: and calculating to obtain a color correction result image by taking the obtained residual image of the initialized matching image and the corresponding target image as the input of the model.
The invention also provides a stereoscopic image color correction system based on depth residual optimization, comprising a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein when the computer program is run by the processor, the method steps as described above are implemented.
Compared with the prior art, the invention has the following beneficial effects: the invention is suitable for the color correction of the stereo images with different color difference types and different distortion degrees. The invention trains the network model by adopting a residual error training mode, which can ensure that the definition of the target image is kept to the maximum extent by the correction result and improve the structural consistency of the correction result and the target image. In the training of the optimization network, in order to improve structural deformation and local color inconsistency existing in the initialization matching graph, pixel-by-pixel loss, structural loss based on SSIM and perceptual loss based on a VGG network are adopted. Due to the fact that the deep convolutional neural network model structure is used, compared with a secondary energy minimization optimization mode in a traditional calculation method, the calculation complexity is greatly reduced, the optimization efficiency is improved, and meanwhile the model can be used as a post-processing step of other three-dimensional image color correction methods.
Drawings
FIG. 1 is a flow chart of a method implementation of an embodiment of the present invention.
Fig. 2 is an overall framework diagram of the color correction residual map optimization model in the embodiment of the present invention.
Detailed Description
The invention is described in further detail below with reference to the figures and the embodiments.
As shown in fig. 1-2, the present invention provides a method for color correction of a stereoscopic image based on depth residual optimization, comprising the following steps:
step S1: and carrying out color distortion processing on the left or right view of the undistorted stereo image in the data set to generate a distorted stereo image with color difference, wherein the view subjected to the color distortion processing in the distorted stereo image is a target image, the other view is a reference image, and a training set comprising the undistorted stereo image and the distorted stereo image is established. The method specifically comprises the following steps:
step S11: without loss of generality, the left view of the undistorted stereo image is used as a reference image, and the right view is used as an ideal target image; carrying out multi-color distortion processing on the right view of each undistorted stereo image in the data set to obtain a plurality of target images, wherein each target image and a corresponding reference image form a reference-target image pair, and each reference image-target image pair forms a distorted stereo image, so that each undistorted stereo image obtains a plurality of distorted stereo images; all undistorted stereo images and distorted stereo images constitute a training set.
In this embodiment, the left view of the stereo image is used as the reference image for subsequent processing, but the generality of the processing method is not affected. In other embodiments of the present invention, it is fully possible to use the right view of the stereoscopic image as the reference image.
In this embodiment, six different color distortion processes are performed on the right view of the undistorted stereo image, including modifying exposure, brightness, contrast, RG channel, hue and saturation, each distortion process has 3 modification granularities, i.e. low, medium and high, so as to obtain a plurality of target images.
Step S12: and (2) carrying out consistent size adjustment and clipping on the undistorted stereo images and the distorted stereo images in the training set, namely, the size adjustment and clipping operation of each distorted stereo image and the undistorted stereo image corresponding to the distorted stereo image are the same, so as to obtain more new undistorted stereo images and distorted stereo images, storing each distorted stereo image and the undistorted stereo image corresponding to the distorted stereo image according to the form of a reference image-target image-ideal target image pair, so that each undistorted stereo image in the step S11 obtains a plurality of reference image-target image-ideal target image pairs, and forming a new training set. The specific method comprises the following steps: the short side of each view is zoomed to 360 pixels, the long side is zoomed in a corresponding equal ratio, the zoomed stereo image is cut into a plurality of image blocks with the size of 256x256 by random number, and a pair of undistorted stereo image and the left view and the right view of the distorted stereo image are cut in a unified mode to obtain a new pair of undistorted stereo image and distorted stereo image, so that the diversity of a training data set is increased. The size of each stereo image in the training set is 256x 256.
Step S2: and calculating disparity maps of all distorted stereo images in a training set by using the pre-trained stereo matching model, and then generating an initialized matching image by using an image deformation technology according to the disparity maps. The method specifically comprises the following steps:
step S21: calculating a disparity map between a reference image and a target image in all distorted stereo images in a training set by using a pre-trained PSmNet stereo matching model; synthesizing a virtual viewpoint disparity map by using the disparity map, wherein the disparity map is set to be DrThe virtual viewpoint disparity map is DvThe synthesis process of the virtual viewpoint disparity map is represented as:
Figure BDA0002657633710000081
wherein, (x, y) represents the position of the pixel point on the image, d represents the parallax value, dr(x, y) denotes a disparity map DrOf the pixel in (x, y)The disparity value, w, represents the width of the image.
Step S22: filling holes in cracks of the virtual viewpoint disparity map, determining positions of pixel points in the reference image corresponding to the target image according to the filled virtual viewpoint disparity map, determining color values of the initialized matched image by using linear interpolation, and initializing the matched image ImThe calculation formula of (a) is as follows:
Im(x,y)＝αIr(x,index(|y+dv(x,y)+1|,w))+(1-α)Ir(x,index(|y+dv(x,y)|,w))
wherein, IrFor reference picture, dv(x, y) represents the filled virtual viewpoint disparity map DvThe disparity value of the pixel at (x, y) in (m); the calculation process of α and index is as follows:
α＝y+dv(x,y)-|y+dv(x,y)|
Figure BDA0002657633710000082
Figure BDA0002657633710000083
step S23: calculating an initialization matching image by using each reference image-target image-ideal target image pair in the training set, forming a reference image-target image-ideal target image-initialization matching image pair, and then normalizing the images in all the reference image-target image-ideal target image-initialization matching image pairs to obtain a normalized reference image-target image-ideal target image-initialization matching image pair, wherein the normalized ideal target image is a training label, and other normalized images are input of a training model. The normalization process divides each pixel value of the image by 255 to bring the pixel values of the image between 0, 1.
Step S3: and constructing a color correction residual image optimization model based on a neural network, taking a residual image obtained by calculation based on each initialized matching image and the corresponding target image as the input of the model, and designing a loss function suitable for color correction. The method specifically comprises the following steps:
step S31: constructing a color correction residual map optimization model based on a neural network, wherein the network structure adopts a basic structure of an enhanced deep super-resolution network (EDSR), the model adopts a residual training mode, and the input of the model is an initialized matching image ImWith the target image ItResidual map R of (2)mI.e. the initial residual map, the calculation formula is as follows:
Figure BDA0002657633710000091
step S32: designing a loss function suitable for a color correction residual image optimization model, wherein the loss function is composed of a perception loss, a structural loss and a pixel-by-pixel loss; the perception loss is calculated by adopting a feature map obtained by 5 activation layers of a pre-trained VGG-19 model; respectively inputting a correction result graph obtained by training a color correction residual image optimization model and an ideal target image into a pre-trained VGG-19 model, and taking out a feature graph corresponding to an activation layer to perform 1-norm distance measurement; the perceptual loss is calculated as follows:
Figure BDA0002657633710000092
wherein, | represents an absolute value, phi represents a pre-trained VGG-19 model, Cj、HjAnd WjRespectively representing the number of channels, the height and the width of a jth feature map, respectively, z, x and y representing the number of channels, the height and the width of the feature map, (z, x and y) representing the position on the feature map, i.e. a point with coordinates (x and y) in the jth channel, and IgtIs an ideal target image (i.e. the right view of an undistorted stereo image), IresultIs a graph of the result of the correction,
Figure BDA0002657633710000093
representing the characteristic diagram of the j activation layer of the image I in the VGG-19 model, wherein j takes values from 1 to 5 and corresponds to the VGG-19 modelThe 1 st to 5 th active layers in the type,
Figure BDA0002657633710000094
and
Figure BDA0002657633710000095
respectively representing the values of the feature maps corresponding to the ideal target image and the correction result map at (z, x, y);
the structural loss adopts an SSIM structural similarity index, the SSIM index respectively calculates the brightness similarity, the contrast similarity and the structural similarity of the two images, and multiplies the brightness similarity, the contrast similarity and the structural similarity to obtain the similarity index of the two images, and the structural loss calculation formula is as follows:
Figure BDA0002657633710000101
Figure BDA0002657633710000102
wherein, muIWhich represents the average value of the image I,
Figure BDA0002657633710000103
which represents the variance of the image I and,
Figure BDA0002657633710000104
is the average value of the ideal target image,
Figure BDA0002657633710000105
in order to correct the average value of the result graph,
Figure BDA0002657633710000106
represents the variance of the ideal target image and,
Figure BDA0002657633710000107
a variance of the correction result map is represented,
Figure BDA0002657633710000108
is the covariance of the ideal target image and the corrected result image; c1And C2Is a constant for maintaining stability;
the pixel-by-pixel loss is the L1 loss between the correction result map and the ideal target image and target image residual map, and the formula is as follows:
Figure BDA0002657633710000109
where W and H represent the width and height of the image, respectively, (x, y) represent the coordinates of the pixel on the image, RresultIs a correction result chart IresultWith the target image ItR (i.e. corrected residual map), RgtIs an ideal target image IgtWith the target image ItThe residual image of the two images is normalized after calculating the difference value of the two images, and the residual image R is correctedresultAnd an ideal residual map RgtThe calculation formula of (a) is as follows:
Figure BDA00026576337100001010
Figure BDA00026576337100001011
the final loss function is a weighted sum of the perceptual, structural and pixel-wise losses:
Figure BDA00026576337100001012
wherein λ is1、λ2、λ3Representing the weight of the perceptual loss, the structural loss and the pixel-by-pixel loss, respectively.
Step S4: and using the loss function training model to obtain a trained color correction residual error image optimization model by minimizing the optimal parameters of the loss function learning model. The method specifically comprises the following steps:
step S41: calculating each initialization matching image ImWith the corresponding target image ItResidual error map R ofm；
Step S42: the residual error map RmAs input to the model, the output of the network is the resulting residual map RresultAnd carrying out reverse normalization on the result residual image and adding the target image to obtain a corrected result image, wherein the calculation formula is as follows:
Iresult＝Rresult×2-1+It
step S43: and calculating a loss function according to the loss function formula of the step S32 and carrying out back propagation, wherein the network minimizes the loss function through multiple iterations, the training set is divided into multiple batches for batch optimization in each iteration, and the batch optimization learning rate of each parameter is adaptively controlled by adopting an ADAM method based on gradient variance.
Step S5: the color correction is carried out on the distorted stereo image to be corrected by using the trained model, and the specific method comprises the following steps:
step S51: calculating by using a pre-trained stereo matching model to obtain a disparity map of a distorted stereo image to be corrected, and then generating a corresponding initialized matching image by using an image deformation technology according to the disparity map;
step S52: and calculating to obtain a color correction result image by taking the obtained residual image of the initialized matching image and the corresponding target image as the input of the model.
The invention also provides a depth residual optimization-based stereoscopic image color correction system for implementing the above method, comprising a memory, a processor and a computer program stored on the memory and capable of running on the processor, which when run by the processor implements the method steps as described above.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The foregoing is directed to preferred embodiments of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow. However, any simple modification, equivalent change and modification of the above embodiments according to the technical essence of the present invention are within the protection scope of the technical solution of the present invention.

Claims (5)

1. A stereoscopic image color correction method based on depth residual optimization is characterized by comprising the following steps:
step S1: performing color distortion processing on a left view or a right view of an undistorted stereo image in a data set to generate a distorted stereo image with color difference, wherein the view subjected to the color distortion processing in the distorted stereo image is a target image, the other view is a reference image, and a training set comprising the undistorted stereo image and the distorted stereo image is established;
step S2: calculating disparity maps of all distorted stereo images in a training set by using a pre-trained stereo matching model, and then generating an initialized matching image by using an image deformation technology according to the disparity maps;
step S3: constructing a color correction residual image optimization model based on a neural network, taking a residual image obtained by calculation based on each initialized matching image and a corresponding target image as the input of the model, and designing a loss function suitable for color correction;
step S4: using the loss function training model to obtain a trained color correction residual error image optimization model by minimizing the optimal parameters of the loss function learning model;
step S5: carrying out color correction on the distorted three-dimensional image to be corrected by using the trained model;
in step S3, constructing a neural network-based color correction residual map optimization model, taking a residual map calculated based on each initialized matching image and the corresponding target image as an input of the model, and designing a loss function suitable for color correction, includes the following steps:
step S31: constructing a color correction residual error map optimization model based on a neural network, wherein the network structure adopts a basic structure of an enhanced deep super-resolution network (EDSR), and the model adopts a residual error training methodFormula (I), the input of the model is the initialization of the matching imagemWith the target image ItResidual error map R ofmI.e. the initial residual map, the calculation formula is as follows:
Figure FDA0003502994990000011
step S32: designing a loss function suitable for a color correction residual image optimization model, wherein the loss function is composed of a perception loss, a structural loss and a pixel-by-pixel loss; the perception loss is calculated by adopting a feature map obtained by 5 activation layers of a pre-trained VGG-19 model; respectively inputting a correction result graph obtained by training a color correction residual image optimization model and an ideal target image into a pre-trained VGG-19 model, and taking out a feature graph corresponding to an activation layer to perform 1-norm distance measurement; the perceptual loss is calculated as follows:
Figure FDA0003502994990000012
wherein, | represents an absolute value, phi represents a pre-trained VGG-19 model, Cj、HjAnd WjRespectively representing the number of channels, the height and the width of a jth feature map, respectively, z, x and y representing the number of channels, the height and the width of the feature map, (z, x and y) representing the position on the feature map, i.e. a point with coordinates (x and y) in the jth channel, and IgtIs a right view of an ideal object image, i.e. a distortion-free stereo image, IresultIs a graph of the result of the correction,
Figure FDA0003502994990000021
a characteristic diagram of the j activation layer of the image I in the VGG-19 model is shown, the value of j is from 1 to 5, the j corresponds to the 1 to 5 activation layers in the VGG-19 model,
Figure FDA0003502994990000022
and
Figure FDA0003502994990000023
respectively representing the values of the feature maps corresponding to the ideal target image and the correction result map at (z, x, y);
the structural loss adopts an SSIM structural similarity index, the SSIM index respectively calculates the brightness similarity, the contrast similarity and the structural similarity of the two images, and multiplies the brightness similarity, the contrast similarity and the structural similarity to obtain a similarity index of the two images, and a calculation formula of the structural loss is as follows:
Figure FDA0003502994990000024
Figure FDA0003502994990000025
wherein, muIWhich represents the average value of the image I,
Figure FDA0003502994990000026
which represents the variance of the image I and,
Figure FDA0003502994990000027
is the average value of the ideal target image,
Figure FDA0003502994990000028
in order to correct the average value of the result graph,
Figure FDA0003502994990000029
the variance of the ideal target image is represented,
Figure FDA00035029949900000210
a variance of the correction result graph is represented,
Figure FDA00035029949900000211
is the covariance of the ideal target image and the corrected result image; c1And C2Is used for maintaining stabilityA constant value is determined;
the pixel-by-pixel loss is the L1 loss between the correction result map and the ideal target image and target image residual map, and the formula is as follows:
Figure FDA00035029949900000212
where W and H represent the width and height of the image, respectively, (x, y) represent the coordinates of the pixel on the image, RresultIs a correction result chart IresultWith the target image ItI.e. corrected residual map, RgtIs an ideal target image IgtAnd a target image ItThe residual image of the two images is normalized after calculating the difference value of the two images, and the residual image R is correctedresultAnd an ideal residual map RgtThe calculation formula of (a) is as follows:
Figure FDA00035029949900000213
Figure FDA0003502994990000031
the final loss function is a weighted sum of the perceptual, structural and pixel-wise losses:
Figure FDA0003502994990000032
wherein λ is1、λ2、λ3Weights representing perceptual, structural and pixel-wise losses, respectively;
in step S4, the method for obtaining a trained color correction residual error map optimization model by using the loss function training model and minimizing the optimal parameters of the loss function learning model includes the following steps:
step S41: calculating each initialized matching image ImWith the corresponding target image ItResidual error map R ofm；
Step S42: the residual error map RmThe output of the network is the resulting residual map R as input to the modelresultAnd carrying out reverse normalization on the result residual image and adding the target image to obtain a corrected result image, wherein the calculation formula is as follows:
Iresult＝Rresult×2-1+It
step S43: and calculating a loss function according to the loss function formula of the step S32 and carrying out back propagation, wherein the network minimizes the loss function through multiple iterations, the training set is divided into multiple batches for batch optimization in each iteration, and the batch optimization learning rate of each parameter is adaptively controlled by adopting an ADAM method based on gradient variance.
2. The method for color correction of stereoscopic images based on depth residual optimization according to claim 1, wherein in step S1, the left or right view of undistorted stereoscopic images in the data set is color-distorted, so as to generate distorted stereoscopic images with color difference, and a training set including undistorted stereoscopic images and distorted stereoscopic images is established, comprising the following steps:
step S11: taking a left view of the undistorted stereo image as a reference image, and taking a right view as an ideal target image; carrying out multi-color distortion processing on the right view of each undistorted stereo image in the data set to obtain a plurality of target images, wherein each target image and a corresponding reference image form a reference-target image pair, and each reference image-target image pair forms a distorted stereo image, so that each undistorted stereo image obtains a plurality of distorted stereo images; all undistorted stereo images and distorted stereo images form a training set;
step S12: and (2) carrying out consistent size adjustment and clipping on the undistorted stereo images and the distorted stereo images in the training set, namely, the size adjustment and clipping operation of each distorted stereo image and the undistorted stereo image corresponding to the distorted stereo image are the same, so as to obtain more new undistorted stereo images and distorted stereo images, storing each distorted stereo image and the undistorted stereo image corresponding to the distorted stereo image according to the form of a reference image-target image-ideal target image pair, so that each undistorted stereo image in the step S11 obtains a plurality of reference image-target image-ideal target image pairs, and forming a new training set.
3. The method for color correction of stereo images based on depth residual optimization according to claim 2, wherein in step S2, the disparity maps of all distorted stereo images in the training set are calculated by using the pre-trained stereo matching model, and then the initialized matching image is generated by using the image deformation technique according to the disparity maps, comprising the following steps:
step S21: calculating a disparity map between a reference image and a target image in all distorted stereo images in a training set by using a pre-trained PSmNet stereo matching model; synthesizing a virtual viewpoint disparity map by using the disparity map, wherein the disparity map is set to be DrThe virtual viewpoint disparity map is DvThe synthesis process of the virtual viewpoint disparity map is represented as:
Figure FDA0003502994990000041
wherein, (x, y) represents the position of the pixel point on the image, d represents the parallax value, dr(x, y) denotes a parallax map DrThe disparity value of the pixel at (x, y) in (m), w represents the width of the image;
step S22: filling holes in cracks of the virtual viewpoint disparity map, determining positions of pixel points in the reference image corresponding to the target image according to the filled virtual viewpoint disparity map, determining color values of an initialization matching image by using linear interpolation, and initializing the matching image ImThe calculation formula of (a) is as follows:
Im(x,y)＝αIr(x,index(|y+dv(x,y)+1|,w))+(1-α)Ir(x,index(|y+dv(x,y)|,w))
wherein, IrFor reference picture, dv(x, y) represents the filled virtual viewpoint disparity map DvThe disparity value of the pixel at (x, y); the calculation process of α and index is as follows:
α＝y+dv(x,y)-|y+dv(x,y)|
Figure FDA0003502994990000042
Figure FDA0003502994990000043
step S23: calculating an initialization matching image by using each reference image-target image-ideal target image pair in the training set, forming a reference image-target image-ideal target image-initialization matching image pair, and then normalizing the images in all the reference image-target image-ideal target image-initialization matching image pairs to obtain a normalized reference image-target image-ideal target image-initialization matching image pair, wherein the normalized ideal target image is a training label, and other normalized images are input of a training model.
4. The method for color correction of stereoscopic images based on depth residual optimization according to claim 1, wherein in step S5, the color correction of the distorted stereoscopic images to be corrected is performed by using the trained model, and the method comprises the following steps:
step S51: calculating by using a pre-trained stereo matching model to obtain a disparity map of a distorted stereo image to be corrected, and then generating a corresponding initialized matching image by using an image deformation technology according to the disparity map;
step S52: and calculating to obtain a color correction result image by taking the obtained residual image of the initialized matching image and the corresponding target image as the input of the model.
5. A depth residual optimization based stereo image color correction system comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the method steps of any of claims 1-4 being carried out when the computer program is executed by the processor.
CN202010893423.6A 2020-08-31 2020-08-31 Three-dimensional image color correction method and system based on depth residual optimization Active CN111988593B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010893423.6A CN111988593B (en) 2020-08-31 2020-08-31 Three-dimensional image color correction method and system based on depth residual optimization

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010893423.6A CN111988593B (en) 2020-08-31 2020-08-31 Three-dimensional image color correction method and system based on depth residual optimization

Publications (2)

Publication Number Publication Date
CN111988593A CN111988593A (en) 2020-11-24
CN111988593B true CN111988593B (en) 2022-06-14

Family

ID=73440334

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010893423.6A Active CN111988593B (en) 2020-08-31 2020-08-31 Three-dimensional image color correction method and system based on depth residual optimization

Country Status (1)

Country Link
CN (1) CN111988593B (en)

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114554174B (en) * 2020-11-25 2024-04-02 京东方科技集团股份有限公司 Naked eye three-dimensional image measuring system, image processing method and device and display equipment
CN112884682B (en) * 2021-01-08 2023-02-21 福州大学 Stereo image color correction method and system based on matching and fusion
CN112802137B (en) * 2021-01-28 2022-06-21 四川大学 Color constancy method based on convolution self-encoder
CN113393360B (en) * 2021-06-08 2022-10-21 陕西科技大学 Correction method for printing and scanning resistant digital watermark image
CN113658038B (en) * 2021-08-26 2023-05-05 北京安德医智科技有限公司 Network training method and device based on space error self-adaption
CN114463196B (en) * 2021-12-28 2023-07-25 浙江大学嘉兴研究院 Image correction method based on deep learning

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102364953A (en) * 2011-11-08 2012-02-29 北京新岸线网络技术有限公司 Color correction method and device for stereo image
CN108462866A (en) * 2018-03-26 2018-08-28 福州大学 A kind of 3D stereo-picture color calibration methods based on matching and optimization
RU2698402C1 (en) * 2018-08-30 2019-08-26 Самсунг Электроникс Ко., Лтд. Method of training a convolutional neural network for image reconstruction and a system for forming an image depth map (versions)
CN111353964B (en) * 2020-02-26 2022-07-08 福州大学 Structure-consistent stereo image style migration method based on convolutional neural network
CN111311517B (en) * 2020-02-26 2022-07-08 福州大学 Color correction optimization method based on cutout

Also Published As

Publication number Publication date
CN111988593A (en) 2020-11-24

Similar Documents

Publication Publication Date Title
CN111988593B (en) Three-dimensional image color correction method and system based on depth residual optimization
US11983893B2 (en) Systems and methods for hybrid depth regularization
CN103236082B (en) Towards the accurate three-dimensional rebuilding method of two-dimensional video of catching static scene
CN112884682B (en) Stereo image color correction method and system based on matching and fusion
CN107277491B (en) Generate the method and corresponding medium of the depth map of image
Nalpantidis et al. Stereo vision for robotic applications in the presence of non-ideal lighting conditions
US20140009462A1 (en) Systems and methods for improving overall quality of three-dimensional content by altering parallax budget or compensating for moving objects
CN109308719B (en) Binocular parallax estimation method based on three-dimensional convolution
KR101560508B1 (en) Method and arrangement for 3-dimensional image model adaptation
CN110211043A (en) A kind of method for registering based on grid optimization for Panorama Mosaic
CN110288712B (en) Sparse multi-view three-dimensional reconstruction method for indoor scene
CN111047510A (en) Large-field-angle image real-time splicing method based on calibration
TW201243763A (en) Method for 3D video content generation
CN102223556A (en) Multi-view stereoscopic image parallax free correction method
US10277877B2 (en) 3D system including a neural network
CN102480626A (en) Image processing apparatus, display apparatus and image processing program
KR20080101998A (en) Method and device for rectifying image in synthesizing arbitary view image
Waidhofer et al. PanoSynthVR: Toward light-weight 360-degree view synthesis from a single panoramic input
CN104851129B (en) A kind of 3D method for reconstructing based on multiple views
Adhikarla et al. Real-time adaptive content retargeting for live multi-view capture and light field display
Liu et al. An efficient depth map preprocessing method based on structure-aided domain transform smoothing for 3D view generation
CN109218706B (en) Method for generating stereoscopic vision image from single image
Knorr et al. From 2D-to stereo-to multi-view video
CN117237431A (en) Training method and device of depth estimation model, electronic equipment and storage medium
Gurrieri et al. Efficient panoramic sampling of real-world environments for image-based stereoscopic telepresence

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant