CN110349099B - Complex scene video shadow detection and elimination method - Google Patents

Complex scene video shadow detection and elimination method Download PDF

Info

Publication number
CN110349099B
CN110349099B CN201910523329.9A CN201910523329A CN110349099B CN 110349099 B CN110349099 B CN 110349099B CN 201910523329 A CN201910523329 A CN 201910523329A CN 110349099 B CN110349099 B CN 110349099B
Authority
CN
China
Prior art keywords
shadow
confidence
pixel
video
point
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910523329.9A
Other languages
Chinese (zh)
Other versions
CN110349099A (en
Inventor
肖春霞
吴文君
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Wuhan University WHU
Original Assignee
Wuhan University WHU
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wuhan University WHU filed Critical Wuhan University WHU
Priority to CN201910523329.9A priority Critical patent/CN110349099B/en
Publication of CN110349099A publication Critical patent/CN110349099A/en
Application granted granted Critical
Publication of CN110349099B publication Critical patent/CN110349099B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof
    • G06T5/94—Dynamic range modification of images or parts thereof based on local image properties, e.g. for local contrast enhancement
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a video shadow detection and elimination method based on depth information, which comprises the steps of firstly estimating normal information and point cloud position information of each pixel point by utilizing the depth information of an image, estimating a shadow confidence value of each pixel point by comparing the characteristic similarity between each pixel point and a space-time local neighborhood pixel point in a video stream, further optimizing the shadow confidence by utilizing a Laplace operator to obtain a final shadow detection result, and finally constructing an illumination recovery optimization equation based on the video stream by utilizing the shadow detection result to obtain a final shadow elimination result. The invention has the following advantages: the texture filtering is utilized to effectively reduce the interference of texture information on shadow detection, the Laplace operator is utilized to optimize the initial shadow confidence coefficient to obtain a more perfect shadow detection result, the chromaticity constraint and the correlation of the previous frame and the next frame are utilized to eliminate the shadow, and the chromaticity invariance and the interframe continuity of the result can be effectively ensured.

Description

Complex scene video shadow detection and elimination method
Technical Field
The invention belongs to the technical field of video processing, and particularly relates to a method for detecting and eliminating video shadows of a complex scene.
Background
Shadows are common natural phenomena in our daily lives, and they can provide important information for understanding of visual scenes, such as lighting environment, scene geometry, and the like. The information plays an important role in illumination analysis, relighting, augmented reality and other applications. Therefore, effectively detecting and eliminating shadows is an important topic in the field of computer vision. However, it is a very difficult task to automatically detect and eliminate the shadow, which is not only affected by the local texture material information, but also needs to consider the global structure information and the illumination environment information in the scene. Most of the existing shadow detection and elimination algorithms are used for detecting and classifying shadows based on local chrominance information, gradient information and the like, and global structure information is not considered, so that the algorithms cannot effectively process complex shadows and shadows in complex scenes.
The shadow processing work in the complex scene means that in the complex environment, the shadow is automatically detected and eliminated by utilizing the global information and the local information, and meanwhile, the shading gradual change information in the scene is kept in the shadow elimination result, so that the visual distortion is prevented. The reason for the difficulty in eliminating the shadow of the complex scene mainly has two aspects, firstly, the texture information of the material in the complex scene is rich, the shadow distribution is disordered and not concentrated, the difficulty is increased for the detection work, even with the help of manual interaction priori knowledge, the labeling burden can be increased for the complex shadow scene, and the efficient batch processing is difficult to perform; secondly, the shadow image of the complex scene lacks a corresponding data set due to difficult labeling, and the shadow in the large scene is difficult to eliminate by utilizing a deep learning method. Aiming at the problems, the algorithm provides an automatic detection and elimination algorithm of the shadow of the complex scene based on the image depth information, the algorithm does not need manual interaction and the depth information of the collected image, and the shadow in the complex scene can be detected and eliminated by utilizing the image depth information estimated by the existing image depth estimation algorithm.
Disclosure of Invention
Aiming at the defects of the prior art, the invention provides a depth information-based complex scene video shadow detection and elimination method.
The technical scheme of the invention is a complex scene video shadow detection and elimination method, which comprises the following steps:
step 1, for an input video stream V, obtaining depth information of the input video stream V;
step 2, inputting a video frame I into each frame, and performing filtering processing by using a texture filtering operator to reduce texture influence and simultaneously keep shadow information in the video frame;
step 3, for each filtered video frame TiSelecting adjacent related video streams, finding out the initial shadow confidence and the brightness confidence of each pixel point in the video frames, and optimizing the shadow confidence of each frame to obtain a final video shadow detection result;
step 4, further calculating the confidence coefficient of the shadow boundary by using the total variation and the inherent variation of the shadow confidence coefficient and the brightness confidence coefficient;
step 5, after obtaining the shadow detection result of each frame, decomposing the current frame image I into an image F without shadow and a shadow factor beta by using a shadow image model beta as I/F, and constructing a shadow removal optimization equation to constrain and optimize each frame;
and 6, carrying out iterative optimization solution on the shadow removing optimization equation to obtain a final video shadow elimination result F and a shadow factor beta.
Further, the specific implementation of step 3 includes the following sub-steps:
step 3.1, performing point cloud estimation by using depth information of each video frame and combining camera parameters to obtain point cloud information of each pixel point, constructing a k-d tree by using the point cloud information, finding a plurality of point clouds which are most similar to the point cloud of each pixel point, and calculating normal information of a space tangent plane where the pixel point is located by using the similar point clouds;
step 3.2, for each filtered video frame TiCalculating the point q ∈ R of each pixel point p and the point q ∈ R in the spatial neighborhood by utilizing the Gaussian similaritypThe chroma similarity, the spatial distance similarity and the normal similarity of the three kinds of similarity are multiplied to obtain the final characteristic similarity alphapq(ii) a Wherein R ispIs the spatial neighborhood of the pixel point p;
step 3.3, comparing each pixel p with all neighbor pixels q e R in the spatial neighborhood by utilizing the similarity between the pixelspThe weighted average value of the image intensity estimates the shadow confidence of each pixel point
Figure GDA0002893612100000021
And confidence of brightness
Figure GDA0002893612100000022
Figure GDA0002893612100000023
Figure GDA0002893612100000024
Wherein, the intensity weighted average value of the neighborhood where the p point is located
Figure GDA0002893612100000025
IpAnd IqRespectively representing the intensities of pixels p and q, sigma being an adjustable parameter, | RpI denotes the neighborhoodDomain range RpThe number of the pixel points in (1);
step 3.4, in each video frame, utilizing a laplacian operator, and combining the results of the initial shadow confidence coefficient and the brightness confidence coefficient calculated in the step 3.3, constructing an optimization equation to obtain a final shadow detection result S:
Figure GDA0002893612100000031
wherein, the first two terms are data constraint terms, and the third term is a smoothing term; n is the number of pixel points in the image, SkIs the shadow confidence optimization result of the kth pixel point, omegakIs the local window where the kth pixel point is located, SiAnd SjIs window omegakThe shadow confidence degree optimization results corresponding to the two pixel points i and j in the window are used for the pixel points w in the smooth windowijIs the matching Laplacian value of the i and j points in the neighborhood.
Further, the confidence of the shadow boundary calculated in step 4 is formulated as,
Figure GDA0002893612100000032
wherein the content of the first and second substances,
Figure GDA0002893612100000033
and
Figure GDA0002893612100000034
respectively representing the total variation and the inherent variation of the p-point confidence coefficient graph, wherein the epsilon is a constant;
Figure GDA0002893612100000035
Figure GDA0002893612100000036
wherein R (p) is a rectangular neighborhood with p as the center point,
Figure GDA0002893612100000037
representing a weight function defined by gaussian filtering,
Figure GDA0002893612100000038
and
Figure GDA0002893612100000039
q-point shadow confidence and luminance confidence, respectively, theta is the partial derivative sign,
Figure GDA00028936121000000310
representing confidence in shading
Figure GDA00028936121000000311
Or confidence of brightness
Figure GDA00028936121000000312
A partial derivation in the x or y direction is performed.
Further, the de-shadow optimization equation in step 5 is,
E(F,β)＝Edata(F,β)+λ1Esmooth(F,β)
+λ2Echromaticity(β)+λ3Econst(β)
wherein the data item Edata＝ωiw∑c∈{R,G,B}ωc·|Ic-Fc·βc|2For constraining each data item of the current frame, using the shadow model to process the data I under different color channelsc，Fc，βcMake a constraint in whichR,ωG,ωBThe constraint weight is the constraint weight of each color channel of RGB; pixel intensity weight ωiw＝1-ωintensity1- | I (x) |, where ω isintensityFor adjustable parameters, i (x) is the pixel intensity of pixel point x;
smoothing term Esmooth＝ESF+γESMWherein gamma is a balance factor, ESFThe deblocked image F is subjected to a smoothing constraint based on the following assumptions: on the same spatial plane, the pixel points with similar chrominance information, normal information and three-dimensional point cloud position information should have similar pixel values (color values) after the shadow is eliminated,
Figure GDA0002893612100000041
the first term is smooth constraint between adjacent pixel points of the current frame, the second term utilizes non-shadow pixel points with similar characteristics in the video stream to constrain shadow pixels, and R issFor shadow pixel points obtained by shadow detection in the current frame,
Figure GDA0002893612100000042
representing a point set of all non-shadow pixel points in a T frame in a spatial-temporal local neighborhood, wherein T is the total frame number in the current video stream;
ESMusing estimated shadow boundary confidence CboundThe shading factor β is smoothly constrained:
Figure GDA0002893612100000043
Echromaticity(β)＝||c(p)-cF(p)||2carrying out chroma consistent constraint on the original video frame and the video frame with the shadow eliminated by using the assumption that the chroma of the image is not influenced by illumination change, wherein c is the chroma of the current frame I, and c is the chroma of the current frame IFThe chromaticity which is the shadow elimination result F;
Figure GDA0002893612100000044
the color of the pixel point of the image non-shadow area after the assumed shadow is eliminated is kept unchanged, namely the shadow factor approaches to 1, and the non-shadow area N is subjected tobConstraint is carried out, and the non-shadow pixel area is pixels except all shadow points and neighbor pixel points thereofAnd (4) point collection, wherein the shadow pixel points are points with confidence degrees larger than 0.1.
Further, in step 6, an iterative optimization method is adopted to solve, the initial value of F is I, the initial value of β is a shadow confidence coefficient S, and a final result is calculated through iterative optimization, wherein the maximum iteration number is 1000.
Further, in the step 1, for the video shot on site, the Kinect V2 is used for collecting the depth information of the video in real time; for the existing video, the depth information of each frame of the video is estimated by using a deep learning method.
The invention has the following advantages: 1. the method carries out texture filtering pretreatment on each frame when estimating the shadow of the video frame, and can effectively reduce the interference of texture information on shadow detection; 2. according to the method, the initial shadow confidence coefficient is optimized by using the Laplace operator, so that a more complete shadow detection result is obtained, and meanwhile, the relative strength of shadow information and the gradient information of a shadow boundary are kept, and the elimination of a complex shadow and a shadow boundary is facilitated; 3. the shadow elimination optimization algorithm of the invention utilizes the principle that the chroma information of the image is kept unchanged before and after the shadow elimination to restrain the chroma information and ensure the invariance of the result chroma; 4, the shadow elimination algorithm of the invention fully utilizes the correlation of the previous and the next frames, and can effectively ensure the inter-frame continuity of the result while eliminating the shadow.
Drawings
FIG. 1 is a flow chart of video shadow removal of the present invention.
FIG. 2 is a flow chart of video shadow detection of the present invention.
FIG. 3 is a diagram of the effects of an example of the invention processing video. The method comprises the steps of (a) inputting a video stream, (b) depth information corresponding to the video stream, (c) a shadow confidence estimation result of the video stream, (d) an optimized shadow detection result (confidence) of the video stream, and (e) a video stream result after shadow removal.
Detailed Description
The present invention will be described in further detail below with reference to examples of implementation and the accompanying drawings, but the embodiments of the present invention are not limited thereto.
Referring to fig. 1, a flow chart of the present invention, a video shadow elimination method, includes the following steps:
step 1, for an input video stream V, acquiring depth information thereof: for a video shot on site, acquiring depth information of the video in real time by using Kinect V2; for the existing video, the depth information of each frame of the video is estimated by using a deep learning method. As shown in fig. 3(a), (b), which are an input video frame and a corresponding depth map in the example, respectively.
And 2, filtering each frame I in the video by using a texture filtering operator, reducing the influence of small-scale texture on shadow detection, and simultaneously keeping original shadow information.
Step 3, for each filtered video frame TiSelect its corresponding associated video stream { T }i-2,Ti-1,Ti,Ti+1,Ti+2Finding out the initial shadow confidence and brightness confidence of each pixel point in the video frame, and optimizing the shadow confidence of each frame to obtain the final video shadow detection result. The step 3 comprises the following steps:
step 3.1, utilizing the depth information of each video frame and combining camera parameters to carry out point cloud estimation; after the point cloud information of each pixel point is obtained, a k-d tree is constructed by using the point cloud information, the most similar 300 point clouds are found for the point clouds of each pixel, and the normal information of the space tangent plane where the pixel point is located is calculated by using the similar point clouds.
Step 3.2, as shown in the flow chart of fig. 2 for shadow detection, for each filtered video frame TiFinding out the space-time local neighborhood pixel point q E R corresponding to each pixel point p in the related video streampAnd calculating the chroma similarity, the spatial distance similarity and the normal similarity of the pixel point p and all the neighbor points q, and multiplying the three similarities to obtain the final characteristic similarity alphapq. Wherein R ispWhich is a spatio-temporal neighborhood of pixel point p, typically a 50 x 5 spatio-temporal pixel block.
Step 3.3, comparing each pixel point p with all neighbor pixel points in the spatial neighborhood thereof by utilizing the similarity between the pixel pointsq∈RpThe weighted average value m (p) of the image intensity, and the shadow confidence coefficient of each pixel point is estimated
Figure GDA0002893612100000051
And confidence of brightness
Figure GDA0002893612100000052
Figure GDA0002893612100000053
Figure GDA0002893612100000054
Fig. 3(c) shows the initial shadow confidence estimation result corresponding to each frame in the example.
Step 3.4, optimization equation is utilized
Figure GDA0002893612100000061
And obtaining the final optimization result of the shadow confidence coefficient S. Wherein, the first two terms are data constraint terms, and the third term is a smoothing term; n is the number of pixel points in the image, SkIs the shadow confidence optimization result of the kth pixel point, omegakIs the local window, s, where the k-th pixel is locatediAnd sjIs window omegakThe shadow confidence degree optimization results corresponding to the two pixel points i and j in the window are used for the pixel points w in the smooth windowijIs the matching Laplacian value of the i and j points in the neighborhood. In each video frame, the result of the shadow detection is constrained by using a Laplacian operator in combination with the result of the initial shadow confidence and the brightness confidence calculated in step 3.3, and gradient correlation smoothing is performed by using a Laplacian matting operator (matting Laplacian) to obtain an optimized shadow detection result. Example results are shown in FIG. 3(d), which is the final shadow confidence optimization result.
Step 4, utilizing total variation of shadow confidence coefficient and brightness confidence coefficient
Figure GDA0002893612100000062
And inherent variation of
Figure GDA0002893612100000063
The shadow boundary confidence is further calculated:
Figure GDA0002893612100000064
wherein the content of the first and second substances,
Figure GDA0002893612100000065
and
Figure GDA0002893612100000066
for total variation and inherent variation, respectively, ∈ is usually set to 0.001 in the experiment, and in order to prevent the denominator from being 0:
Figure GDA0002893612100000067
Figure GDA0002893612100000068
where R (p) is a 7 × 7 rectangular neighborhood with p as the center point,
Figure GDA0002893612100000069
representing a weight function defined by gaussian filtering,
Figure GDA00028936121000000610
and
Figure GDA00028936121000000611
q-point shadow confidence and luminance confidence respectively,
Figure GDA00028936121000000612
representing confidence in shading
Figure GDA00028936121000000613
Or confidence of brightness
Figure GDA00028936121000000614
Performing partial derivatives in the x or y direction, total variation
Figure GDA00028936121000000615
The inherent variation is obtained by taking the absolute value of the partial derivative result and then carrying out Gaussian filtering
Figure GDA00028936121000000616
The method is obtained by performing Gaussian filtering on the partial derivative result and then taking an absolute value.
And 5, after obtaining the shadow detection result of each frame, decomposing the current frame image I into an image F without shadow and a shadow factor beta by using a shadow image model beta as I/F, wherein the two are unknown quantities. The invention constructs the following optimization equation, and restrains and optimizes each frame based on the video stream to obtain the final video shadow elimination result F and the shadow factor beta.
E(F,β)＝Edata(F,β)+λ1Esmooth(F,β)
+λ2Echromaticity(β)+λ3Econst(β)
In experimental operation, we will generally refer to the parameter λ1，λ2And λ3Set to 1, 0.5 and 1, respectively.
Data item Edata＝ωiw∑c∈{R,G,B}ωc·|Ic-Fc·βc|2For constraining each data item of the current frame, using the shadow model to process the data I under different color channelsc，Fc，βcAnd (6) carrying out constraint. Wherein, ω iscThe weight is restricted for three color channels of RGB, and the weight of each channel is { omegaR,ωG,ωB}＝{0.299,0.587,0.144}。ωiwIs a weight related to the intensity of the pixel,ωiw＝1-ωintensity1- | I (x) |, where ω isintensityFor the adjustable parameter, i (x) is the pixel intensity of pixel point x.
Smoothing term Esmooth＝ESF+γESMWhere γ is the balance factor, and is typically set to 1 in experimental work. Based on the assumptions: on the same spatial plane, pixel points with similar chrominance information, normal information and three-dimensional point cloud position information should have similar pixel values (color values) after the shadow is eliminated. And carrying out smooth constraint on the image F after the shadow is removed:
Figure GDA0002893612100000071
the first term is smooth constraint between adjacent pixel points of the current frame, the second term utilizes non-shadow pixel points with similar characteristics in the video stream to constrain shadow pixels, and R issFor shadow pixel points obtained by shadow detection in the current frame,
Figure GDA0002893612100000072
and representing the point set of all non-shaded pixel points in the T frame in the spatial-temporal local neighborhood, wherein T is the total frame number in the current video stream.
ESM＝∑p∈I(1-|Cbound(p)|)∑q∈Rp||βp-βq||2Using estimated shadow boundary confidence CboundThe shading factor beta is smoothly constrained.
Echromaticity(β)＝||c(p)-cF(p)||2Carrying out chroma consistent constraint on the original video frame and the video frame with the shadow eliminated by using the assumption that the chroma of the image is not influenced by illumination change, wherein c is the chroma of the current frame I, and c is the chroma of the current frame IFThe chromaticity of the shadow elimination result F.
Figure GDA0002893612100000073
The color of the pixel points in the non-shadow area of the image after the assumed shadow is eliminated is kept unchanged,i.e. the shading factor approaches 1, for the non-shaded area NbAnd (4) carrying out constraint, wherein the shadow pixel points are points with confidence coefficient greater than 0.1, and the non-shadow pixel areas are pixel point sets except all shadow points and neighbor pixel points thereof.
And 6, because the equation contains two unknowns of the image F without shadow and the shadow factor beta, the algorithm adopts an iterative optimization method to solve, the initial value of the F is I, the initial value of the beta is the shadow confidence coefficient S, the final result is calculated through iterative optimization, and the maximum iteration frequency is 1000. As shown in fig. 3(e), the shadow elimination result of the example is shown.
The specific embodiments described herein are merely illustrative of the spirit of the invention. Various modifications or additions may be made to the described embodiments or alternatives may be employed by those skilled in the art without departing from the spirit or ambit of the invention as defined in the appended claims.

Claims (5)

1. A method for detecting and eliminating complex scene video shadow is characterized by comprising the following steps:
step 1, for an input video stream V, obtaining depth information of the input video stream V;
step 2, inputting a video frame I into each frame, and performing filtering processing by using a texture filtering operator to reduce texture influence and simultaneously keep shadow information in the video frame;
step 3, for each filtered video frame TiSelecting adjacent related video streams, finding out the initial shadow confidence and the brightness confidence of each pixel point in the video frames, and optimizing the shadow confidence of each frame to obtain a final video shadow detection result;
step 4, further calculating the confidence coefficient of the shadow boundary by using the total variation and the inherent variation of the shadow confidence coefficient and the brightness confidence coefficient;
step 5, after obtaining the shadow detection result of each frame, decomposing the current frame image I into an image F without shadow and a shadow factor beta by using a shadow image model beta as I/F, and constructing a shadow removal optimization equation to constrain and optimize each frame;
the de-shadowing optimization equation in step 5 is,
E(F，β)＝Edata(F，β)+λ1Esmooth(F，β)+λ2Echromaticity(β)+λ3Econst(β)
wherein the data item Edata＝ωiw∑c∈{R，G，B}ωc·|Ic-Fc·βc|2For constraining each data item of the current frame, using the shadow model to process the data I under different color channelsc，Fc，βcMake a constraint in whichR，ωG，ωBThe constraint weight is the constraint weight of each color channel of RGB; pixel intensity weight ωiw＝1-ωintensity1- | I (x) |, where ω isintensityFor adjustable parameters, i (x) is the pixel intensity of pixel point x;
smoothing term Esmooth＝ESF+γESMWherein gamma is a balance factor, ESFThe deblocked image F is subjected to a smoothing constraint based on the following assumptions: on the same spatial plane, the pixel points with similar chrominance information, normal information and three-dimensional point cloud position information should have similar pixel values after the shadow is eliminated,
Figure FDA0002893612090000011
the first term is smooth constraint between adjacent pixel points of the current frame, the second term utilizes non-shadow pixel points with similar characteristics in the video stream to constrain shadow pixels, and R issFor shadow pixel points obtained by shadow detection in the current frame,
Figure FDA0002893612090000012
representing a point set of all non-shadow pixel points in a T frame in a spatial-temporal local neighborhood, wherein T is the total frame number in the current video stream;
ESMusing estimated shadow edgesConfidence of boundary CboundThe shading factor β is smoothly constrained:
Figure FDA0002893612090000021
Echromaticity(β)＝||c(p)-cF(p)||2carrying out chroma consistent constraint on the original video frame and the video frame with the shadow eliminated by using the assumption that the chroma of the image is not influenced by illumination change, wherein c is the chroma of the current frame I, and c is the chroma of the current frame IFThe chromaticity which is the shadow elimination result F;
Figure FDA0002893612090000022
the color of the pixel point of the image non-shadow area after the assumed shadow is eliminated is kept unchanged, namely the shadow factor approaches to 1, and the non-shadow area N is subjected tobConstraint is carried out, the non-shadow pixel area is a pixel point set except all shadow points and neighbor pixel points thereof, and the shadow pixel points are points with confidence coefficient greater than 0.1;
and 6, carrying out iterative optimization solution on the shadow removing optimization equation to obtain a final video shadow elimination result F and a shadow factor beta.
2. The method as claimed in claim 1, wherein the method comprises: the specific implementation of the step 3 comprises the following substeps:
step 3.1, performing point cloud estimation by using depth information of each video frame and combining camera parameters to obtain point cloud information of each pixel point, constructing a k-d tree by using the point cloud information, finding a plurality of point clouds which are most similar to the point cloud of each pixel point, and calculating normal information of a space tangent plane where the pixel point is located by using the similar point clouds;
step 3.2, for each filtered video frame TiCalculating the point q ∈ R of each pixel point p and the point q ∈ R in the spatial neighborhood by utilizing the Gaussian similaritypThe chroma similarity, the spatial distance similarity and the normal similarity of the color data, and thenMultiplying the three similarity degrees to obtain the final feature similarity degree alphapq(ii) a Wherein R ispIs the spatial neighborhood of the pixel point p;
step 3.3, comparing each pixel p with all neighbor pixels q e R in the spatial neighborhood by utilizing the similarity between the pixelspThe weighted average value of the image intensity estimates the shadow confidence of each pixel point
Figure FDA0002893612090000023
And confidence of brightness
Figure FDA0002893612090000024
Figure FDA0002893612090000031
Figure FDA0002893612090000032
Wherein, the intensity weighted average value of the neighborhood where the p point is located
Figure FDA0002893612090000033
IpAnd IqRespectively representing the intensities of pixels p and q, sigma being an adjustable parameter, | RpI represents the neighborhood region RpThe number of the pixel points in (1);
step 3.4, in each video frame, utilizing a laplacian operator, and combining the results of the initial shadow confidence coefficient and the brightness confidence coefficient calculated in the step 3.3, constructing an optimization equation to obtain a final shadow detection result S:
Figure FDA0002893612090000034
wherein, the first two terms are data constraint terms, and the third term is a smoothing term; n is a pixel in the imageNumber of dots, SkIs the shadow confidence optimization result of the kth pixel point, omegakIs the local window where the kth pixel point is located, SiAnd SjIs window omegakThe shadow confidence degree optimization results corresponding to the two pixel points i and j in the window are used for the pixel points w in the smooth windowijIs the matching Laplacian value of the i and j points in the neighborhood.
3. The method as claimed in claim 2, wherein the method comprises the steps of: the formula for calculating the confidence of the shadow boundary in step 4 is,
Figure FDA0002893612090000035
wherein the content of the first and second substances,
Figure FDA0002893612090000036
and
Figure FDA0002893612090000037
respectively representing the total variation and the inherent variation of the p-point confidence coefficient graph, wherein the epsilon is a constant;
Figure FDA0002893612090000038
Figure FDA0002893612090000039
wherein R (p) is a rectangular neighborhood with p as the center point,
Figure FDA00028936120900000310
representing a weight function defined by gaussian filtering,
Figure FDA00028936120900000311
and
Figure FDA00028936120900000312
q-point shadow confidence and luminance confidence respectively,
Figure FDA00028936120900000313
is a sign of the partial derivative,
Figure FDA00028936120900000314
representing confidence in shading
Figure FDA00028936120900000315
Or confidence of brightness
Figure FDA00028936120900000316
A partial derivation in the x or y direction is performed.
4. The method as claimed in claim 3, wherein the method comprises the steps of: and 6, solving by adopting an iterative optimization method, setting the initial value of F as I and the initial value of beta as a shadow confidence coefficient S, and calculating a final result through iterative optimization, wherein the maximum iteration number is 1000.
5. The method as claimed in claim 1, wherein the method comprises: in the step 1, for a video shot on site, acquiring depth information of the video in real time by using Kinect V2; for the existing video, the depth information of each frame of the video is estimated by using a deep learning method.
CN201910523329.9A 2019-06-17 2019-06-17 Complex scene video shadow detection and elimination method Active CN110349099B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910523329.9A CN110349099B (en) 2019-06-17 2019-06-17 Complex scene video shadow detection and elimination method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910523329.9A CN110349099B (en) 2019-06-17 2019-06-17 Complex scene video shadow detection and elimination method

Publications (2)

Publication Number Publication Date
CN110349099A CN110349099A (en) 2019-10-18
CN110349099B true CN110349099B (en) 2021-04-02

Family

ID=68182147

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910523329.9A Active CN110349099B (en) 2019-06-17 2019-06-17 Complex scene video shadow detection and elimination method

Country Status (1)

Country Link
CN (1) CN110349099B (en)

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112419196B (en) * 2020-11-26 2022-04-26 武汉大学 Unmanned aerial vehicle remote sensing image shadow removing method based on deep learning
CN112686936B (en) * 2020-12-18 2023-08-04 北京百度网讯科技有限公司 Image depth completion method, apparatus, computer device, medium, and program product
CN112598592A (en) * 2020-12-24 2021-04-02 广东博智林机器人有限公司 Image shadow removing method and device, electronic equipment and storage medium
CN113361360B (en) * 2021-05-31 2023-07-25 山东大学 Multi-person tracking method and system based on deep learning
CN113378775B (en) * 2021-06-29 2023-04-07 武汉大学 Video shadow detection and elimination method based on deep learning
CN114782616B (en) * 2022-06-20 2022-09-20 北京飞渡科技有限公司 Model processing method and device, storage medium and electronic equipment
CN116704316A (en) * 2023-08-03 2023-09-05 四川金信石信息技术有限公司 Substation oil leakage detection method, system and medium based on shadow image reconstruction

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9430715B1 (en) * 2015-05-01 2016-08-30 Adobe Systems Incorporated Identifying and modifying cast shadows in an image
CN107203975A (en) * 2017-04-18 2017-09-26 南京航空航天大学 Shadow removal method based on YCbCr color spaces and topology cutting
CN107808366A (en) * 2017-10-21 2018-03-16 天津大学 A kind of adaptive optical transfer single width shadow removal method based on Block- matching
CN109064411A (en) * 2018-06-13 2018-12-21 长安大学 A kind of pavement image based on illumination compensation removes shadow method

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106339995A (en) * 2016-08-30 2017-01-18 电子科技大学 Space-time multiple feature based vehicle shadow eliminating method
CN107038690B (en) * 2017-03-27 2020-04-28 湘潭大学 Moving shadow removing method based on multi-feature fusion

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9430715B1 (en) * 2015-05-01 2016-08-30 Adobe Systems Incorporated Identifying and modifying cast shadows in an image
CN107203975A (en) * 2017-04-18 2017-09-26 南京航空航天大学 Shadow removal method based on YCbCr color spaces and topology cutting
CN107808366A (en) * 2017-10-21 2018-03-16 天津大学 A kind of adaptive optical transfer single width shadow removal method based on Block- matching
CN109064411A (en) * 2018-06-13 2018-12-21 长安大学 A kind of pavement image based on illumination compensation removes shadow method

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
Shadow Removal from Single RGB-D Images;Yao Xiao et al;《2014 IEEE Conference on Computer Vision and Pattern Recognition》;20140925;3011-3018页 *

Also Published As

Publication number Publication date
CN110349099A (en) 2019-10-18

Similar Documents

Publication Publication Date Title
CN110349099B (en) Complex scene video shadow detection and elimination method
CN109859171B (en) Automatic floor defect detection method based on computer vision and deep learning
US10574905B2 (en) System and methods for depth regularization and semiautomatic interactive matting using RGB-D images
JP4074062B2 (en) Semantic object tracking in vector image sequences
US9042662B2 (en) Method and system for segmenting an image
CN107240084B (en) Method and device for removing rain from single image
CN108377374B (en) Method and system for generating depth information related to an image
WO2018053952A1 (en) Video image depth extraction method based on scene sample library
CN104809709A (en) Single-image self-adaptation defogging method based on domain transformation and weighted quadtree decomposition
CN111310768B (en) Saliency target detection method based on robustness background prior and global information
CN111161219B (en) Robust monocular vision SLAM method suitable for shadow environment
Li et al. Optimal seamline detection in dynamic scenes via graph cuts for image mosaicking
Fan et al. Haze editing with natural transmission
He et al. Single-image shadow removal using 3D intensity surface modeling
Xiaoxu et al. Image dehazing base on two-peak channel prior
Shen et al. Re-texturing by intrinsic video
CN111932470A (en) Image restoration method, device, equipment and medium based on visual selection fusion
CN111932469A (en) Significance weight quick exposure image fusion method, device, equipment and medium
RU2522044C1 (en) Apparatus for selecting object contours on textured background when processing digital images
CN112446820A (en) Method for removing irrelevant portrait of scenic spot photo
Kang et al. FPGA implementation for enhancing image using pixel-based median channel prior
CN113781329B (en) Fog removing method for remote sensing image
Zhu et al. A novel rain detection and removal approach using guided filtering and formation modeling
US20220414820A1 (en) Method of inserting an object into a sequence of images
CN110189415B (en) Outdoor scene reconstruction method based on delayed video

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant