CN110349099B - Complex scene video shadow detection and elimination method - Google Patents
Complex scene video shadow detection and elimination method Download PDFInfo
- Publication number
- CN110349099B CN110349099B CN201910523329.9A CN201910523329A CN110349099B CN 110349099 B CN110349099 B CN 110349099B CN 201910523329 A CN201910523329 A CN 201910523329A CN 110349099 B CN110349099 B CN 110349099B
- Authority
- CN
- China
- Prior art keywords
- shadow
- confidence
- pixel
- video
- point
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 31
- 238000000034 method Methods 0.000 title claims abstract description 31
- 230000008030 elimination Effects 0.000 title claims abstract description 21
- 238000003379 elimination reaction Methods 0.000 title claims abstract description 21
- 238000005457 optimization Methods 0.000 claims abstract description 28
- 238000001914 filtration Methods 0.000 claims abstract description 13
- 238000005286 illumination Methods 0.000 claims abstract description 6
- 238000009499 grossing Methods 0.000 claims description 9
- 238000012545 processing Methods 0.000 claims description 6
- 230000008859 change Effects 0.000 claims description 4
- 238000013135 deep learning Methods 0.000 claims description 4
- 230000008569 process Effects 0.000 claims description 4
- 238000013459 approach Methods 0.000 claims description 3
- 230000006870 function Effects 0.000 claims description 3
- 238000009795 derivation Methods 0.000 claims description 2
- 238000011084 recovery Methods 0.000 abstract 1
- 230000003993 interaction Effects 0.000 description 2
- 238000002372 labelling Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/90—Dynamic range modification of images or parts thereof
- G06T5/94—Dynamic range modification of images or parts thereof based on local image properties, e.g. for local contrast enhancement
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a video shadow detection and elimination method based on depth information, which comprises the steps of firstly estimating normal information and point cloud position information of each pixel point by utilizing the depth information of an image, estimating a shadow confidence value of each pixel point by comparing the characteristic similarity between each pixel point and a space-time local neighborhood pixel point in a video stream, further optimizing the shadow confidence by utilizing a Laplace operator to obtain a final shadow detection result, and finally constructing an illumination recovery optimization equation based on the video stream by utilizing the shadow detection result to obtain a final shadow elimination result. The invention has the following advantages: the texture filtering is utilized to effectively reduce the interference of texture information on shadow detection, the Laplace operator is utilized to optimize the initial shadow confidence coefficient to obtain a more perfect shadow detection result, the chromaticity constraint and the correlation of the previous frame and the next frame are utilized to eliminate the shadow, and the chromaticity invariance and the interframe continuity of the result can be effectively ensured.
Description
Technical Field
The invention belongs to the technical field of video processing, and particularly relates to a method for detecting and eliminating video shadows of a complex scene.
Background
Shadows are common natural phenomena in our daily lives, and they can provide important information for understanding of visual scenes, such as lighting environment, scene geometry, and the like. The information plays an important role in illumination analysis, relighting, augmented reality and other applications. Therefore, effectively detecting and eliminating shadows is an important topic in the field of computer vision. However, it is a very difficult task to automatically detect and eliminate the shadow, which is not only affected by the local texture material information, but also needs to consider the global structure information and the illumination environment information in the scene. Most of the existing shadow detection and elimination algorithms are used for detecting and classifying shadows based on local chrominance information, gradient information and the like, and global structure information is not considered, so that the algorithms cannot effectively process complex shadows and shadows in complex scenes.
The shadow processing work in the complex scene means that in the complex environment, the shadow is automatically detected and eliminated by utilizing the global information and the local information, and meanwhile, the shading gradual change information in the scene is kept in the shadow elimination result, so that the visual distortion is prevented. The reason for the difficulty in eliminating the shadow of the complex scene mainly has two aspects, firstly, the texture information of the material in the complex scene is rich, the shadow distribution is disordered and not concentrated, the difficulty is increased for the detection work, even with the help of manual interaction priori knowledge, the labeling burden can be increased for the complex shadow scene, and the efficient batch processing is difficult to perform; secondly, the shadow image of the complex scene lacks a corresponding data set due to difficult labeling, and the shadow in the large scene is difficult to eliminate by utilizing a deep learning method. Aiming at the problems, the algorithm provides an automatic detection and elimination algorithm of the shadow of the complex scene based on the image depth information, the algorithm does not need manual interaction and the depth information of the collected image, and the shadow in the complex scene can be detected and eliminated by utilizing the image depth information estimated by the existing image depth estimation algorithm.
Disclosure of Invention
Aiming at the defects of the prior art, the invention provides a depth information-based complex scene video shadow detection and elimination method.
The technical scheme of the invention is a complex scene video shadow detection and elimination method, which comprises the following steps:
step 1, for an input video stream V, obtaining depth information of the input video stream V;
step 2, inputting a video frame I into each frame, and performing filtering processing by using a texture filtering operator to reduce texture influence and simultaneously keep shadow information in the video frame;
step 3, for each filtered video frame TiSelecting adjacent related video streams, finding out the initial shadow confidence and the brightness confidence of each pixel point in the video frames, and optimizing the shadow confidence of each frame to obtain a final video shadow detection result;
step 4, further calculating the confidence coefficient of the shadow boundary by using the total variation and the inherent variation of the shadow confidence coefficient and the brightness confidence coefficient;
step 5, after obtaining the shadow detection result of each frame, decomposing the current frame image I into an image F without shadow and a shadow factor beta by using a shadow image model beta as I/F, and constructing a shadow removal optimization equation to constrain and optimize each frame;
and 6, carrying out iterative optimization solution on the shadow removing optimization equation to obtain a final video shadow elimination result F and a shadow factor beta.
Further, the specific implementation of step 3 includes the following sub-steps:
step 3.1, performing point cloud estimation by using depth information of each video frame and combining camera parameters to obtain point cloud information of each pixel point, constructing a k-d tree by using the point cloud information, finding a plurality of point clouds which are most similar to the point cloud of each pixel point, and calculating normal information of a space tangent plane where the pixel point is located by using the similar point clouds;
step 3.2, for each filtered video frame TiCalculating the point q ∈ R of each pixel point p and the point q ∈ R in the spatial neighborhood by utilizing the Gaussian similaritypThe chroma similarity, the spatial distance similarity and the normal similarity of the three kinds of similarity are multiplied to obtain the final characteristic similarity alphapq(ii) a Wherein R ispIs the spatial neighborhood of the pixel point p;
step 3.3, comparing each pixel p with all neighbor pixels q e R in the spatial neighborhood by utilizing the similarity between the pixelspThe weighted average value of the image intensity estimates the shadow confidence of each pixel pointAnd confidence of brightness
Wherein, the intensity weighted average value of the neighborhood where the p point is locatedIpAnd IqRespectively representing the intensities of pixels p and q, sigma being an adjustable parameter, | RpI denotes the neighborhoodDomain range RpThe number of the pixel points in (1);
step 3.4, in each video frame, utilizing a laplacian operator, and combining the results of the initial shadow confidence coefficient and the brightness confidence coefficient calculated in the step 3.3, constructing an optimization equation to obtain a final shadow detection result S:
wherein, the first two terms are data constraint terms, and the third term is a smoothing term; n is the number of pixel points in the image, SkIs the shadow confidence optimization result of the kth pixel point, omegakIs the local window where the kth pixel point is located, SiAnd SjIs window omegakThe shadow confidence degree optimization results corresponding to the two pixel points i and j in the window are used for the pixel points w in the smooth windowijIs the matching Laplacian value of the i and j points in the neighborhood.
Further, the confidence of the shadow boundary calculated in step 4 is formulated as,
wherein,andrespectively representing the total variation and the inherent variation of the p-point confidence coefficient graph, wherein the epsilon is a constant;
wherein R (p) is a rectangular neighborhood with p as the center point,representing a weight function defined by gaussian filtering,andq-point shadow confidence and luminance confidence, respectively, theta is the partial derivative sign,representing confidence in shadingOr confidence of brightnessA partial derivation in the x or y direction is performed.
Further, the de-shadow optimization equation in step 5 is,
E(F,β)=Edata(F,β)+λ1Esmooth(F,β)
+λ2Echromaticity(β)+λ3Econst(β)
wherein the data item Edata=ωiw∑c∈{R,G,B}ωc·|Ic-Fc·βc|2For constraining each data item of the current frame, using the shadow model to process the data I under different color channelsc,Fc,βcMake a constraint in whichR,ωG,ωBThe constraint weight is the constraint weight of each color channel of RGB; pixel intensity weight ωiw=1-ωintensity1- | I (x) |, where ω isintensityFor adjustable parameters, i (x) is the pixel intensity of pixel point x;
smoothing term Esmooth=ESF+γESMWherein gamma is a balance factor, ESFThe deblocked image F is subjected to a smoothing constraint based on the following assumptions: on the same spatial plane, the pixel points with similar chrominance information, normal information and three-dimensional point cloud position information should have similar pixel values (color values) after the shadow is eliminated,
the first term is smooth constraint between adjacent pixel points of the current frame, the second term utilizes non-shadow pixel points with similar characteristics in the video stream to constrain shadow pixels, and R issFor shadow pixel points obtained by shadow detection in the current frame,representing a point set of all non-shadow pixel points in a T frame in a spatial-temporal local neighborhood, wherein T is the total frame number in the current video stream;
ESMusing estimated shadow boundary confidence CboundThe shading factor β is smoothly constrained:
Echromaticity(β)=||c(p)-cF(p)||2carrying out chroma consistent constraint on the original video frame and the video frame with the shadow eliminated by using the assumption that the chroma of the image is not influenced by illumination change, wherein c is the chroma of the current frame I, and c is the chroma of the current frame IFThe chromaticity which is the shadow elimination result F;
the color of the pixel point of the image non-shadow area after the assumed shadow is eliminated is kept unchanged, namely the shadow factor approaches to 1, and the non-shadow area N is subjected tobConstraint is carried out, and the non-shadow pixel area is pixels except all shadow points and neighbor pixel points thereofAnd (4) point collection, wherein the shadow pixel points are points with confidence degrees larger than 0.1.
Further, in step 6, an iterative optimization method is adopted to solve, the initial value of F is I, the initial value of β is a shadow confidence coefficient S, and a final result is calculated through iterative optimization, wherein the maximum iteration number is 1000.
Further, in the step 1, for the video shot on site, the Kinect V2 is used for collecting the depth information of the video in real time; for the existing video, the depth information of each frame of the video is estimated by using a deep learning method.
The invention has the following advantages: 1. the method carries out texture filtering pretreatment on each frame when estimating the shadow of the video frame, and can effectively reduce the interference of texture information on shadow detection; 2. according to the method, the initial shadow confidence coefficient is optimized by using the Laplace operator, so that a more complete shadow detection result is obtained, and meanwhile, the relative strength of shadow information and the gradient information of a shadow boundary are kept, and the elimination of a complex shadow and a shadow boundary is facilitated; 3. the shadow elimination optimization algorithm of the invention utilizes the principle that the chroma information of the image is kept unchanged before and after the shadow elimination to restrain the chroma information and ensure the invariance of the result chroma; 4, the shadow elimination algorithm of the invention fully utilizes the correlation of the previous and the next frames, and can effectively ensure the inter-frame continuity of the result while eliminating the shadow.
Drawings
FIG. 1 is a flow chart of video shadow removal of the present invention.
FIG. 2 is a flow chart of video shadow detection of the present invention.
FIG. 3 is a diagram of the effects of an example of the invention processing video. The method comprises the steps of (a) inputting a video stream, (b) depth information corresponding to the video stream, (c) a shadow confidence estimation result of the video stream, (d) an optimized shadow detection result (confidence) of the video stream, and (e) a video stream result after shadow removal.
Detailed Description
The present invention will be described in further detail below with reference to examples of implementation and the accompanying drawings, but the embodiments of the present invention are not limited thereto.
Referring to fig. 1, a flow chart of the present invention, a video shadow elimination method, includes the following steps:
step 1, for an input video stream V, acquiring depth information thereof: for a video shot on site, acquiring depth information of the video in real time by using Kinect V2; for the existing video, the depth information of each frame of the video is estimated by using a deep learning method. As shown in fig. 3(a), (b), which are an input video frame and a corresponding depth map in the example, respectively.
And 2, filtering each frame I in the video by using a texture filtering operator, reducing the influence of small-scale texture on shadow detection, and simultaneously keeping original shadow information.
Step 3, for each filtered video frame TiSelect its corresponding associated video stream { T }i-2,Ti-1,Ti,Ti+1,Ti+2Finding out the initial shadow confidence and brightness confidence of each pixel point in the video frame, and optimizing the shadow confidence of each frame to obtain the final video shadow detection result. The step 3 comprises the following steps:
step 3.1, utilizing the depth information of each video frame and combining camera parameters to carry out point cloud estimation; after the point cloud information of each pixel point is obtained, a k-d tree is constructed by using the point cloud information, the most similar 300 point clouds are found for the point clouds of each pixel, and the normal information of the space tangent plane where the pixel point is located is calculated by using the similar point clouds.
Step 3.2, as shown in the flow chart of fig. 2 for shadow detection, for each filtered video frame TiFinding out the space-time local neighborhood pixel point q E R corresponding to each pixel point p in the related video streampAnd calculating the chroma similarity, the spatial distance similarity and the normal similarity of the pixel point p and all the neighbor points q, and multiplying the three similarities to obtain the final characteristic similarity alphapq. Wherein R ispWhich is a spatio-temporal neighborhood of pixel point p, typically a 50 x 5 spatio-temporal pixel block.
Step 3.3, comparing each pixel point p with all neighbor pixel points in the spatial neighborhood thereof by utilizing the similarity between the pixel pointsq∈RpThe weighted average value m (p) of the image intensity, and the shadow confidence coefficient of each pixel point is estimatedAnd confidence of brightness
Fig. 3(c) shows the initial shadow confidence estimation result corresponding to each frame in the example.
Step 3.4, optimization equation is utilizedAnd obtaining the final optimization result of the shadow confidence coefficient S. Wherein, the first two terms are data constraint terms, and the third term is a smoothing term; n is the number of pixel points in the image, SkIs the shadow confidence optimization result of the kth pixel point, omegakIs the local window, s, where the k-th pixel is locatediAnd sjIs window omegakThe shadow confidence degree optimization results corresponding to the two pixel points i and j in the window are used for the pixel points w in the smooth windowijIs the matching Laplacian value of the i and j points in the neighborhood. In each video frame, the result of the shadow detection is constrained by using a Laplacian operator in combination with the result of the initial shadow confidence and the brightness confidence calculated in step 3.3, and gradient correlation smoothing is performed by using a Laplacian matting operator (matting Laplacian) to obtain an optimized shadow detection result. Example results are shown in FIG. 3(d), which is the final shadow confidence optimization result.
Step 4, utilizing total variation of shadow confidence coefficient and brightness confidence coefficientAnd inherent variation ofThe shadow boundary confidence is further calculated:
wherein,andfor total variation and inherent variation, respectively, ∈ is usually set to 0.001 in the experiment, and in order to prevent the denominator from being 0:
where R (p) is a 7 × 7 rectangular neighborhood with p as the center point,representing a weight function defined by gaussian filtering,andq-point shadow confidence and luminance confidence respectively,representing confidence in shadingOr confidence of brightnessPerforming partial derivatives in the x or y direction, total variationThe inherent variation is obtained by taking the absolute value of the partial derivative result and then carrying out Gaussian filteringThe method is obtained by performing Gaussian filtering on the partial derivative result and then taking an absolute value.
And 5, after obtaining the shadow detection result of each frame, decomposing the current frame image I into an image F without shadow and a shadow factor beta by using a shadow image model beta as I/F, wherein the two are unknown quantities. The invention constructs the following optimization equation, and restrains and optimizes each frame based on the video stream to obtain the final video shadow elimination result F and the shadow factor beta.
E(F,β)=Edata(F,β)+λ1Esmooth(F,β)
+λ2Echromaticity(β)+λ3Econst(β)
In experimental operation, we will generally refer to the parameter λ1,λ2And λ3Set to 1, 0.5 and 1, respectively.
Data item Edata=ωiw∑c∈{R,G,B}ωc·|Ic-Fc·βc|2For constraining each data item of the current frame, using the shadow model to process the data I under different color channelsc,Fc,βcAnd (6) carrying out constraint. Wherein, ω iscThe weight is restricted for three color channels of RGB, and the weight of each channel is { omegaR,ωG,ωB}={0.299,0.587,0.144}。ωiwIs a weight related to the intensity of the pixel,ωiw=1-ωintensity1- | I (x) |, where ω isintensityFor the adjustable parameter, i (x) is the pixel intensity of pixel point x.
Smoothing term Esmooth=ESF+γESMWhere γ is the balance factor, and is typically set to 1 in experimental work. Based on the assumptions: on the same spatial plane, pixel points with similar chrominance information, normal information and three-dimensional point cloud position information should have similar pixel values (color values) after the shadow is eliminated. And carrying out smooth constraint on the image F after the shadow is removed:
the first term is smooth constraint between adjacent pixel points of the current frame, the second term utilizes non-shadow pixel points with similar characteristics in the video stream to constrain shadow pixels, and R issFor shadow pixel points obtained by shadow detection in the current frame,and representing the point set of all non-shaded pixel points in the T frame in the spatial-temporal local neighborhood, wherein T is the total frame number in the current video stream.
ESM=∑p∈I(1-|Cbound(p)|)∑q∈Rp||βp-βq||2Using estimated shadow boundary confidence CboundThe shading factor beta is smoothly constrained.
Echromaticity(β)=||c(p)-cF(p)||2Carrying out chroma consistent constraint on the original video frame and the video frame with the shadow eliminated by using the assumption that the chroma of the image is not influenced by illumination change, wherein c is the chroma of the current frame I, and c is the chroma of the current frame IFThe chromaticity of the shadow elimination result F.
The color of the pixel points in the non-shadow area of the image after the assumed shadow is eliminated is kept unchanged,i.e. the shading factor approaches 1, for the non-shaded area NbAnd (4) carrying out constraint, wherein the shadow pixel points are points with confidence coefficient greater than 0.1, and the non-shadow pixel areas are pixel point sets except all shadow points and neighbor pixel points thereof.
And 6, because the equation contains two unknowns of the image F without shadow and the shadow factor beta, the algorithm adopts an iterative optimization method to solve, the initial value of the F is I, the initial value of the beta is the shadow confidence coefficient S, the final result is calculated through iterative optimization, and the maximum iteration frequency is 1000. As shown in fig. 3(e), the shadow elimination result of the example is shown.
The specific embodiments described herein are merely illustrative of the spirit of the invention. Various modifications or additions may be made to the described embodiments or alternatives may be employed by those skilled in the art without departing from the spirit or ambit of the invention as defined in the appended claims.
Claims (5)
1. A method for detecting and eliminating complex scene video shadow is characterized by comprising the following steps:
step 1, for an input video stream V, obtaining depth information of the input video stream V;
step 2, inputting a video frame I into each frame, and performing filtering processing by using a texture filtering operator to reduce texture influence and simultaneously keep shadow information in the video frame;
step 3, for each filtered video frame TiSelecting adjacent related video streams, finding out the initial shadow confidence and the brightness confidence of each pixel point in the video frames, and optimizing the shadow confidence of each frame to obtain a final video shadow detection result;
step 4, further calculating the confidence coefficient of the shadow boundary by using the total variation and the inherent variation of the shadow confidence coefficient and the brightness confidence coefficient;
step 5, after obtaining the shadow detection result of each frame, decomposing the current frame image I into an image F without shadow and a shadow factor beta by using a shadow image model beta as I/F, and constructing a shadow removal optimization equation to constrain and optimize each frame;
the de-shadowing optimization equation in step 5 is,
E(F,β)=Edata(F,β)+λ1Esmooth(F,β)+λ2Echromaticity(β)+λ3Econst(β)
wherein the data item Edata=ωiw∑c∈{R,G,B}ωc·|Ic-Fc·βc|2For constraining each data item of the current frame, using the shadow model to process the data I under different color channelsc,Fc,βcMake a constraint in whichR,ωG,ωBThe constraint weight is the constraint weight of each color channel of RGB; pixel intensity weight ωiw=1-ωintensity1- | I (x) |, where ω isintensityFor adjustable parameters, i (x) is the pixel intensity of pixel point x;
smoothing term Esmooth=ESF+γESMWherein gamma is a balance factor, ESFThe deblocked image F is subjected to a smoothing constraint based on the following assumptions: on the same spatial plane, the pixel points with similar chrominance information, normal information and three-dimensional point cloud position information should have similar pixel values after the shadow is eliminated,
the first term is smooth constraint between adjacent pixel points of the current frame, the second term utilizes non-shadow pixel points with similar characteristics in the video stream to constrain shadow pixels, and R issFor shadow pixel points obtained by shadow detection in the current frame,representing a point set of all non-shadow pixel points in a T frame in a spatial-temporal local neighborhood, wherein T is the total frame number in the current video stream;
ESMusing estimated shadow edgesConfidence of boundary CboundThe shading factor β is smoothly constrained:
Echromaticity(β)=||c(p)-cF(p)||2carrying out chroma consistent constraint on the original video frame and the video frame with the shadow eliminated by using the assumption that the chroma of the image is not influenced by illumination change, wherein c is the chroma of the current frame I, and c is the chroma of the current frame IFThe chromaticity which is the shadow elimination result F;
the color of the pixel point of the image non-shadow area after the assumed shadow is eliminated is kept unchanged, namely the shadow factor approaches to 1, and the non-shadow area N is subjected tobConstraint is carried out, the non-shadow pixel area is a pixel point set except all shadow points and neighbor pixel points thereof, and the shadow pixel points are points with confidence coefficient greater than 0.1;
and 6, carrying out iterative optimization solution on the shadow removing optimization equation to obtain a final video shadow elimination result F and a shadow factor beta.
2. The method as claimed in claim 1, wherein the method comprises: the specific implementation of the step 3 comprises the following substeps:
step 3.1, performing point cloud estimation by using depth information of each video frame and combining camera parameters to obtain point cloud information of each pixel point, constructing a k-d tree by using the point cloud information, finding a plurality of point clouds which are most similar to the point cloud of each pixel point, and calculating normal information of a space tangent plane where the pixel point is located by using the similar point clouds;
step 3.2, for each filtered video frame TiCalculating the point q ∈ R of each pixel point p and the point q ∈ R in the spatial neighborhood by utilizing the Gaussian similaritypThe chroma similarity, the spatial distance similarity and the normal similarity of the color data, and thenMultiplying the three similarity degrees to obtain the final feature similarity degree alphapq(ii) a Wherein R ispIs the spatial neighborhood of the pixel point p;
step 3.3, comparing each pixel p with all neighbor pixels q e R in the spatial neighborhood by utilizing the similarity between the pixelspThe weighted average value of the image intensity estimates the shadow confidence of each pixel pointAnd confidence of brightness
Wherein, the intensity weighted average value of the neighborhood where the p point is locatedIpAnd IqRespectively representing the intensities of pixels p and q, sigma being an adjustable parameter, | RpI represents the neighborhood region RpThe number of the pixel points in (1);
step 3.4, in each video frame, utilizing a laplacian operator, and combining the results of the initial shadow confidence coefficient and the brightness confidence coefficient calculated in the step 3.3, constructing an optimization equation to obtain a final shadow detection result S:
wherein, the first two terms are data constraint terms, and the third term is a smoothing term; n is a pixel in the imageNumber of dots, SkIs the shadow confidence optimization result of the kth pixel point, omegakIs the local window where the kth pixel point is located, SiAnd SjIs window omegakThe shadow confidence degree optimization results corresponding to the two pixel points i and j in the window are used for the pixel points w in the smooth windowijIs the matching Laplacian value of the i and j points in the neighborhood.
3. The method as claimed in claim 2, wherein the method comprises the steps of: the formula for calculating the confidence of the shadow boundary in step 4 is,
wherein,andrespectively representing the total variation and the inherent variation of the p-point confidence coefficient graph, wherein the epsilon is a constant;
wherein R (p) is a rectangular neighborhood with p as the center point,representing a weight function defined by gaussian filtering,andq-point shadow confidence and luminance confidence respectively,is a sign of the partial derivative,representing confidence in shadingOr confidence of brightnessA partial derivation in the x or y direction is performed.
4. The method as claimed in claim 3, wherein the method comprises the steps of: and 6, solving by adopting an iterative optimization method, setting the initial value of F as I and the initial value of beta as a shadow confidence coefficient S, and calculating a final result through iterative optimization, wherein the maximum iteration number is 1000.
5. The method as claimed in claim 1, wherein the method comprises: in the step 1, for a video shot on site, acquiring depth information of the video in real time by using Kinect V2; for the existing video, the depth information of each frame of the video is estimated by using a deep learning method.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910523329.9A CN110349099B (en) | 2019-06-17 | 2019-06-17 | Complex scene video shadow detection and elimination method |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910523329.9A CN110349099B (en) | 2019-06-17 | 2019-06-17 | Complex scene video shadow detection and elimination method |
Publications (2)
Publication Number | Publication Date |
---|---|
CN110349099A CN110349099A (en) | 2019-10-18 |
CN110349099B true CN110349099B (en) | 2021-04-02 |
Family
ID=68182147
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201910523329.9A Active CN110349099B (en) | 2019-06-17 | 2019-06-17 | Complex scene video shadow detection and elimination method |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN110349099B (en) |
Families Citing this family (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112419196B (en) * | 2020-11-26 | 2022-04-26 | 武汉大学 | Unmanned aerial vehicle remote sensing image shadow removing method based on deep learning |
CN112686936B (en) * | 2020-12-18 | 2023-08-04 | 北京百度网讯科技有限公司 | Image depth completion method, apparatus, computer device, medium, and program product |
CN112598592A (en) * | 2020-12-24 | 2021-04-02 | 广东博智林机器人有限公司 | Image shadow removing method and device, electronic equipment and storage medium |
CN113361360B (en) * | 2021-05-31 | 2023-07-25 | 山东大学 | Multi-person tracking method and system based on deep learning |
CN113378775B (en) * | 2021-06-29 | 2023-04-07 | 武汉大学 | Video shadow detection and elimination method based on deep learning |
CN114782616B (en) * | 2022-06-20 | 2022-09-20 | 北京飞渡科技有限公司 | Model processing method and device, storage medium and electronic equipment |
CN116704316A (en) * | 2023-08-03 | 2023-09-05 | 四川金信石信息技术有限公司 | Substation oil leakage detection method, system and medium based on shadow image reconstruction |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9430715B1 (en) * | 2015-05-01 | 2016-08-30 | Adobe Systems Incorporated | Identifying and modifying cast shadows in an image |
CN107203975A (en) * | 2017-04-18 | 2017-09-26 | 南京航空航天大学 | Shadow removal method based on YCbCr color spaces and topology cutting |
CN107808366A (en) * | 2017-10-21 | 2018-03-16 | 天津大学 | A kind of adaptive optical transfer single width shadow removal method based on Block- matching |
CN109064411A (en) * | 2018-06-13 | 2018-12-21 | 长安大学 | A kind of pavement image based on illumination compensation removes shadow method |
Family Cites Families (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106339995A (en) * | 2016-08-30 | 2017-01-18 | 电子科技大学 | Space-time multiple feature based vehicle shadow eliminating method |
CN107038690B (en) * | 2017-03-27 | 2020-04-28 | 湘潭大学 | Moving shadow removing method based on multi-feature fusion |
-
2019
- 2019-06-17 CN CN201910523329.9A patent/CN110349099B/en active Active
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9430715B1 (en) * | 2015-05-01 | 2016-08-30 | Adobe Systems Incorporated | Identifying and modifying cast shadows in an image |
CN107203975A (en) * | 2017-04-18 | 2017-09-26 | 南京航空航天大学 | Shadow removal method based on YCbCr color spaces and topology cutting |
CN107808366A (en) * | 2017-10-21 | 2018-03-16 | 天津大学 | A kind of adaptive optical transfer single width shadow removal method based on Block- matching |
CN109064411A (en) * | 2018-06-13 | 2018-12-21 | 长安大学 | A kind of pavement image based on illumination compensation removes shadow method |
Non-Patent Citations (1)
Title |
---|
Shadow Removal from Single RGB-D Images;Yao Xiao et al;《2014 IEEE Conference on Computer Vision and Pattern Recognition》;20140925;3011-3018页 * |
Also Published As
Publication number | Publication date |
---|---|
CN110349099A (en) | 2019-10-18 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN110349099B (en) | Complex scene video shadow detection and elimination method | |
US10574905B2 (en) | System and methods for depth regularization and semiautomatic interactive matting using RGB-D images | |
CN109859171B (en) | Automatic floor defect detection method based on computer vision and deep learning | |
CN104794688B (en) | Single image to the fog method and device based on depth information separation sky areas | |
JP4074062B2 (en) | Semantic object tracking in vector image sequences | |
KR102138950B1 (en) | Depth map generation from a monoscopic image based on combined depth cues | |
US9042662B2 (en) | Method and system for segmenting an image | |
CN108377374B (en) | Method and system for generating depth information related to an image | |
JP2009500752A (en) | Cut and paste video objects | |
WO2018227882A1 (en) | A priori constraint and outlier suppression based image deblurring method | |
WO2018053952A1 (en) | Video image depth extraction method based on scene sample library | |
CN111310768B (en) | Saliency target detection method based on robustness background prior and global information | |
CN111161219B (en) | Robust monocular vision SLAM method suitable for shadow environment | |
CN105898111A (en) | Video defogging method based on spectral clustering | |
Li et al. | Optimal seamline detection in dynamic scenes via graph cuts for image mosaicking | |
He et al. | Single-image shadow removal using 3D intensity surface modeling | |
Shen et al. | Re-texturing by intrinsic video | |
Sooknanan et al. | Improving underwater visibility using vignetting correction | |
CN106296740B (en) | A kind of target fine definition tracking based on low-rank sparse expression | |
CN111932469A (en) | Significance weight quick exposure image fusion method, device, equipment and medium | |
CN113781329B (en) | Fog removing method for remote sensing image | |
Liu et al. | Temporal-consistency-aware video color transfer | |
CN109886901A (en) | A kind of nighttime image enhancing method decomposed based on multi-path | |
KR101631023B1 (en) | Neighbor-based intensity correction device, background acquisition device and method thereof | |
Kuse et al. | Graph modelling of 3D geometric information for color consistency of multiview images |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |