CN114742847A - Light field matting method and device based on hollow angle consistency - Google Patents

Light field matting method and device based on hollow angle consistency Download PDF

Info

Publication number
CN114742847A
CN114742847A CN202210405016.5A CN202210405016A CN114742847A CN 114742847 A CN114742847 A CN 114742847A CN 202210405016 A CN202210405016 A CN 202210405016A CN 114742847 A CN114742847 A CN 114742847A
Authority
CN
China
Prior art keywords
light field
image
alpha
formula
map
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210405016.5A
Other languages
Chinese (zh)
Inventor
何迪
刘畅
刘天艺
邱钧
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Information Science and Technology University
Original Assignee
Beijing Information Science and Technology University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Information Science and Technology University filed Critical Beijing Information Science and Technology University
Priority to CN202210405016.5A priority Critical patent/CN114742847A/en
Publication of CN114742847A publication Critical patent/CN114742847A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10052—Images from lightfield camera

Abstract

The invention discloses a method and a device for matting based on a hollow angle consistency light field, which comprises the following steps: step 1, extracting a central subaperture image of light field data and calculating the parallax of the central subaperture image; step 2, calculating a central subaperture image alpha diagram; step 3, transmitting the central subaperture image alpha image through the light field alpha image transmission model to obtain a light field alpha imageu,v(x, y). By adopting the method provided by the invention, accurate and consistent light field matting can be realized and the hollow angle consistency of the light field matting can be evaluated.

Description

Light field matting method and device based on hollow angle consistency
Technical Field
The invention relates to the technical field of optical information processing, in particular to a method and a device for matting an optical field based on hollow angle consistency.
Background
The light field describes the distribution of light rays in free space, light field data captures the radiation of light rays from all directions, the spatial information, the angle information and the intensity information of the light rays are recorded, the light field imaging can capture richer visual information, the angle information lost in the traditional photography is compensated, the stronger capability is provided for scene understanding, and the performance of the traditional computer vision problem is improved.
One 4D light field data can be represented by a 2D sub-aperture image array, so performing a 2D image matting algorithm on sub-aperture images by sub-aperture images can be considered to achieve matting of the 4D light field data to some extent. The method of operating by sub-aperture images does not consider the relation between different sub-aperture images, the hollow angle consistency of the light field data cannot be kept, and the obtained result is difficult to meet the requirements of downstream technology and application. And the size of light field data is large, a large amount of redundant information exists among the sub-aperture images, too much redundant calculation exists in simple sub-aperture-by-sub-aperture image processing, and the efficiency is too low.
Disclosure of Invention
The invention aims to provide a method for null angle consistency optical field matting and an evaluation index thereof, which can avoid the damage to the null angle consistency of optical field data in the matting process and reduce the calculation redundancy.
In order to achieve the above object, the present invention provides a method for matting based on a hollow angle consistency, which comprises:
step 1, extracting a central subaperture image of light field data and calculating the parallax of the central subaperture image;
step 2, calculating a central subaperture image alpha diagram;
and 3, transmitting the central subaperture image alpha image through the light field alpha image transmission model provided by the formula (1) to obtain the light field alpha imageu,v(x,y)；
Figure BDA0003601452550000011
In the formula (I), the compound is shown in the specification,
Figure BDA0003601452550000012
as a central viewpoint (u)0,v0) In the alpha map of the lower sub-aperture image, disp (x, y) is the parallax corresponding to the image point (x, y) in the central sub-aperture image, and Δ u and Δ v are respectively an arbitrary viewpoint (u, v) and a central viewpoint (u, v)0,v0) Distances in the u, v directions.
Further, the calculation formula of the disparity map disp (x, y) in step 2 is formula (2):
disp(x,y)＝F/Z(x,y)+F (2)
where F is the focal length of the camera and Z (x, y) is the depth map of the scene.
Further, the method for acquiring the central sub-aperture image alpha map in step 2 specifically includes:
step 21, the confidence q of any pixel i in the disparity map disp (x, y)iIs set as formula (3):
Figure BDA0003601452550000021
in the formula, piD ═ max (disp (x, y)) is the maximum value in the disparity map, and D ═ min (disp (x, y)) is the minimum value in the disparity map, for the disparity value of any pixel i in the disparity map disp (x, y);
step 22, setting a confidence threshold t1And t2When the confidence degree q isiGreater than t1When the confidence coefficient q is less than the threshold, the region belongs to the foregroundiLess than t2If the area belongs to the background, and the other areas are uncertain areas in the trimap image, obtaining the trimap image described by the formula (4):
Figure BDA0003601452550000022
step 23, converting the trimap image into I1And I2，I1The pixel at the middle foreground and background mark is 1, the pixel value at the rest part is 0, and I is calculated1Converting into a sparse diagonal matrix which is Ds，I2The pixel value of the middle foreground mark is 1, the other pixel values are 0, and the pixel values are converted into column vectors which are bs：
Figure BDA0003601452550000023
Figure BDA0003601452550000024
Step 24, solving an alpha map of the central subaperture image by the formula (7):
α＝λ(L+λ)-1bs (7)
in the formula, λ is a weight coefficient of the mark, and L is a laplacian matrix of the matte.
Further, the method for coherent light field matting based on the hollow angle further comprises:
and 4, evaluating the hollow angle consistency of the light field cutout.
Further, in step 4, specifically, the hollow angle consistency evaluation index S-cons obtained in the spatial domain provided by equation (8) is used to evaluate the hollow angle consistency in the light field alpha diagram: the value range of the index S-cons is (0-1), the closer the index S-cons is to 1, the smaller the variance of the representative boundary area is, the better the consistency of the light field data is, and the closer the index S-cons is to 0, the worse the consistency of the light field data is;
Figure BDA0003601452550000031
in the formula, NEPIThe number of EPIs (Epipolar Plane images, EPIs) which are the light field alpha maps, EjFor the corresponding jth EPI, there are l boundary regions, denoted as Ω, in each EPIl，NlIs omega in the jth EPIlTotal number of pixels of the area, EjiFor the ith pixel in the jth EPI of the light field alpha map,
Figure BDA0003601452550000032
is omega in the jth EPIlMean of pixel values at the boundary;
or, in the step 4, the hollow angle consistency in the light field alpha diagram is evaluated by the hollow angle consistency evaluation index F-cons obtained in the frequency domain and provided by the formula (9): the value range of the index F-cons is (0-1), the larger the index F-cons is, the smaller the occupied range of the representative support is, the better the null angle consistency of the light field data is, and otherwise, the worse is;
Figure BDA0003601452550000033
in the formula:
Figure BDA0003601452550000034
wherein, L' (Ω)u,Ωv,Ωx,Ωy) Is a spectrogram L (omega)u,Ωv,Ωx,Ωy) After binarization, the spectrogram is counted, wherein epsilon is a smaller number, M is the number of integral pixels of the spectrogram, and omegau、Ωv、Ωx、ΩyAre the coordinate axes of the light field spectrogram respectively.
The invention also provides a device for matting the light field based on the hollow angle consistency, which comprises:
a central sub-aperture image extraction unit for extracting a central sub-aperture image of the light field data;
a calculation unit for calculating a parallax of pixels in a central sub-aperture image of the light field data and a central sub-aperture image alpha map;
the light field matting unit is used for transmitting the central subaperture image alpha image through the light field alpha image transmission model provided by the formula (1) to acquire the light field alpha imageu,v(x,y)；
Figure BDA0003601452550000035
In the formula (I), the compound is shown in the specification,
Figure BDA0003601452550000041
as a central viewpoint (u)0,v0) In the alpha map of the lower sub-aperture image, disp (c, y) is the parallax corresponding to the image point (x, y) in the central sub-aperture image, and Δ u and Δ v are the arbitrary viewpoint (u, v) and the central viewpoint (u, v), respectively0,v0) Distances in the u, v directions.
Further, the calculation unit includes:
a disparity calculation subunit for calculating a disparity map disp (x, y) from a depth map of the light field data by equation (2):
disp(x,y)＝F/Z(x,y)+F (2)
where F is the focal length of the camera and Z (x, y) is the depth map of the scene.
Further, the calculation unit includes:
a central sub-aperture image alpha map calculating subunit, configured to obtain, according to the central sub-aperture image of the light field data, a central sub-aperture image alpha map by a specific alpha closed solution method as follows:
step 21, the confidence q of any pixel i in the disparity map disp (x, y)iIs set as formula (3):
Figure BDA0003601452550000042
in the formula, piD ═ max (disp (x, y)) is the maximum value in the disparity map, and D ═ min (disp (x, y)) is the minimum value in the disparity map;
step 22, setting a confidence threshold t1And t2When the confidence degree q isiGreater than t1When the confidence coefficient q is less than the threshold, the region belongs to the foregroundiLess than t2If the area belongs to the background, and the other areas are uncertain areas in the trimap image, obtaining the trimap image described by the formula (4):
Figure BDA0003601452550000043
step 23, converting the trimap image into I1And I2，I1The pixel at the middle foreground and background mark is 1, the pixel value at the rest part is 0, and I is calculated1Converting into a sparse diagonal matrix which is Ds，I2The pixel value of the middle foreground mark is 1, the other pixel values are 0, and the middle foreground mark is converted into a column vector, namelyIs b iss：
Figure BDA0003601452550000044
Figure BDA0003601452550000045
Step 24, solving an alpha map of the central subaperture image by the formula (7):
α＝λ(L+λ)-1bs (7)
in the formula, λ is a weight coefficient of the mark, and L is a laplacian matrix of the matting.
Further, the device for consistent light field matting based on hollow angles further comprises:
and the evaluation unit is used for evaluating the hollow angle consistency in the optical field matting.
Further, the evaluation unit specifically includes a spatial domain evaluation subunit or a frequency domain evaluation subunit:
the spatial domain evaluation subunit is used for evaluating the light field data null angle consistency in the light field alpha diagram through a null angle consistency index S-cons obtained in the spatial domain and provided by the formula (8): the value range of the index S-cons is (0-1), the closer the index S-cons is to 1, the smaller the variance of the representative boundary area is, the better the null angle consistency of the light field data is, and the closer the index S-cons is to 0, the worse the consistency of the light field data is;
Figure BDA0003601452550000051
in the formula, NEPIThe number of EPIs (Epipolar Plane images, EPIs) which are the light field alpha maps, EjFor the corresponding jth EPI, there are l boundary regions, denoted as Ω, in each EPIl，NlIs omega in the jth EPIlTotal number of pixels of the area, EjiFor the ith pixel in the jth EPI of the light field alpha map,
Figure BDA0003601452550000052
is omega in the jth EPIlMean of pixel values at the boundary;
the frequency domain evaluation subunit is used for providing a null angle consistency evaluation index F-cons acquired in a frequency domain through an equation (9) and evaluating the null angle consistency of the light field data in the light field alpha diagram: the value range of the index F-cons is (0-1), the larger the index F-cons is, the smaller the occupied range of the representative support is, the better the null angle consistency of the light field data is, and otherwise, the worse is;
Figure BDA0003601452550000053
in the formula:
Figure BDA0003601452550000054
wherein, L' (Ω)u,Ωv,Ωx,Ωy) Is a spectrogram L (omega)u,Ωv,Ωx,Ωy) After binarization, the spectrogram is counted, wherein epsilon is a smaller number, M is the number of integral pixels of the spectrogram, and omegau、Ωv、Ωx、ΩyAre the coordinate axes of the light field spectrogram respectively.
Due to the adoption of the technical scheme, the invention has the following advantages:
by adopting the method based on the hollow angle consistency light field matting and the evaluation index thereof, the accurate and consistent light field matting is realized, the hollow angle consistency of the light field data is ensured, the calculation redundancy is reduced, and the established evaluation index can effectively evaluate the hollow angle consistency of the obtained light field data.
Drawings
Fig. 1 is a schematic flow chart of a method for matting light field based on null angle consistency according to an embodiment of the present invention.
Fig. 2 is a schematic flow chart of a method for evaluating light field matting based on null angle consistency according to an embodiment of the present invention.
Fig. 3 is an alpha diagram disparity map of the central subaperture image.
Fig. 4 is a schematic view of a propagation model.
Fig. 5 is a schematic diagram of a spatial domain evaluation index.
Fig. 6 is a schematic diagram of frequency domain evaluation indexes.
Detailed Description
In the drawings, the same or similar reference numerals are used to denote the same or similar elements or elements having the same or similar functions. Embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
As shown in fig. 1, the method for matting light field based on hollow angle consistency according to the embodiment of the present invention includes:
step 1, extracting a central subaperture image of light field data and calculating the parallax of the central subaperture image.
And 2, calculating an alpha map of the central subaperture image.
Step 3, transmitting the central subaperture image alpha image through the light field alpha image transmission model provided by the formula (1) to obtain a light field alpha imageu,v(x,y)；
Figure BDA0003601452550000061
In the formula (I), the compound is shown in the specification,
Figure BDA0003601452550000062
as a central viewpoint (u)0,v0) In the alpha map of the lower sub-aperture image, disp (x, y) is the parallax corresponding to the image point (x, y) in the central sub-aperture image, and Δ u and Δ v are respectively an arbitrary viewpoint (u, v) and a central viewpoint (u, v)0,v0) Distances in the u, v directions.
According to the propagation model constructed in the step 3, the alpha map in the step 2 is propagated to obtain light field alpha maps at 9 × 9 viewpoints, for example, fig. 4 (a) shows the alpha map 2 and the parallax map 3 of the central sub-aperture image, and fig. 4 (b) shows the alpha map 4 of the sub-aperture images at 9 × 9 viewpoints obtained by propagation.
In the invention, an optical field matting propagation model meeting the hollow angle consistency is established. The model is established on the basis of the hollow angle coupling property of four-dimensional light field data, a high-quality light field alpha Image with better hollow angle consistency can be quickly obtained, in order to realize consistent light field matting, a Cho and other people light field matting method can be used, and the hollow angle consistency of the estimated alpha Image in a sub-aperture Image is enhanced by introducing smooth constraint on an Epipolar Plane Image (EPI).
In one embodiment, in step 2, as shown in a in fig. 3, from the central sub-aperture image of the light field data, a disparity map disp (x, y) of the light field data is calculated using equation (2), since such an inverse relationship between depth and disparity is satisfied:
disp(x,y)＝F/Z(x,y)+F (2)
where F is the focal length of the camera and Z (x, y) is the depth map of the scene.
The method for estimating the depth of the rapid light field image based on the multi-view stereo matching can be provided by Mishiba K and the like. Depth can also be estimated by the occlusion perception based depth estimation algorithm proposed by Wang TC et al. After the depth map exists, the disparity map can be obtained through the conversion of the formula (2).
In an embodiment, as shown in b in fig. 3, the method for obtaining the central sub-aperture image alpha map in step 2 specifically includes:
step 21, in order to automatically acquire the trimap image of the central sub-aperture image and solve an alpha image, assuming that the parallax image of the light field data is accurate, introducing a confidence level into the parallax image and selecting a threshold, and setting an accurate foreground, a background and an uncertain area according to the confidence level threshold, wherein the confidence level of the parallax image is set as follows:
Figure BDA0003601452550000071
in the formula, piIs the disparity value of any pixel i in the disparity map disp (x, y), the confidence q at this pixel pointiRepresenting the probability that the point belongs to the foreground, D ═ max (disp (x, y)) is the most significant in the disparity mapLarge value, d ═ min (disp (x, y)) is the minimum value in the disparity map. Setting a threshold t1And t2When the confidence is greater than t1When the confidence coefficient is less than t, the part of the area is considered to belong to the foreground2Then, considering the part of the area as the background, and the other areas as the uncertain areas in the three-segment map, the three-segment map can be obtained:
Figure BDA0003601452550000072
in the formula, t1And t2Is a point on a straight line formed by the minimum and maximum confidence values, and t1And t2The selection of (1) is related to the ratio of the respective depth ranges of the foreground and the background in the scene in the whole depth. When the current depth of field is large in range ratio, t1The larger, t2The smaller; when the background depth range is large, t1The smaller, t2The larger.
Obtaining trisection graph and converting the trisection graph into I1And I2：I1The pixel at the middle foreground and background mark is 1, the pixel value at the rest part is 0, and I is calculated1Converting into a sparse diagonal matrix Ds；I2The pixel value of the middle foreground mark is 1, the other pixel values are 0, and the pixel values are converted into column vectors which are bs. Of course, manual marking I may be used1And I2The marking can also be performed in an existing automatic manner.
Figure BDA0003601452550000081
Figure BDA0003601452550000082
Step 22, based on the obtained DsAnd bsThe alpha map of the central subaperture image can be solved by equation (7):
α＝λ(L+λ)-1bs (7)
in the formula, λ is a weighting coefficient of the mark, and a larger number is generally selected, for example, 100, and L is a laplacian matrix of the matting.
In addition, the central sub-aperture image can be subjected to matting by Random Walk matting, Poisson matting and other methods.
In one embodiment, as shown in fig. 5 and 6, the method for coherent light field matting based on a null angle further comprises:
step 4, as shown in fig. 5, object points located at the boundary in the scene form a bundle of straight lines in the EPI of the alpha map, and the smaller the difference of the whole straight lines is, the better the consistency at the boundary is, so the region located at the boundary is marked as Ω, and the consistency of the obtained light field data is measured by calculating the variance of pixel values at the region of Ω, and if the variance is smaller, the consistency is better. Alpha chart for light fieldu,vThe uniformity of the hollow angle of (x, y) was evaluated.
Step 4, specifically, obtaining an evaluation index S-cons of the hollow angle consistency in the space domain provided by the formula (8), and evaluating the hollow angle consistency of the light field data in the light field alpha diagram: the value range of the index S-cons is (0-1), the closer the index S-cons is to 1, the smaller the variance of the representative boundary area is, the better the null angle consistency of the light field data is, and the closer the index S-cons is to 0, the worse the null angle consistency of the light field data is;
Figure BDA0003601452550000083
in the formula, NEPIThe number of EPIs (Epipolar Plane images, EPIs) which are the light field alpha maps, EjFor the corresponding jth EPI, there are l boundary regions, denoted as Ω, in each EPIl，NlIs omega in the jth EPIlTotal number of pixels of the area, EjiFor the ith pixel in the jth EPI of the light field alpha map,
Figure BDA0003601452550000091
is omega in the jth EPIlMean of pixel values at the boundary;
or, as shown in a in fig. 6, the light field data spectrogram satisfies the shape of hypercone, so the null angle consistency of the light field data can be embodied by the range of the intra-spectrum support, and if the null angle consistency of the light field data is better, the linear shape can be better maintained in the frequency domain, the smaller the range of the intra-spectrum support is, the smaller the proportion of the intra-spectrum support is; on the contrary, if the consistency is damaged, the structure of hypercone is damaged, the linear structure can not be maintained in the frequency domain, and the larger the support range is, the larger the proportion of the whole spectrogram is.
Based on this, step 4 provides binarization of the spectrogram by equation (9) as shown in b in fig. 6, and then evaluates the null angle consistency of the light field data in the light field alpha diagram by the null angle consistency evaluation index F-cons obtained in the frequency domain by equation (10): the value range of the index F-cons is (0-1), the larger the index F-cons is, the smaller the occupied range of the representative support is, the better the null angle consistency of the light field data is, and otherwise, the worse is;
Figure BDA0003601452550000092
in the formula:
Figure BDA0003601452550000093
wherein, L' (Ω)u,Ωv,Ωx,Ωy) Is a spectrogram L (omega)u,Ωv,Ωx,Ωy) After binarization, the spectrogram is counted, wherein epsilon is a smaller number, M is the number of integral pixels of the spectrogram, and omegau、Ωv、Ωx、ΩyAre the coordinate axes of the light field spectrogram respectively.
The invention provides a quantitative evaluation index of a light field alpha image, the establishment of the index is based on the essential attribute of light field data, the hollow angle consistency of the light field alpha image after propagation can be objectively evaluated, Cho et al also provide the index for evaluating the hollow angle consistency of the light field data, and MSE, SAD and the like can also be used for evaluation.
The light field matting device based on the null angle consistency provided by the embodiment of the invention comprises a central subaperture image extraction unit, a calculation unit and a light field matting unit, wherein:
the central sub-aperture image extraction unit is used for extracting a central sub-aperture image of the light field data.
The computing unit is used for computing parallax of pixels in the central sub-aperture image of the light field data and an alpha map of the central sub-aperture image.
The light field matting unit is used for transmitting the central subaperture image alpha image through the light field alpha image transmission model provided by the formula (1) to obtain the light field alpha imageu,v(x,y)；
Figure BDA0003601452550000101
In the formula (I), the compound is shown in the specification,
Figure BDA0003601452550000102
as a central viewpoint (u)0,v0) In the alpha map of the lower sub-aperture image, disp (c, y) is the parallax corresponding to the image point (x, y) in the central sub-aperture image, and Δ u and Δ v are the arbitrary viewpoint (u, v) and the central viewpoint (u, v), respectively0,v0) Distances in the u, v directions.
In one embodiment, the calculation unit comprises a disparity calculation subunit configured to calculate, from the depth map of the light field data, a disparity map disp (x, y) by equation (2):
disp(x,y)＝F/Z(x,y)+F (2)
where F is the focal length of the camera and Z (x, y) is the depth map of the scene.
In one embodiment, the calculation unit includes a central sub-aperture image alpha map calculation subunit, and the central sub-aperture image alpha map calculation subunit is configured to obtain, according to the central sub-aperture image of the light field data, a central sub-aperture image alpha map by a specific alpha closed solution method as follows:
step 21, the confidence q of any pixel i in the disparity map disp (x, y)iIs set as formula (3):
Figure BDA0003601452550000103
in the formula, piD ═ max (disp (x, y)) is the maximum value in the disparity map, and D ═ min (disp (x, y)) is the minimum value in the disparity map;
step 22, setting a confidence threshold t1And t2When the confidence degree q isiGreater than t1When the confidence coefficient q is less than the threshold, the region belongs to the foregroundiLess than t2If the area belongs to the background, and the other areas are uncertain areas in the trimap image, obtaining the trimap image described by the formula (4):
Figure BDA0003601452550000104
step 23, converting the trimap image into I1And I2，I1The pixel at the middle foreground and background mark is 1, the pixel value at the rest part is 0, and I is calculated1Converting into a sparse diagonal matrix which is Ds，I2The pixel value of the middle foreground mark is 1, the other pixel values are 0, and the pixel values are converted into column vectors which are bs：
Figure BDA0003601452550000105
Figure BDA0003601452550000106
Step 24, solving an alpha map of the central subaperture image by the formula (7):
α＝λ(L+λ)-1bs (7)
in the formula, λ is a weight coefficient of the mark, and L is a laplacian matrix of the matte.
In one embodiment, the hollow-angle-based coherent light fieldThe matting device also comprises an evaluation unit which is used for mapping alpha to the light fieldu,vThe uniformity of the hollow angle of (x, y) was evaluated.
In one embodiment, the evaluation unit specifically includes a spatial domain evaluation subunit or a frequency domain evaluation subunit:
the spatial domain evaluation subunit is used for evaluating the light field data null angle consistency in the light field alpha diagram through a null angle consistency evaluation index S-cons obtained in the spatial domain and provided by the formula (8): the value range of the index S-cons is (0-1), the closer the index S-cons is to 1, the smaller the variance of the representative boundary area is, the better the null angle consistency of the light field data is, and the closer the index S-cons is to 0, the worse the null angle consistency of the light field data is;
Figure BDA0003601452550000111
in the formula, NEPIThe number of EPIs (Epipolar Plane images) which are the alpha diagrams of the light field, EjFor the corresponding jth EPI, there are l boundary regions, denoted as Ω, in each EPIl，NlIs omega in the jth EPIlTotal number of pixels of the area, EjiFor the ith pixel in the jth EPI of the light field alpha map,
Figure BDA0003601452550000112
is omega in the jth EPIlMean of pixel values at the boundary;
the frequency domain evaluation subunit is used for evaluating the light field data null angle consistency in the light field alpha diagram through a null angle consistency evaluation index F-cons obtained in a frequency domain and provided by an equation (9): the value range of the index F-cons is (0-1), the larger the index F-cons is, the smaller the occupied range of the representative support is, the better the null angle consistency of the light field data is, and otherwise, the worse is;
Figure BDA0003601452550000113
in the formula:
Figure BDA0003601452550000114
wherein, L' (Ω)u,Ωv,Ωx,Ωy) Is a spectrogram L (omega)u,Ωv,Ωx,Ωy) After binarization, the spectrogram is counted, wherein epsilon is a smaller number, M is the number of integral pixels of the spectrogram, and omegau、Ωv、Ωx、ΩyAre the coordinate axes of the light field spectrogram respectively.
Finally, it should be pointed out that: the above examples are only intended to illustrate the technical solution of the present invention, and not to limit it. Those of ordinary skill in the art will understand that: modifications can be made to the technical solutions described in the foregoing embodiments, or some technical features may be equivalently replaced; such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims (10)

1. A method for matting light field based on hollow angle consistency is characterized by comprising the following steps:
step 1, extracting a central subaperture image of light field data and calculating the parallax of the central subaperture image;
step 2, calculating a central sub-aperture image alpha image;
and 3, transmitting the central subaperture image alpha image through the light field alpha image transmission model provided by the formula (1) to obtain the light field alpha imageu，v(x，y)；
Figure FDA0003601452540000011
In the formula (I), the compound is shown in the specification,
Figure FDA0003601452540000012
as a central viewpoint (u)0，v0) The alpha map, disp (x, y) of the lower sub-aperture image corresponds to the image point (x, y) in the central sub-aperture imageParallax, Δ u and Δ v are an arbitrary viewpoint (u, v) and a central viewpoint (u, v), respectively0，v0) Distances in the u, v directions.
2. The method for null-angle-based consistent light field matting according to claim 1, wherein the formula for calculating the disparity map disp (x, y) in step 2 is formula (2):
disp(x，y)＝F/Z(x，y)+F (2)
where F is the focal length of the camera and Z (x, y) is the depth map of the scene.
3. The method for optical field matting based on null angle consistency according to claim 1 or 2, wherein the method for obtaining the central sub-aperture image alpha map in the step 2 specifically includes:
step 21, the confidence q of any pixel i in the disparity map disp (x, y)iIs set as formula (3):
Figure FDA0003601452540000013
in the formula, piD ═ max (disp (x, y)) is the maximum value in the disparity map, and D ═ min (disp (x, y)) is the minimum value in the disparity map, for the disparity value of any pixel i in the disparity map disp (x, y);
step 22, setting confidence threshold t1And t2When the confidence degree q isiGreater than t1When the confidence coefficient q is less than the threshold, the region belongs to the foregroundiLess than t2If the area belongs to the background, and the other areas are uncertain areas in the trimap image, obtaining the trimap image described by the formula (4):
Figure FDA0003601452540000014
step 23, converting the trimap image into I1And I2，I1The pixels at the middle foreground and background marks are 1, the pixel values at the rest positions are 0,will I1Converting into a sparse diagonal matrix which is Ds，I2The pixel value of the middle foreground mark is 1, the other pixel values are 0, and the pixel values are converted into column vectors which are bs：
Figure FDA0003601452540000021
Figure FDA0003601452540000022
Step 24, solving an alpha map of the central subaperture image by the formula (7):
α＝λ(L+λ)-1bs (7)
in the formula, λ is a weight coefficient of the mark, and L is a laplacian matrix of the matte.
4. The method of null-angle consistency-based light field matting according to any one of claims 1 to 3, further comprising:
and 4, evaluating the hollow angle consistency of the light field cutout.
5. The method for optical field matting based on null angle consistency according to claim 4, wherein the step 4 evaluates the null angle consistency in the optical field alpha diagram specifically by the null angle consistency evaluation index S-cons obtained in the spatial domain provided by the formula (8): the value range of the index S-cons is (0-1), the closer the index S-cons is to 1, the smaller the variance of the representative boundary area is, the better the consistency of the light field data is, and the closer the index S-cons is to 0, the worse the consistency of the light field data is;
Figure FDA0003601452540000023
in the formula, NEPIThe number of EPIs (Epipolar Plane images, EPIs) of the light field alpha map, EjFor the corresponding jth EPI, there are l boundary regions, denoted as Ω, in each EPIl，NlIs omega in the jth EPIlTotal number of pixels of the area, EjiFor the ith pixel in the jth EPI of the light field alpha map,
Figure FDA0003601452540000024
is omega in the jth EPIlMean of pixel values at the boundary;
or, in the step 4, the hollow angle consistency in the light field alpha diagram is evaluated by the hollow angle consistency evaluation index F-cons obtained in the frequency domain and provided by the formula (9): the value range of the index F-cons is (0-1), the larger the index F-cons is, the smaller the occupied range of the representative support is, the better the null angle consistency of the light field data is, and otherwise, the worse is;
Figure FDA0003601452540000025
in the formula:
Figure FDA0003601452540000031
wherein, L' (Ω)u，Ωv，Ωx，Ωy) Is a spectrogram L (omega)u，Ωv，Ωx，Ωy) After binarization, the spectrogram is counted, wherein epsilon is a smaller number, M is the number of integral pixels of the spectrogram, and omegau、Ωv、Ωx、ΩyRespectively, the coordinate axes of the light field spectrogram.
6. The utility model provides a device is scratched to light field based on hollow angle uniformity which characterized in that includes:
a central sub-aperture image extraction unit for extracting a central sub-aperture image of the light field data;
a calculation unit for calculating a parallax of pixels in a central sub-aperture image of the light field data and a central sub-aperture image alpha map;
the light field matting unit is used for transmitting the central subaperture image alpha image through the light field alpha image transmission model provided by the formula (1) to acquire the light field alpha imageu，v(x，y)；
Figure FDA0003601452540000032
In the formula (I), the compound is shown in the specification,
Figure FDA0003601452540000033
as a central viewpoint (u)0，v0) In the alpha map of the lower sub-aperture image, disp (x, y) is the parallax corresponding to the image point (x, y) in the central sub-aperture image, and Δ u and Δ v are respectively an arbitrary viewpoint (u, v) and a central viewpoint (u, v)0，v0) Distances in the u, v directions.
7. The null-angle-based coherent light field matting apparatus according to claim 5, wherein the calculation unit includes:
a disparity calculation subunit configured to calculate, from the depth map of the light-field data, a disparity map disp (x, y) by equation (2):
disp(x，y)＝F/Z(x，y)+F (2)
where F is the focal length of the camera and Z (x, y) is the depth map of the scene.
8. The null-angle-based coherent light field matting apparatus according to claim 5, wherein the calculation unit comprises:
a central sub-aperture image alpha map calculating subunit, configured to obtain, according to the central sub-aperture image of the light field data, a central sub-aperture image alpha map by a specific alpha closed solution method as follows:
step 21, the confidence q of any pixel i in the disparity map disp (x, y)iIs set as formula (3):
Figure FDA0003601452540000034
in the formula, piD ═ max (disp (x, y)) is the maximum value in the disparity map, and D ═ min (disp (x, y)) is the minimum value in the disparity map;
step 22, setting a confidence threshold t1And t2When the confidence degree q isiGreater than t1When the confidence coefficient q is less than the threshold, the region belongs to the foregroundiLess than t2If the area belongs to the background, and the other areas are uncertain areas in the trimap image, obtaining the trimap image described by the formula (4):
Figure FDA0003601452540000041
step 23, converting the trimap image into I1And I2，I1The pixel at the middle foreground and background mark is 1, the pixel value at the rest part is 0, and I is calculated1Converting into a sparse diagonal matrix which is Ds，I2The pixel value of the middle foreground mark is 1, the other pixel values are 0, and the pixel values are converted into column vectors which are bs：
Figure FDA0003601452540000042
Figure FDA0003601452540000043
Step 24, solving an alpha map of the central subaperture image by the formula (7):
α＝λ(L+λ)-1bs (7)
in the formula, λ is a weight coefficient of the mark, and L is a laplacian matrix of the matte.
9. The null-angle-based coherent light field matting device according to claim 5, further comprising:
and the evaluation unit is used for evaluating the hollow angle consistency of the light field matting.
10. The apparatus for coherent light field matting based on nulls as claimed in claim 9, wherein the evaluation unit specifically comprises a spatial domain evaluation subunit or a frequency domain evaluation subunit:
the spatial domain evaluation subunit is used for evaluating the light field data null angle consistency in the light field alpha diagram through a null angle consistency index S-cons obtained in the spatial domain and provided by the formula (8): the value range of the index S-cons is (0-1), the closer the index S-cons is to 1, the smaller the variance of the representative boundary area is, the better the null angle consistency of the light field data is, and the closer the index S-cons is to 0, the worse the consistency of the light field data is;
Figure FDA0003601452540000044
in the formula, NEPIThe number of EPIs (Epipolar Plane images, EPIs) which are the light field alpha maps, EjFor the corresponding jth EPI, there are l boundary regions, denoted as Ω, in each EPIl，NlIs omega in the jth EPIlTotal number of pixels of the area, EjiFor the ith pixel in the jth EPI of the light field alpha map,
Figure FDA0003601452540000051
is omega in the jth EPIlMean of pixel values at the boundary;
the frequency domain evaluation subunit is used for providing a hollow angle consistency evaluation index F-cons acquired in a frequency domain through an equation (9) and evaluating the hollow angle consistency of the light field data in the light field alpha diagram: the value range of the index F-cons is (0-1), the larger the index F-cons is, the smaller the occupied range of the representative support is, the better the null angle consistency of the light field data is, and otherwise, the worse is;
Figure FDA0003601452540000052
in the formula:
Figure FDA0003601452540000053
wherein, L' (Ω)u，Ωv，Ωx，Ωy) Is a spectrogram L (omega)u，Ωv，Ωx，Ωy) After binarization, the spectrogram is counted, wherein epsilon is a smaller number, M is the number of integral pixels of the spectrogram, and omegau、Ωv、Ωx、ΩyRespectively, the coordinate axes of the light field spectrogram.
CN202210405016.5A 2022-04-18 2022-04-18 Light field matting method and device based on hollow angle consistency Pending CN114742847A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210405016.5A CN114742847A (en) 2022-04-18 2022-04-18 Light field matting method and device based on hollow angle consistency

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210405016.5A CN114742847A (en) 2022-04-18 2022-04-18 Light field matting method and device based on hollow angle consistency

Publications (1)

Publication Number Publication Date
CN114742847A true CN114742847A (en) 2022-07-12

Family

ID=82281508

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210405016.5A Pending CN114742847A (en) 2022-04-18 2022-04-18 Light field matting method and device based on hollow angle consistency

Country Status (1)

Country Link
CN (1) CN114742847A (en)

Similar Documents

Publication Publication Date Title
CN108074218B (en) Image super-resolution method and device based on light field acquisition device
CN108596965B (en) Light field image depth estimation method
WO2016184099A1 (en) Depth estimation method based on light field data distribution
JP2019532451A (en) Apparatus and method for obtaining distance information from viewpoint
CN113034568B (en) Machine vision depth estimation method, device and system
CN109345502B (en) Stereo image quality evaluation method based on disparity map stereo structure information extraction
CN110120071A (en) A kind of depth estimation method towards light field image
CN111028281B (en) Depth information calculation method and device based on light field binocular system
CN106355621A (en) Method for acquiring depth information on basis of array images
US20160148386A1 (en) Method and apparatus for estimating depth of focused plenoptic data
CN115546442A (en) Multi-view stereo matching reconstruction method and system based on perception consistency loss
CN109218706B (en) Method for generating stereoscopic vision image from single image
CN112927348B (en) High-resolution human body three-dimensional reconstruction method based on multi-viewpoint RGBD camera
Liu et al. Adaptive matching norm based disparity estimation from light field data
CN109993764A (en) A kind of light field depth estimation method based on frequency domain energy distribution
CN116402908A (en) Dense light field image reconstruction method based on heterogeneous imaging
CN114742847A (en) Light field matting method and device based on hollow angle consistency
CN114998532B (en) Three-dimensional image visual transmission optimization method based on digital image reconstruction
CN111260712A (en) Depth estimation method and device based on refocusing focal polar line diagram neighborhood distribution
Gava et al. Dense scene reconstruction from spherical light fields
CN110827338B (en) Regional self-adaptive matching light field data depth reconstruction method
CN111197976A (en) Three-dimensional reconstruction method considering multi-stage matching propagation of weak texture region
CN114742846A (en) Method and device for evaluating optical field cutout based on hollow angle consistency
CN114742845A (en) Light field matting method and device based on hollow angle consistency
CN109741389A (en) One kind being based on the matched sectional perspective matching process of region base

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination