CN104851089A - Static scene foreground segmentation method and device based on three-dimensional light field - Google Patents
Static scene foreground segmentation method and device based on three-dimensional light field Download PDFInfo
- Publication number
- CN104851089A CN104851089A CN201510209026.1A CN201510209026A CN104851089A CN 104851089 A CN104851089 A CN 104851089A CN 201510209026 A CN201510209026 A CN 201510209026A CN 104851089 A CN104851089 A CN 104851089A
- Authority
- CN
- China
- Prior art keywords
- straight line
- scene
- depth
- image
- polar plane
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Image Analysis (AREA)
Abstract
The invention discloses a static scene foreground segmentation method and a static scene foreground segmentation device based on a three-dimensional light field. The static scene foreground segmentation method comprises the steps of: shooting sequential images of a scene on a one-dimensional straight line at equal-intervals through a camera to establish the three-dimensional light field, and generating an antipodal planar graph of the scene; extracting straight line features in the antipodal planar graph by utilizing a straight line detection algorithm and calculating slope information, recovering depth information of different objects in the scene by using the slope information, and generating a depth image of the entire scene by using a fast interpolation algorithm; and setting corresponding depth threshold values for different objects in the depth image, and segmenting the different objects according to the depth threshold values quickly. The static scene foreground segmentation method and the static scene foreground segmentation device can recover the spatial relationship among the plurality of objects in the scene accurately, overcome the over-segmentation problem existing in the complex scene application based on methods such as region clustering and mathematical morphology, and has high segmentation efficiency when aiming at specific target extraction.
Description
Technical field
The present invention relates to technical field of image processing, refer to a kind of static scene foreground segmentation method based on 3 d light fields and device especially.
Background technology
Iamge Segmentation refers to that the zones of different having special connotation in image makes a distinction.Each region that segmentation produces is communicated with, and in region, pixel meets specific conformance criteria, and different image-regions mutually disjoints, each other the property of there are differences.On segmentation concept basis, can be " prospect " and " background " two classifications by image abstraction.Wherein, the concept of image " prospect ", for " background ", is often referred to the interested region of people in picture scene.Effective foreground segmentation is prerequisite and the basis of the contour level application of virtual scene structure, intelligent video monitoring and natural man-machine interaction.
Foreground segmentation method in existing static scene is mainly divided into based on threshold value, based on edge, based on region and the several classification such as method in conjunction with other particular theory.Wherein, the watershed algorithm that L.Vincent etc. propose regards topological landforms as image, builds gathering ground, can obtain continuous print partitioning boundary, but often there will be over-segmentation problem in actual applications centered by the local minimum points of gray scale; Felzenszwalb etc. propose the figure dividing method adopting greedy search strategy, and the method can obtain globally optimal solution by paired regional compare method; Chan etc. devise geometry active contour model based on level set, achieve single goal contours extract preferably, but still accurate not at edge local.
Summary of the invention
In view of this, the object of the invention is to propose a kind of can effectively, the static scene foreground segmentation method based on 3 d light fields of effective implemention Iamge Segmentation and device.
Based on above-mentioned purpose a kind of static scene foreground segmentation method based on 3 d light fields provided by the invention, comprise the following steps:
By camera at the sequence image of one dimension straight line first-class interval shooting one scene to build 3 d light fields, and generating scene to polar plane figure;
Use line detection algorithm to extract described to the linear feature in polar plane figure and slope calculations information, by the depth information of different objects in described slope information restoration scenario, and use Fast Interpolation to generate the depth image of whole scene;
The depth threshold corresponding to the different objects setting in described depth image, and according to described depth threshold, Fast Segmentation is carried out to different objects.
Preferably, in described 3 d light fields, any light L is expressed as:
L＝LF(x,y,t)
Wherein, t is the starting point of light, i.e. the coordinate of described camera on described one dimension straight line; (x, y) represents the direction of light, corresponding to the two-dimensional coordinate value in image;
Described is described sequence image pixels across stacking under identical y value condition to polar plane figure, namely perpendicular to (x, t) tangent plane of y coordinate; In scene same object pixel described to polar plane figure in form straight line track, and the space length between object and camera straight-line trajectory be proportional to this object described to polar plane figure in the slope of line correspondence.
Preferably, the step generating described depth image comprises further:
The width choosing described sequence image is as the target image of depth recovery and foreground segmentation;
Use line detection algorithm from described to extracting straight line polar plane figure and determining all linearity regions;
According to described linearity region, generate the slope distribution of linear feature point at described target image;
According to the slope distribution of described linear feature point, interpolation algorithm is adopted to generate the slope distribution of all pixels of described target image;
The slope distribution transformed depth of all for described target image pixels is distributed, then between linear mapping to gray area on, final generate described depth image.
Preferably, before using line detection algorithm to extract straight line, carry out Gauss's convergent-divergent to described to polar plane figure, pantograph ratio is 0.9.
Preferably, determine that the step of described linearity region comprises:
To described to each pixel in polar plane figure, calculate the angle of the consistent point of proximity direction of its relative color and horizontal direction, the pixel that this angle is close forms straight line candidates district;
Cover straight line candidates district described in each with approximate rectangle, structure noise model performs checking to described straight line candidates district, draws the straight probability of described straight line candidates district structure;
The probability threshold value that setting straight line judges, finally determines described linearity region.
Preferably, after the described step to extracting straight line polar plane figure, the Screening Treatment step to extracting result is also comprised:
Only extracting end points drops on described to the straight line in polar plane figure in the y-axis direction front ten pixels;
To not have and the described extend directly crossing to polar plane figure coboundary, calculate and infer intersection point;
Reject and infer that intersection point exceeds the straight line of image boundary, occur due to prolongation two coincidence straight lines are merged into single straight line.
Present invention also offers a kind of static scene foreground segmentation device based on 3 d light fields, comprising:
Build module, for by camera at the sequence image of one dimension straight line first-class interval shooting one scene to build 3 d light fields, and generating scene to polar plane figure;
Depth recovery module, described to the linear feature in polar plane figure and slope calculations information for using line detection algorithm to extract, by the depth information of different objects in described slope information restoration scenario, and Fast Interpolation is used to generate the depth image of whole scene;
Segmentation module, for the depth threshold corresponding to the different objects setting in described depth image, and carries out Fast Segmentation according to described depth threshold to different objects.
Preferably, in the described 3 d light fields of described structure CMOS macro cell, any light L is expressed as:
L＝LF(x,y,t)
Wherein, t is the starting point of light, i.e. the coordinate of described camera on described one dimension straight line; (x, y) represents the direction of light, corresponding to the two-dimensional coordinate value in image;
Described is described sequence image pixels across stacking under identical y value condition to polar plane figure, namely perpendicular to (x, t) tangent plane of y coordinate; In scene same object pixel described to polar plane figure in form straight line track, and the space length between object and camera straight-line trajectory be proportional to this object described to polar plane figure in the slope of line correspondence.
Preferably, described depth recovery module is further used for: the width choosing described sequence image is as the target image of depth recovery and foreground segmentation; Use line detection algorithm from described to extracting straight line polar plane figure and determining all linearity regions; According to described linearity region, generate the slope distribution of linear feature point at described target image; According to the slope distribution of described linear feature point, interpolation algorithm is adopted to generate the slope distribution of all pixels of described target image; The slope distribution transformed depth of all for described target image pixels is distributed, then between linear mapping to gray area on, final generate described depth image.
Preferably, described depth recovery module also comprises for before use line detection algorithm extracts straight line, and to described Zoom module polar plane figure being carried out to Gauss's convergent-divergent, the pantograph ratio that described Zoom module carries out convergent-divergent is 0.9.
As can be seen from above, static scene foreground segmentation method based on 3 d light fields provided by the invention and device, building 3 d light fields by the sequence image in straight line equally spaced different points of view photographs scene, extracting scene edge and depth information thereof by line detection algorithm to analyzing polar plane figure; Recover the depth information of whole scene by Fast Interpolation, realize the segmentation to the foreground object of different depth eventually through threshold method.The present invention can spatial relationship more exactly in restoration scenario between multiple object, foreground segmentation result overcomes the existing over-segmentation problem existed in complex scene application based on the method such as region clustering and mathematical morphology preferably, has higher segmentation efficiency when extracting for specific objective.
Accompanying drawing explanation
In order to be illustrated more clearly in the embodiment of the present invention or technical scheme of the prior art, be briefly described to the accompanying drawing used required in embodiment or description of the prior art below, apparently, accompanying drawing in the following describes is only some embodiments of the present invention, for those of ordinary skill in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to these accompanying drawings.
Fig. 1 is the static scene foreground segmentation method process flow diagram based on 3 d light fields of the preferred embodiment of the present invention;
Fig. 2 is that the 3 d light fields of the preferred embodiment of the present invention describes schematic diagram;
Fig. 3 is the geometric relationship schematic diagram between object in the preferred embodiment of the present invention, imaging plane, imaging center path;
Fig. 4 (a) determines the original image pixels in the step of linearity region for the preferred embodiment of the present invention;
Fig. 4 (b) determines the pixel of the calculated angle in the step of linearity region for the preferred embodiment of the present invention;
Fig. 4 (c) determines the straight line candidates district in the step of linearity region for the preferred embodiment of the present invention;
Fig. 5 (a) is straight-line detection result when Gauss's pantograph ratio is 0.5 in the preferred embodiment of the present invention;
Fig. 5 (b) is straight-line detection result when Gauss's pantograph ratio is 0.9 in the preferred embodiment of the present invention;
Fig. 5 (c) is straight-line detection result when Gauss's pantograph ratio is 1.5 in the preferred embodiment of the present invention;
Fig. 6 (a) is the original scene image in the preferred embodiment of the present invention;
Fig. 6 (b) is the EPI in the preferred embodiment of the present invention;
Fig. 6 (c) is the depth image in the preferred embodiment of the present invention;
Fig. 7 is " Mansion " scene depth image histogram statistics;
The segmentation result that Fig. 8 (a) is split for applying method of the present invention to " Church " scene;
The segmentation result that Fig. 8 (b) is split for applying method of the present invention to " Mansion " scene;
The segmentation result that Fig. 8 (c) is split for applying method of the present invention to " Statue " scene;
Fig. 9 (a) is the segmentation result to " Church ", " Mansion ", " Statue " three scenes use fractional spins;
Fig. 9 (b) is the segmentation result to " Church ", " Mansion ", " Statue " three scenes use Graph Cut partitioning algorithms;
Fig. 9 (c) is based on the partitioning algorithm of K-means cluster) for make segmentation result to " Church ", " Mansion ", " Statue " three scenes;
Figure 10 is the static scene foreground segmentation apparatus structure schematic diagram based on 3 d light fields of the embodiment of the present invention.
Embodiment
For making the object, technical solutions and advantages of the present invention clearly understand, below in conjunction with specific embodiment, and with reference to accompanying drawing, the present invention is described in more detail.
Embodiments provide a kind of static scene foreground segmentation method based on 3 d light fields, the method builds 3 d light fields by the sequence image in straight line equally spaced different points of view photographs scene, extracts scene edge and depth information thereof by line detection algorithm to analyzing polar plane figure; Recover the depth information of whole scene by Fast Interpolation, realize the segmentation to the foreground object of different depth eventually through threshold method.Wherein, light field (Light Fields) is the description to light all in space.Scene in real world comprises abundant three-dimensional information, and single two-dimensional image can not describe the spatial relationship between object completely.Use dense sequence image can be used for portraying static scene.In the sequence image of quick sampling, time continuity is roughly equal to the space continuity of scene, adopts this describing method of " to polar plane figure " (Epipolar Plane Image, EPI) to carry out analyzing three-dimensional scene simultaneously.Below, the described static scene foreground segmentation method based on 3 d light fields is further illustrated by the preferred embodiments of the present invention.
With reference to figure 1, it is the static scene foreground segmentation method process flow diagram based on 3 d light fields of the preferred embodiment of the present invention.
The static scene foreground segmentation method based on 3 d light fields of the preferred embodiment of the present invention comprises the following steps:
Step 1: by camera at the sequence image of one dimension straight line first-class interval shooting one scene to build 3 d light fields, and generating scene to polar plane figure;
Step 2: use line detection algorithm to extract described to the linear feature in polar plane figure and slope calculations information, by the depth information of different objects in described slope information restoration scenario, and use Fast Interpolation to generate the depth image of whole scene;
Step 3: the depth threshold corresponding to the different objects setting in described depth image, and according to described depth threshold, Fast Segmentation is carried out to different objects.
(1) the concrete enforcement of step 1
In step 1, first set up 3 d light fields.Typical light field building process needs under different visual angles, to take great amount of images to same scene usually, then adopts suitable geometric model to describe light distribution situation in space.For the application background of the scene foreground segmentation in the present embodiment, imaging center situation on a horizontal linear of study tour image sequence, namely sets up 3 d light fields by obtaining orderly two dimensional image on an one dimension straight line.
With reference to figure 2, for the 3 d light fields of the preferred embodiment of the present invention describes schematic diagram.
The generation of image can be regarded as the projection process of light on imaging plane.Any light L in space is described below:
L＝LF(x,y,t)
Wherein, t is the starting point of light, i.e. the coordinate of described camera on described one dimension straight line; (x, y) represents the direction of light, corresponding to the two-dimensional coordinate value in image.In the 3 d light fields described by LF, perpendicular to the scene image under the corresponding different visual angles of (x, y) tangent plane of t coordinate, and perpendicular to (x, t) tangent plane of y coordinate corresponding be exactly to polar plane figure, i.e. EPI.
Intuitively, EPI corresponds to sequence image pixels across stacking under identical y value condition.In order to the convenience of subsequent treatment, the present embodiment supposition arbitrary neighborhood two width sequence image photocentre (photocentre is also camera position) spacing is consistent, photocentre is the center of imaging, camera position equidistantly can ensure that photocentre is equidistant, namely ensures the continuity of same object corresponding pixel points in the EPI formed.When sampled images quantity is abundant, in scene, same object pixel point forms straight line track in EPI.By can obtain the spatial information of object in scene to the geometrical Characteristics Analysis of straight line.
With reference to figure 3, it is the geometric relationship schematic diagram in the preferred embodiment of the present invention between object, imaging plane, imaging center path.
As shown in Figure 3, assuming that imaging plane is h to imaging center distance, object P is D, x to the distance in path, optical center
1, x
2be respectively the horizontal ordinate of P pixel in the picture, Δ t is imaging center displacement.
Can be obtained by triangle similarity relation:
Cancellation t, can obtain:
Wherein, Δ x=|x
1-x
2|.
In EPI,
be proportional to straight slope k, then the space length D between known object and camera and EPI cathetus slope k are also proportional relationships.On the basis obtaining the corresponding slope of each pixel of scene, estimation can be made to depth information wherein further.
(2) the concrete enforcement of step 2
As preferred embodiment, the step generating described depth image specifically comprises:
The width choosing described sequence image is as the target image of depth recovery and foreground segmentation;
Use line detection algorithm from described to extracting straight line polar plane figure and determining all linearity regions;
According to described linearity region, generate the slope distribution of linear feature point at described target image;
According to the slope distribution of described linear feature point, interpolation algorithm is adopted to generate the slope distribution of all pixels of described target image;
The slope distribution transformed depth of all for described target image pixels is distributed, then between linear mapping to gray area on, final generate described depth image.
Concrete, first carry out straight-line detection.The number detected at EPI cathetus and quality estimate the key of the object scene degree of depth.Usually there is the Rapid Variable Design of color in object scene border, in EPI, then shows as obvious linear feature.Straight-line detection in the present embodiment adopts LSD (Line Segment Detection) method, and carries out screening and processing for the specific features of EPI image.
For each pixel, the present embodiment calculates the angle α of the consistent point of proximity direction of its relative color and horizontal direction.The pixel that angle α is close constitutes straight line candidates district (Line Support Regions), that is possible linearity region.With reference to figure 4 (a), Fig. 4 (b), Fig. 4 (c), it is respectively the original image pixels in this preferred embodiment, the pixel of calculated angle, straight line candidates district.
For each straight line candidates district, cover this area pixel with approximate rectangle.Choose the mode α ' of pixel level angle α in rectangle for rectangle principal direction, straight line proof procedure is performed to each rectangle.Prescribed skew tolerance τ (angle, value 0 ~ π), with rectangle principal direction angle β=| α-α ' | the pixel being less than τ is similar to a little as straight line.Then for any one pixel of rectangular area, it belongs to the probability of straight line
all pixel angle β is independent and meet binomial distribution:
β～B(n,s,p)
Wherein, n is total pixel number in rectangular area, and to be that straight line is approximate count s.By structure noise model (Noise Model), proof procedure is performed to rectangular area, draw the straight probability Q of rectangular area structure.The probability threshold value that setting straight line judges, finally determines linearity region.
Further, because the scaling in image processing process can the effect of appreciable impact EPI straight-line detection, the present embodiment is tested and is contrasted under different Gauss's scalings.With reference to figure 5 (a), Fig. 5 (b), Fig. 5 (c), the straight-line detection result that it is Gauss's pantograph ratio respectively when being 0.5,0.9,1.5.Find out from above-mentioned a few figure, when Gauss's pantograph ratio is lower, straight-line detection result is comparatively sparse, lost partial straight lines feature; More interference straight line is then there is when pantograph ratio is higher.In actual tests, comprehensive depth information recovery effects is considered, the pantograph ratio that the present embodiment is chosen is 0.9.
It should be noted that, in other embodiments of the invention, when straight-line detection is carried out to EPI, can also use: based on the lines detection of Hough transform, the lines detection based on Canny rim detection, the lines detection based on chain code or other the method can carrying out straight-line detection accurately.
Based on the linearity region that the above-mentioned straight-line detection for EPI obtains, next carry out recovery depth image.The present embodiment the first width of choosing sequence image is as the target image of depth recovery and foreground segmentation.EPI has gathered the scene information of all image sequences, because visual angle changes, may constantly occur new object in follow-up scene; Because image acquisition time is inconsistent, take the object (as pedestrian etc.) flashed in follow-up scene frame and can form noise in EPI.In addition, still there is limitation in LSD straight-line detection, as the situation that straight line interrupts and cannot extract.For the problems referred to above, the present embodiment makes screening further and process to lines detection result, performs following steps:
1. only extracting end points drops on described to the straight line in polar plane figure in the y-axis direction front ten pixels;
2. will not have and the described extend directly crossing to polar plane figure coboundary, calculate and infer intersection point;
3. reject and infer that intersection point exceeds the straight line of image boundary, occur due to prolongation two coincidence straight lines are merged into single straight line.
For given y value, corresponding EPI
yorderly binary array (u, k) can be produced through above process.Wherein, u represents EPI straight line and coboundary intersecting point coordinate, i.e. target image corresponding abscissa value under ordinate y, and k represents the corresponding slope on this horizontal ordinate.Travel through all y values, the slope value of target image at coordinate (u, y, k) place can be obtained, namely generate the slope distribution of linear feature point at target image; I.e. (u, y), it is the coordinate position of pixel in target image (original image) showing as linear feature in EPI to described linear feature point; In the target image, linear feature point is shown as some pixels of discrete distribution.
In order to obtain dense depth information, the present embodiment is mapped in x-axis direction by interpolation and obtains complete slope and represent.Adopt the rationality of interpolation method because of what time following:
1) the pixel color value that in original image, change in depth is larger is changed significantly, and can form obvious linear feature in EPI;
2) straight line in EPI more intactly can be detected by LSD algorithm, can estimate depth information comparatively accurately to detecting line correspondences pixel;
3) two the depth value changes of EPI straight line between original image corresponding point are continuous and level and smooth.
Therefore, the selection of interpolation method just should meet:
1) in order to ensure the accuracy of the depth value detected, interpolating function f has to pass through given reference mark, namely meets f (u)=k;
2) interpolating function f can obtain the estimating depth value at all pixel places, and namely f is at [0, X
max] field of definition on continuously;
3) interpolating function f seamlessly transits between two reference mark, namely for given u
1, u
2,
x
2∈ [u
1, u
2], following formula is set up:
f(x
1)′×f(x
2)′＞0
Comprehensive above consideration, the present embodiment adopts Pchip (Piecewise Cubic Hermite Interpolating Polynomial) as interpolation method, in order to obtain the slop estimation value k at any pixel (x, y) place in former figure.
Due to Pchip interpolation method itself, at close [0, X
max] the pixel slop estimation value at two ends often there will be excessive or too small situation, now sets [0, k further
max] interval to standardize outlier as boundary.According to the space length between object and camera be proportional to this object described to polar plane figure in the corresponding relation of slope of line correspondence, linear mapping can be adopted slope distribution (x, y, k) depth profile (x is transformed to, y, d), then between sublinear function to gray area on, finally obtain the output directly perceived of depth image.Extract the visual process of depth image with reference to shown in figure 6 (a), 6 (b), 6 (c).Red line in above-mentioned each figure identifies the correspondence position of EPI in former figure.
In other embodiments of the invention, when carrying out above-mentioned depth recovery, can also use: linear interpolation method, lagrange-interpolation, Spline Interpolation Method, Newton interpolation method or other interpolation methods.
(3) the concrete enforcement of step 3
In reality scene, the differentiation of object usually shows as the uncontinuity on locus, and the depth information of scene obtained herein is the support that foreground segmentation process provides third dimension data.In depth map image basis, for the object of different spaces level, set the rapid extraction that corresponding depth threshold can realize for subject.
The image histogram of Fig. 7 for obtaining from " Mansion " scene depth image statistics Fig. 8 (b).In grey level histogram, transverse axis represents number of greyscale levels (be generally 0 ~ 255 totally 256 grades), and the longitudinal axis represents that gray-scale value is in the pixel number of current gray.Pixels all in gray level image are added up, grey level histogram distribution as shown in Figure 7 can be formed.Observe known histogram and shown comparatively significantly depth profile feature, on slope axle, selected appropriate threshold point can distinguish the different objects in scene.
Based on the method for the invention described above preferred embodiment, next by the assessment of segmentation result and compare the beneficial effect of the present invention further illustrated with the segmentation result of existing common segmentation methods.
The situation that is suitable for of embodiment of the present invention research is segmentation based on object in static scene, and data demand is used in a large amount of photo sequence of the different angles shooting of same level straight line.Experiment adopts the frequently-used data collection constructed in Iamge Segmentation process prior art to carry out assessing and contrasting.The present invention mainly have chosen " Church ", " Mansion ", " Statue " three scenes are as experimental subjects, wherein each contextual data collection contain by computer-controlled sliding platform to the 101 width images that same static scene is taken at different imaging point, image all overcorrection and alignment pre-service.
Foreground segmentation test is carried out respectively three data centralizations.Corresponding gray threshold is set for different target objects, draw segmentation result, shown in figure 8 (a), 8 (b), 8 (c), be respectively and the segmentation result that method of the present invention is split is applied to " Church ", " Mansion ", " Statue " three scenes.Three pictures in Fig. 8 (a) are followed successively by the segmentation result of electric pole and flowering shrubs in " Church " scene, tower, tree; Three pictures in 8 (b) are followed successively by the segmentation result in tree, fence, house in " Mansion " scene; Three pictures in 8 (c) are followed successively by the segmentation result of statue in " Statue " scene, thick grass, statue and automobile.See that method scenic focal point object detail of the present invention has good segmentation effect intuitively.And in Church data centralization, the object close to this kind of color characteristic in the trees in scene, tower and house, also achieves and splits preferably.
Adopt the recall ratio and precision ratio two quantizating index analysis segmentation results that define in prior art.Wherein recall ratio represents the ratio of the correct pixel count of segmentation and Standard Segmentation pixel count, and precision ratio represents the ratio splitting correct pixel count and segmentation total pixel number in segmentation image.Wherein, for the Standard Segmentation image of comparison by manually demarcating acquisition.Calculate as shown in table 1 to the statue segmentation situation in the tree in the tower in " Church ", " Mansion " and " Statue " respectively.
Table 1 uses the quantitative evaluation of the inventive method segmentation result
In order to verify the validity of the inventive method foreground segmentation in complex static scene further, choosing fractional spins, Graph Cut partitioning algorithm and doing intuitively to contrast based on the partitioning algorithm of K-means cluster and method of the present invention.
With reference to figure 9 (a), 9 (b), 9 (c), be respectively the segmentation result to " Church ", " Mansion ", " Statue " three scenes uses fractional spins, Graph Cut partitioning algorithm and partitioning algorithms based on K-means cluster.Because color in complex scene (gray scale) feature is complicated and changeable, several colors (trees as in Mansion scene) that same object may have contrast larger, different objects also may have close color (trees as in Church scene), above-mentioned several common method can produce obvious over-segmentation problem, and the follow-up processing procedure of needs could form the segmentation result for specific objective object usually.By contrast, method of the present invention is more simple and effective.
The embodiment of the present invention additionally provides a kind of static scene foreground segmentation device based on 3 d light fields, with reference to Figure 10, is the static scene foreground segmentation apparatus structure schematic diagram based on 3 d light fields of the embodiment of the present invention.Described device comprises:
Build module 101, for by camera at the sequence image of one dimension straight line first-class interval shooting one scene to build 3 d light fields, and generating scene to polar plane figure;
Depth recovery module 102, described to the linear feature in polar plane figure and slope calculations information for using line detection algorithm to extract, by the depth information of different objects in described slope information restoration scenario, and Fast Interpolation is used to generate the depth image of whole scene;
Segmentation module 103, for the depth threshold corresponding to the different objects setting in described depth image, and carries out Fast Segmentation according to described depth threshold to different objects.
Concrete, in the described 3 d light fields that described structure module 101 generates, any light L is expressed as:
L＝LF(x,y,t)
Wherein, t is the starting point of light, i.e. the coordinate of described camera on described one dimension straight line; (x, y) represents the direction of light, corresponding to the two-dimensional coordinate value in image;
Described is described sequence image pixels across stacking under identical y value condition to polar plane figure, namely perpendicular to (x, t) tangent plane of y coordinate; In scene same object pixel described to polar plane figure in form straight line track, and the space length between object and camera straight-line trajectory be proportional to this object described to polar plane figure in the slope of line correspondence.
As preferred embodiment, described depth recovery module 102 is further used for: the width choosing described sequence image is as the target image of depth recovery and foreground segmentation; Use line detection algorithm from described to extracting straight line polar plane figure and determining all linearity regions; According to described linearity region, generate the slope distribution of linear feature point at described target image; According to the slope distribution of described linear feature point, interpolation algorithm is adopted to generate the slope distribution of all pixels of described target image; The slope distribution transformed depth of all for described target image pixels is distributed, then between linear mapping to gray area on, final generate described depth image.
In a preferred embodiment, described depth recovery module 102 also comprises for before use line detection algorithm extracts straight line, and to described Zoom module polar plane figure being carried out to Gauss's convergent-divergent, the pantograph ratio that described Zoom module carries out convergent-divergent is 0.9.
In a preferred embodiment, when described depth recovery module 102 determines described linearity region, first to described to each pixel in polar plane figure, calculate the angle of the consistent point of proximity direction of its relative color and horizontal direction, the close pixel of this angle forms straight line candidates district; Cover straight line candidates district described in each with approximate rectangle again, structure noise model performs checking to described straight line candidates district, draws the straight probability of described straight line candidates district structure; The probability threshold value that setting straight line judges, finally determines described linearity region.
In a preferred embodiment, described depth recovery module 102 also for carrying out Screening Treatment to the result extracting straight line in polar plane figure, specifically comprises:
Only extracting end points drops on described to the straight line in polar plane figure in the y-axis direction front ten pixels;
To not have and the described extend directly crossing to polar plane figure coboundary, calculate and infer intersection point;
Reject and infer that intersection point exceeds the straight line of image boundary, occur due to prolongation two coincidence straight lines are merged into single straight line.
In sum, the present invention builds 3 d light fields by camera at the first-class interval shooting sequence image of one dimension straight path, in EPI analyzes, estimate object scene depth information, recovered entire depth image by Fast Interpolation method, and achieved a kind of dividing method of foreground object on this basis.In the segmentation of complex open country scene, comparing the over-segmentation problem effectively overcoming classic method, having higher segmentation efficiency when extracting for specific objective.
Those of ordinary skill in the field are to be understood that: the foregoing is only specific embodiments of the invention; be not limited to the present invention; within the spirit and principles in the present invention all, any amendment made, equivalent replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (10)
1., based on a static scene foreground segmentation method for 3 d light fields, it is characterized in that, comprise the following steps:
By camera at the sequence image of one dimension straight line first-class interval shooting one scene to build 3 d light fields, and generating scene to polar plane figure;
Use line detection algorithm to extract described to the linear feature in polar plane figure and slope calculations information, by the depth information of different objects in described slope information restoration scenario, and use Fast Interpolation to generate the depth image of whole scene;
The depth threshold corresponding to the different objects setting in described depth image, and according to described depth threshold, Fast Segmentation is carried out to different objects.
2. method according to claim 1, is characterized in that, in described 3 d light fields, any light L is expressed as:
L＝LF(x,y,t)
Wherein, t is the starting point of light, i.e. the coordinate of described camera on described one dimension straight line; (x, y) represents the direction of light, corresponding to the two-dimensional coordinate value in image;
Described is described sequence image pixels across stacking under identical y value condition to polar plane figure, namely perpendicular to (x, t) tangent plane of y coordinate; In scene same object pixel described to polar plane figure in form straight line track, and the space length between object and camera straight-line trajectory be proportional to this object described to polar plane figure in the slope of line correspondence.
3. method according to claim 2, is characterized in that, the step generating described depth image comprises further:
The width choosing described sequence image is as the target image of depth recovery and foreground segmentation;
Use line detection algorithm from described to extracting straight line polar plane figure and determining all linearity regions;
According to described linearity region, generate the slope distribution of linear feature point at described target image;
According to the slope distribution of described linear feature point, interpolation algorithm is adopted to generate the slope distribution of all pixels of described target image;
The slope distribution transformed depth of all for described target image pixels is distributed, then between linear mapping to gray area on, final generate described depth image.
4. method according to claim 3, is characterized in that, before using line detection algorithm to extract straight line, carry out Gauss's convergent-divergent to described to polar plane figure, pantograph ratio is 0.9.
5. method according to claim 3, is characterized in that, determines that the step of described linearity region comprises:
To described to each pixel in polar plane figure, calculate the angle of the consistent point of proximity direction of its relative color and horizontal direction, the pixel that this angle is close forms straight line candidates district;
Cover straight line candidates district described in each with approximate rectangle, structure noise model performs checking to described straight line candidates district, draws the straight probability of described straight line candidates district structure;
The probability threshold value that setting straight line judges, finally determines described linearity region.
6. method according to claim 3, is characterized in that, after the described step to extracting straight line polar plane figure, also comprises the Screening Treatment step to extracting result:
Only extracting end points drops on described to the straight line in polar plane figure in the y-axis direction front ten pixels;
To not have and the described extend directly crossing to polar plane figure coboundary, calculate and infer intersection point;
Reject and infer that intersection point exceeds the straight line of image boundary, occur due to prolongation two coincidence straight lines are merged into single straight line.
7., based on a static scene foreground segmentation device for 3 d light fields, it is characterized in that, comprising:
Build module, for by camera at the sequence image of one dimension straight line first-class interval shooting one scene to build 3 d light fields, and generating scene to polar plane figure;
Depth recovery module, described to the linear feature in polar plane figure and slope calculations information for using line detection algorithm to extract, by the depth information of different objects in described slope information restoration scenario, and Fast Interpolation is used to generate the depth image of whole scene;
Segmentation module, for the depth threshold corresponding to the different objects setting in described depth image, and carries out Fast Segmentation according to described depth threshold to different objects.
8. device according to claim 7, is characterized in that, in the described 3 d light fields of described structure CMOS macro cell, any light L is expressed as:
L＝LF(x,y,t)
Wherein, t is the starting point of light, i.e. the coordinate of described camera on described one dimension straight line; (x, y) represents the direction of light, corresponding to the two-dimensional coordinate value in image;
Described is described sequence image pixels across stacking under identical y value condition to polar plane figure, namely perpendicular to (x, t) tangent plane of y coordinate; In scene same object pixel described to polar plane figure in form straight line track, and the space length between object and camera straight-line trajectory be proportional to this object described to polar plane figure in the slope of line correspondence.
9. device according to claim 7, is characterized in that, described depth recovery module is further used for: the width choosing described sequence image is as the target image of depth recovery and foreground segmentation; Use line detection algorithm from described to extracting straight line polar plane figure and determining all linearity regions; According to described linearity region, generate the slope distribution of linear feature point at described target image; According to the slope distribution of described linear feature point, interpolation algorithm is adopted to generate the slope distribution of all pixels of described target image; The slope distribution transformed depth of all for described target image pixels is distributed, then between linear mapping to gray area on, final generate described depth image.
10. device according to claim 9, it is characterized in that, described depth recovery module also comprises for before use line detection algorithm extracts straight line, and to described Zoom module polar plane figure being carried out to Gauss's convergent-divergent, the pantograph ratio that described Zoom module carries out convergent-divergent is 0.9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510209026.1A CN104851089A (en) | 2015-04-28 | 2015-04-28 | Static scene foreground segmentation method and device based on three-dimensional light field |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510209026.1A CN104851089A (en) | 2015-04-28 | 2015-04-28 | Static scene foreground segmentation method and device based on three-dimensional light field |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104851089A true CN104851089A (en) | 2015-08-19 |
Family
ID=53850716
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510209026.1A Pending CN104851089A (en) | 2015-04-28 | 2015-04-28 | Static scene foreground segmentation method and device based on three-dimensional light field |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104851089A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105184808A (en) * | 2015-10-13 | 2015-12-23 | 中国科学院计算技术研究所 | Automatic segmentation method for foreground and background of optical field image |
| CN106257537A (en) * | 2016-07-18 | 2016-12-28 | 浙江大学 | A kind of spatial depth extracting method based on field information |
| CN107977938A (en) * | 2017-11-24 | 2018-05-01 | 北京航空航天大学 | A kind of Kinect depth image restorative procedure based on light field |
| CN108171674A (en) * | 2018-01-15 | 2018-06-15 | 西北工业大学 | For the vision correcting method of visual angle projector image |
| CN109033972A (en) * | 2018-06-27 | 2018-12-18 | 上海数迹智能科技有限公司 | A kind of object detection method, device, equipment and storage medium |
| CN110036410A (en) * | 2016-10-18 | 2019-07-19 | 弗托斯传感与算法公司 | For obtaining the device and method of range information from view |
| CN110232684A (en) * | 2019-06-13 | 2019-09-13 | 大连理工大学 | A kind of 3 d medical images automatic division method based on spectrum analysis |
| CN111754561A (en) * | 2020-06-10 | 2020-10-09 | 江南大学 | Light field image depth recovery method and system based on self-supervision depth learning |
| CN112233162A (en) * | 2020-10-06 | 2021-01-15 | 武汉烽火凯卓科技有限公司 | Measuring method and device for plane image analysis based on structural illumination characteristics |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110069152A1 (en) * | 2009-09-24 | 2011-03-24 | Shenzhen Tcl New Technology Ltd. | 2D to 3D video conversion |
| CN102223553A (en) * | 2011-05-27 | 2011-10-19 | 山东大学 | Method for converting two-dimensional video into three-dimensional video automatically |
| CN102308320A (en) * | 2009-02-06 | 2012-01-04 | 香港科技大学 | Generating three-dimensional models from images |
-
2015
- 2015-04-28 CN CN201510209026.1A patent/CN104851089A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102308320A (en) * | 2009-02-06 | 2012-01-04 | 香港科技大学 | Generating three-dimensional models from images |
| US20110069152A1 (en) * | 2009-09-24 | 2011-03-24 | Shenzhen Tcl New Technology Ltd. | 2D to 3D video conversion |
| CN102223553A (en) * | 2011-05-27 | 2011-10-19 | 山东大学 | Method for converting two-dimensional video into three-dimensional video automatically |
Non-Patent Citations (1)
| Title |
|---|
| 魏巍 等: "基于三维光场的静态场景分割", 《中国学术期刊（网络版）》 * |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105184808A (en) * | 2015-10-13 | 2015-12-23 | 中国科学院计算技术研究所 | Automatic segmentation method for foreground and background of optical field image |
| CN105184808B (en) * | 2015-10-13 | 2018-09-07 | 中国科学院计算技术研究所 | Scape automatic division method before and after a kind of light field image |
| CN106257537A (en) * | 2016-07-18 | 2016-12-28 | 浙江大学 | A kind of spatial depth extracting method based on field information |
| CN106257537B (en) * | 2016-07-18 | 2019-04-09 | 浙江大学 | A kind of spatial depth extracting method based on field information |
| CN110036410A (en) * | 2016-10-18 | 2019-07-19 | 弗托斯传感与算法公司 | For obtaining the device and method of range information from view |
| CN110036410B (en) * | 2016-10-18 | 2023-06-30 | 弗托斯传感与算法公司 | Apparatus and method for obtaining distance information from view |
| CN107977938A (en) * | 2017-11-24 | 2018-05-01 | 北京航空航天大学 | A kind of Kinect depth image restorative procedure based on light field |
| CN108171674A (en) * | 2018-01-15 | 2018-06-15 | 西北工业大学 | For the vision correcting method of visual angle projector image |
| CN108171674B (en) * | 2018-01-15 | 2020-07-14 | 西北工业大学 | Vision correction method for projector image with any visual angle |
| CN109033972A (en) * | 2018-06-27 | 2018-12-18 | 上海数迹智能科技有限公司 | A kind of object detection method, device, equipment and storage medium |
| CN110232684A (en) * | 2019-06-13 | 2019-09-13 | 大连理工大学 | A kind of 3 d medical images automatic division method based on spectrum analysis |
| CN110232684B (en) * | 2019-06-13 | 2023-05-23 | 大连理工大学 | Automatic three-dimensional medical image segmentation method based on spectrum analysis |
| CN111754561A (en) * | 2020-06-10 | 2020-10-09 | 江南大学 | Light field image depth recovery method and system based on self-supervision depth learning |
| CN112233162A (en) * | 2020-10-06 | 2021-01-15 | 武汉烽火凯卓科技有限公司 | Measuring method and device for plane image analysis based on structural illumination characteristics |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104851089A (en) | Static scene foreground segmentation method and device based on three-dimensional light field | |
| KR101856401B1 (en) | Method, apparatus, storage medium, and device for processing lane line data | |
| CN104574375B (en) | Image significance detection method combining color and depth information | |
| CN104850850B (en) | A kind of binocular stereo vision image characteristic extracting method of combination shape and color | |
| Su et al. | Vanishing point constrained lane detection with a stereo camera | |
| CN108830171B (en) | Intelligent logistics warehouse guide line visual detection method based on deep learning | |
| Zhang et al. | Semi-automatic road tracking by template matching and distance transformation in urban areas | |
| CN104978567B (en) | Vehicle checking method based on scene classification | |
| CN107862698A (en) | Light field foreground segmentation method and device based on K mean cluster | |
| CN104517095B (en) | A kind of number of people dividing method based on depth image | |
| Pang et al. | SGM-based seamline determination for urban orthophoto mosaicking | |
| CN104778701A (en) | Local image describing method based on RGB-D sensor | |
| CN105160649A (en) | Multi-target tracking method and system based on kernel function unsupervised clustering | |
| Hinz et al. | Car detection in aerial thermal images by local and global evidence accumulation | |
| CN106558051A (en) | A kind of improved method for detecting road from single image | |
| CN105138987A (en) | Vehicle detection method based on aggregation channel characteristic and motion estimation | |
| Yuan et al. | Road segmentation in aerial images by exploiting road vector data | |
| CN103632376A (en) | Method for suppressing partial occlusion of vehicles by aid of double-level frames | |
| Ding et al. | Efficient vanishing point detection method in complex urban road environments | |
| Li et al. | Unsupervised road extraction via a Gaussian mixture model with object-based features | |
| CN106709432B (en) | Human head detection counting method based on binocular stereo vision | |
| Miao et al. | Guided superpixel method for topographic map processing | |
| Munajat et al. | Road detection system based on RGB histogram filterization and boundary classifier | |
| WO2017161544A1 (en) | Single-camera video sequence matching based vehicle speed measurement method and system | |
| CN105844666B (en) | One kind being based on the matched vehicle speed measuring method of single camera video sequence and its system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| EXSB | Decision made by sipo to initiate substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20150819 |