KR101812664B1 - Method and apparatus for extracting multi-view object with fractional boundaries - Google Patents

Abstract

An operating method of an object extracting apparatus operated by at least one processor includes the steps of: extracting an interest volume in which an object is included in a 3D space based on a plurality of view images to photograph the object in multiple views; obtaining an initial panoramic region in which the object is included in each view image by projecting the interest volume to each view image; extracting the panoramic region of each view image converged from the initial panoramic region while updating an external energy model related to a color and a texture and an initialized geometric energy model from the initial panoramic region; and outputting a binary mask to divide a panorama and a background of each view image. Accordingly, the present invention can elaborately extract an alpha channel mask of the object.

Description

[0001] METHOD AND APPARATUS FOR EXTRACTING MULTI-VIEW OBJECT WITH FRACTIONAL BOUNDARIES [0002]

The present invention relates to multi-view object co-segmentation.

Multi-view object co-segmentation refers to the simultaneous segmentation of arbitrary foreground objects observed at two or more viewpoints. So far, the multi - view object co - ordination method obtains the initial view using the input camera attitude information. Then, the statistical characteristics of color are modeled with probability for the initial foreground and background, and the foreground and background areas are renewed based on the model. In this process, optimization is repeatedly performed using a Markov random field (MRF).

However, the method of convergence of multi - object objects studied until now acquires the initial foreground region as the camera posture information, and the obtained initial foreground region is often not sufficient in general. Therefore, the probability model based on insufficient information also has a limitation that obscures the foreground and background characteristics of the input image. In addition, the method of multi-object co-operation that has been studied so far may require additional intervention by the user for final results depending on the object extraction application. Especially, the method of multi-object co-operation, which has been studied so far, has a limitation in that it can not obtain an accurate alpha channel mask of an object having a complex outline such as hair.

A problem to be solved by the present invention is to limit the initial foreground area of each viewpoint image to a volume of interest in the space including the object, to divide the multi-viewpoint object into two, and to generate an uncertain area And extracting an alpha channel mask of an object having a contour outline by searching and matting the object with a dynamic window.

According to another embodiment of the present invention, there is provided a method of operating an object extraction apparatus operated by at least one processor, the method comprising: extracting a volume of interest including an object in a three-dimensional space based on a plurality of viewpoint images, Projecting the volume of interest to each point-in-time image to obtain an initial foreground region containing the object in each point-in-view image; generating a geometric energy model initialized from the initial foreground region and an outline energy model associated with color and texture, Extracting a foreground region of each viewpoint image converged from the initial foreground region while updating, and outputting a binary mask dividing foreground and background of each viewpoint image.

The extracting of the volume of interest may extract the three-dimensional space in which the frustums of the camera view related to each of the plurality of viewpoint images overlap, into the volume of interest.

The geometric energy model indicates whether an area inferred as a foreground or a background in a point-in-time image is inferred as foreground or background in other warped point-in-time images in a state where each point-in-time image is binary- And may be modeled to measure geometric coherence between time points.

Wherein the step of extracting the foreground region of each viewpoint image comprises: connecting the superpixels of each viewpoint images to corresponding three-dimensional points in the current volume of interest based on the geometric information of each viewpoint image to determine geometric consistency between the points of view; A super-pixel can be a unit for dividing an image.

Wherein extracting the foreground region of each viewpoint image comprises: adding a three-dimensional point around a point at which a geometric coherence score is higher than a reference value in the current volume of interest; Update the current volume of interest, and update the geometric energy model and the external energy model associated with the color and texture based on the updated volume of interest.

The outer shape energy model related to the texture is modeled to calculate energy using texture information shared in the plurality of viewpoint images, and the outer shape energy model related to the color includes color information of the foreground and background inferred from each viewpoint image To calculate the energy.

Wherein the external energy model related to the texture is modeled to calculate classification scores related to the texture of each super pixel based on classifiers learned with texture properties of foreground and background deduced from the plurality of viewpoint images, Lt; / RTI >

The method includes generating a trimap that divides the perimeter of the boundary into a background, a foreground, and an uncertain boundary region based on color information and geometric information around the perimeter of the binary mask, And generating an alpha channel mask by mating an uncertain boundary region.

In accordance with another embodiment of the present invention, there is provided a method of operating an object extraction apparatus operated by at least one processor, the method comprising: inputting a binary mask of an input image; Calculating entropy to dynamically determine a local window size corresponding to each pixel, dividing the region of each local window into a background, a foreground, and an uncertain boundary region to produce a trimap, And extracting an alpha channel value of the triimage to generate an alpha channel mask.

The step of dynamically determining the local window size may include calculating a Kullback-Leibler divergence in a plurality of window sizes at the sampled pixels, and extracting a window size at which the Kullback-Leibler divergence becomes maximum .

The step of generating the tri-map may divide the area of each local window into a background, a foreground, and an uncertain boundary area using a Markov random field (MRF).

The method of operation may further include acquiring the binary mask by space-dividing an object included in the plurality of view-point images, wherein the input image includes an arbitrary viewpoint image of a plurality of viewpoint images, Lt; / RTI >

The matting equation may include constraints for geometrically sharing the matting results of the plurality of viewpoint images.

According to another embodiment of the present invention, there is provided an object extracting apparatus which is operated by at least one processor. The object extracting apparatus extracts a volume of interest including an object in a three-dimensional space based on a plurality of view images, A co-splitting unit splitting the foreground and the background in the plurality of viewpoint images using the geometric constraint on the volume of interest and the color and texture information of the plurality of viewpoint images, And extracts an alpha channel value of the triumph in a mathematical expression by extracting an alpha channel value of the triumph by a matting expression, And generates an alpha channel mask.

Wherein the sharing unit extracts a three-dimensional space in which the frustums of the camera view related to each of the plurality of viewpoint images are overlapped with the volume of interest, projects the volume of interest into each viewpoint image, And extracts the foreground region of each viewpoint image converged from the initial foreground region while updating the geometric energy model initialized from the initial foreground region and the outline energy model related to color and texture .

The geometric energy model is used to determine whether superimposition of superimposed superimposed superimposed superimposed superimposed superimposed superimposed superimposed superimposed superimposed superimposed superimposed superimposed superimposed superimposed superimposed superimposed superimposed superimposed superimposing images Wherein the external energy model associated with the texture is based on a classifier learned with texture properties of the foreground and background deduced from the plurality of viewpoint images, the texture of each superpixel based on the learned classifier, And the super-pixel may be a unit for dividing the image.

The extraction unit may calculate the entropy of the foreground and the background in the pixels sampled around the boundary line of the binary mask to dynamically determine the size of the local window corresponding to each pixel and set the area of each local window as the background, It is possible to generate the triimage by dividing the boundary region into the boundary regions.

According to the embodiment of the present invention, it is possible to extract the alpha channel mask of an object having a complicated and discrete contour line precisely. The object segmentation and alpha channel mask extraction method according to the embodiment of the present invention can be applied to advanced image processing, robotics, computer vision and graphics application, and accurate objects can be extracted.

1 is a configuration diagram of an object extraction apparatus according to an embodiment of the present invention.
2 and 3 are diagrams for explaining the initial foreground region and the binary mask extraction method for each viewpoint image.
4 is an illustration of an alpha channel mask extracted from an object extraction apparatus according to an embodiment of the present invention.
5 is a flowchart of a multi-point object cooperative method according to an embodiment of the present invention.
6 is a diagram illustrating a geometric constraint according to one embodiment of the present invention.
7 is a view for explaining a texture energy model according to an embodiment of the present invention.
8 is a flowchart of an alpha channel mask extraction method according to an embodiment of the present invention.
9 is an exemplary diagram for explaining an alpha channel mask extraction method according to an embodiment of the present invention.
10 is a view for explaining a method of detecting a matching area through local window optimization according to an embodiment of the present invention.
11 is a view for explaining an initial foreground area setting by a user according to an embodiment of the present invention.
12 is an example of a result of extracting a multi-view object according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

Throughout the specification, when an element is referred to as "comprising ", it means that it can include other elements as well, without excluding other elements unless specifically stated otherwise. Also, the terms " part, "" module," and " module ", etc. in the specification mean a unit for processing at least one function or operation and may be implemented by hardware or software or a combination of hardware and software have.

Hereinafter, the object extraction apparatus 100 extracts the alpha channel mask of the object by dividing the multi-view images into two, but may extract the alpha channel mask of the object included in the single view image.

FIG. 1 is a block diagram of an object extracting apparatus according to an embodiment of the present invention. FIGS. 2 and 3 are views for explaining an initial foreground region and a binary mask extracting method for each viewpoint image, and FIG. FIG. 8 is an illustration of an alpha channel mask extracted from an object extraction apparatus according to an embodiment. FIG.

Referring to FIG. 1, the object extracting apparatus 100 may share an object having a kinked outline in each viewpoint image, sharing geometric information and texture information between the viewpoints, based on a plurality of viewpoint images obtained by shooting an object at a different point do. The object extraction apparatus 100 extracts an alpha channel mask of an object by matching an uncertain boundary of the binary mask of each view image. Thereafter, the object extracting apparatus 100 extracts only the object by applying an alpha channel mask to the captured image, and can change the background in various ways. The alpha channel mask can be referred to as various terms such as an alpha mask, an alpha channel, and the like, which separates an object region into white and a background into black so that the objects included in the color image can be separated from the background.

The object extracting apparatus 100 includes an input unit 110 operated by at least one processor and operated by at least one processor, a multi-point object sharer 130, and an alpha channel mask extractor 150 . Each of the input unit 110, the multi-point object covariance unit 130, and the alpha channel mask extraction unit 150 may be physically implemented or implemented as a software functional block.

The input unit 110 receives a plurality of viewpoint images obtained by capturing an object at a different point.

The multi-viewpoint object sharer 130 extracts a volume of interest including an object in a three-dimensional space based on a plurality of view images obtained by capturing an object at a different point. The multi-point object collation unit 130 can extract a space in which the frustums of the camera viewpoints overlap, as a volume of interest. The re-point object co-ordinator 130 can optimize the size of the volume of interest based on the initial structure of the foreground (three-dimensional points in the volume of interest) or the volume of interest input by the user (see FIG. 10). In particular, the initial foreground region of the multi-view video segmentation is obtained on the assumption that the object is completely visible to all cameras. Here, the camera posture information can use a conventional multi-point structure restoration algorithm (structure from motion), and the multi-point object covariance block 130 further restricts the initial foreground area using restored three-dimensional points, Accuracy can be increased.

The multi-point object collation unit 130 projects an interested volume for each viewpoint image to obtain an initial foreground area including the object. Then, the multi-point object collation unit 130 repeatedly updates the geometric model initialized from the initial foreground region for each viewpoint image and the appearance model related to color and texture, thereby obtaining a specific region (object region) To obtain a converged final foreground area. The multi-viewpoint object sharer 130 extracts a binary mask that divides each viewpoint image into foreground and background. At this time, the multi-point object covariance unit 130 models each of the geometric model and the external model as an energy term of a Markov random field (MRF) as shown in Equation (1) Binary division into background. The outline model and the geometric model are designed to share the common color and texture of the multi-view images and the geometric assumptions, thereby obtaining a binary mask in which the inter-view features are shared.

)가 결정될 수 있다. 평활화항은 외형 정보 중 컬러만을 주요하게 사용하기 위한 평활화용 컬러항(Enc), 그리고 모든 시점에 적용되는 기하학 정보를 사용하기 위한 평활화용 기하학항(Eng)으로 설계될 수 있다. λ_nc와 λ_ng은 가중치 값이다.As shown in Equation (1), the MRF energy term can be composed of a data term (E _d ) and a smoothness term (E _n ). The data term (E _d ) can be designed as a geometric term (E _g ) modeled to take into account the appearance term (E _a ) modeling the foreground and background contours, respectively, and geometric constraints . The relative influence of E _a and E _g depends on the reliability of the external model

) Can be determined. The smoothing term may be designed as a smoothing color term (Enc) for mainly using only color among appearance information, and a geometry term (Eng) for smoothing to use geometry information applied at all times. λ _nc and λ _ng are weight values.

Referring to FIG. 2, the multi-point object sharer 130 projects an initial foreground area 10 including an object by projecting a volume of interest for each view image. The multi-point object sharer 130 repeatedly applies an iterative graphcut from the initial foreground area 10 to obtain a boundary line 20 between the foreground and the background. The border line 20 bisects the foreground and background.

Referring to FIG. 3, the multi-viewpoint object shaper 130 includes a plurality of viewpoint images acquired by the camera as it approaches the object as shown in (a), a plurality of viewpoint images acquired by the camera Images, or a plurality of viewpoint images acquired by the camera while passing through objects at various distances, as shown in (c), can be extracted as a volume of interest (red convex volume) that is common to each viewpoint. Then, the multi-point object collation unit 130 obtains an initial foreground region (blue) including an object by projecting a volume of interest for each viewpoint image, and obtains a foreground region (green) converged to the object outline.

In this manner, the multi-point object sharer 130 extracts the space common to each viewpoint as the volume of interest and obtains the initial foreground area of each viewpoint image, so that there is no restriction on the arrangement and movement of the camera. That is, the multi-viewpoint object sharer 130 does not necessarily require a plurality of cameras, and can selectively process images taken by one moving camera. Accordingly, the present invention has an advantage in that it can be used for extracting objects of multi-view images photographed under various conditions, as compared with the conventional multi-viewpoint object sharing method of acquiring an initial foreground area as camera posture information.

Referring again to FIG. 1, the alpha channel mask extracting unit 150 detects a matching area that needs to be matched by checking the boundary line of the binary mask output from the multi-point object shaper 130, and generates a triangle do.

)을 추정하여 경계선을 매팅한다. 알파 채널 마스크 추출부(150)는 수학식 2와 같은 매팅 식을 통해 알파 채널 값(

)을 구할 수 있다.The alpha channel mask extraction unit 150 dynamically determines the shape and the size of the window along the outline of the object to divide the tri map, and then, for each window, the alpha channel value of the object (alpha matte,

) And then marsh the boundary line. The alpha channel mask extraction unit 150 extracts alpha channel values (

) Can be obtained.

In Equation (1), L is a similarity matrix normalized with a matting affinity matrix. W is a diagonal matrix. m _c is a hard constraint obtained by converting the result of the binary division into a tri-map, m _g is a value for using the result of the binary division obtained from the geometry model shared by all the view images, Is a soft constraint, and the mapped results at each time point are geometrically shared by m _g . λ _c and λ _g are weight values.

Referring to FIG. 4, the alpha channel mask extraction unit 150 obtains the alpha channel masks (30, 31) at each view by matching the outlines of the respective viewpoints. Only the objects 40 and 41 at each view point can be extracted through the alpha channel mask.

FIG. 5 is a flowchart of a multi-point object co-ordinating method according to an embodiment of the present invention, FIG. 6 is a view for explaining a geometric constraint according to an embodiment of the present invention, And Fig.

Referring to FIG. 5, the multi-point object sharer 130 projects an interest volume for each view image to obtain an initial foreground area including the object (S110). The volume of interest can be an convex space consisting of three-dimensional points that contain objects and are observed at all times.

The multi-view object collation unit 130 extracts features of each viewpoint image (S120). The feature may include information of the image vectors ( I ) and superpixels ( S ) that have divided the image. A superpixel may be a collection of a plurality of pixels.

In step S130, the multi-point object sharer 130 determines whether the currently inferred foreground is in a convergence state based on the geometric information of the foreground and background derived from each viewpoint image, and the appearance information related to color and texture. The multi-point object covariance unit 130 determines whether the designed MRF energy is maximized as shown in Equation (1).

If not, the multi-viewpoint object sharer 130 updates the area inferred as foreground in each view image (S140). All data terms and smoothing terms in the MRF are computed each time the foreground / background is updated. The updated foreground converges gradually into an object area enclosed by a decimal outline.

In the convergence state, the multi-point object sharer 130 outputs a binary mask that divides the current foreground and the background (S150).

In the following, a method for dividing an image based on the MRF modeled by geometric energy and external energy in step S130 and step S140 will be described in detail.

First, we explain how to calculate and limit the geometric energy of MRF.

Referring to FIG. 6, the multi-viewpoint object coordinator 130 determines whether the area inferred as the foreground / background in a certain point-in-time image in the current state in which each point-in-time image is binary- (Geometric coherence) is measured. If an area inferred as foreground / background in a point-in-time image is inferred as foreground / background in other point-in-view images, this area can be said to be true foreground / background. To this end, the multi-point object sharer 130 warps each superimposed viewpoint (superpixel or pixel) from the viewpoint image to the foreground / background based on the geometric information of each viewpoint image Evaluate whether the point is equally inferred as foreground / background.

The multi-point object coordinate unit 130 models the geometric energy (E _g ) using a point-to-point geometric consistency score.

]는 수학식 3과 같이 모델링될 수 있다. 전경에 대한 기하학 에너지[

]는

로 모델링될 수 있다. 수학식 3에서, k는 픽셀이고, en_b는 배경에 연결된 에너지의 최대값으로서 예를 들면 10일 수 있다. V_k는 픽셀k에 대한 시점간 기하학적 일관성 점수이고, V_th는 임계값이다. 평활화용 기하학 에너지(Eng)는 수퍼픽셀들의 시점간 연결(inter-view links)를 고려하도록 모델링된다. Geometric energy for background [

] Can be modeled as shown in Equation (3). Geometric energy for foreground [

]

Lt; / RTI > In Equation (3), k is a pixel and en _b is a maximum value of energy connected to the background, for example, 10. V _k is a point-to-point geometric consistency score for pixel k, and V _th is a threshold. The smoothing geometry energy (Eng) is modeled to take into account the inter-view links of superpixels.

The multi-point object coordinator 130 can update the geometric energy model by further restricting the volume of interest. That is, the multi-point object coarse dividing unit 130 samples three-dimensional points in the volume of interest. The multi-viewpoint object sharer 130 connects the super-pixels of each viewpoint image with each of the three-dimensional points based on the geometric information. Referring to Figure 6 (a), point A is maintained at a valid three-dimensional point in the volume of interest since it is linked to superpixels of the two viewpoint images (i. E. The invisible B point is excluded from the volume of interest. Point C does not have a geometric relationship between points in the MRF optimization, but is referred to in determining geometric consistency for superpixels. The multi-point object coordinate unit 130 can use the graph model shown in FIG. 6 (b) for geometric modeling.

The multi-point object covariance unit 130 can check the geometric consistency by giving more three-dimensional points to the region having a high coherence score and delete the invisible three-dimensional points at all points in time. The foreground area of each viewpoint image is updated with the corrected volume of interest.

Next, a method of calculating the external energy of the MRF will be described. The multi-viewpoint object shaper 130 obtains the external shape energy E _a related to color and texture, as shown in Equation (4), in a current state in which each viewpoint image is divided into a foreground and a background. At this time, the multi-point object collation unit 130 calculates energy (E _c ) related to the color using the color information of the foreground and background inferred for each viewpoint image, and the energy ( _Et ) related to the texture is all And calculates using the texture information shared at the viewpoint. The energy (E _c ) associated with the color can use a known energy model. The texture energy model of the present invention can compensate for the ambiguity of the initial color energy model.

A method of modeling the energy (E _t ) related to the texture will be described with reference to FIG.

Referring to FIG. 7A, the multi-point object covariance unit 130 generates a feature vector representing each super-pixel as a texture. The multi-viewpoint object collation unit 130 learns the classifier based on areas previously inferred as foreground / background. Then, the multi-point object covariance unit 130 allocates classification points for each super pixel of the area inferred as foreground / background through the learned classifier. The multi-point object shaper 130 models the texture energy E _t using classification scores related to the texture of each super-pixel. The texture energy model is an important clue when a foreground object in which a specific pattern is repeatedly found is observed in a background with a relatively unusual texture. On the other hand, the feature vector may be a descriptor that simultaneously expresses color and texture characteristics.

Referring to FIG. 7 (b), in the MRF, one superpixel is connected to neighboring superpixels, and neighboring superpixels can be connected by the similarity of color and texture between superpixels.

The multi-point object segmentation considers recall rather than precision as compared to single-view image segmentation. This is because a multi-viewpoint object segmentation that repeatedly uses geometric constraints can limit the segmentation result at other points even if the result of the segmentation contains some noise and the precision is low, but important parts such as hands or feet are inferred as foreground If the recurrence rate drops, it will be propagated at all times.

At the initial stage of MRF, texture information contributes to the detection of approximate foreground regions at higher recall than color information or geometric constraints. However, the higher the precision of the inferred foreground, the more geometric constraints may be more effective. Thus, weights of texture information, color information, and geometric information can be adjusted while repeating MRF.

Since the foreground and background of each viewpoint are modeled differently from texture information or geometric information, color information can be provided with adaptive weights to enhance the stability of the algorithm. In other words, it allows a large change in the initial stage of the MRF, but gradually increases the weight of the geometric energy term as the foreground object boundary is exposed, so that only small changes occur at all times.

After the super pixel division is completed, the multi-viewpoint object sharer 130 can perform division by pixel at least once. At this time, the multi-point object covariance unit 130 can obtain the final binary division result (binary mask) of each viewpoint image using only the color information and the geometric constraint.

FIG. 8 is a flowchart illustrating a method of extracting an alpha channel mask according to an exemplary embodiment of the present invention, and FIG. 9 is an exemplary view illustrating an alpha channel mask extraction method according to an embodiment of the present invention.

8, the alpha channel mask extraction unit 150 checks the boundary lines (refer to 50 and 51 in FIG. 9A) of the binary masks output from the multi-point object collation unit 130, Detects the matting area, and generates an initial triimap (S210). 9 (b), the tri-map 60 displays a certain background (black), a certain foreground (white), and an uncertain boundary area (gray). The alpha channel mask extraction unit 150 detects the matching area by determining a window size (a window size that effectively separates foreground and background) that maximizes entropy between foreground and background in an outline pixel. Referring to FIG. 9, the window (yellow box) size differs depending on the sample point of the object. The alpha channel mask extraction unit 150 can sample the outline pixels necessary for the calculation amount and dynamically determine the shape and size of the window by examining entropy. The entropy between the foreground / background can be measured by the Kullback-Leibler divergence method. In a window of a size that maximizes the Kullback-Leibler dispersion, the two probability distributions of the mixed foreground / background samples are minimally overlapped, making the two groups easily separable.

The alpha channel mask extracting unit 150 optimizes the tri-map so that a reliable background, a certain foreground, and an uncertain region are optimally divided in each local window on the outline (S220). At this time, the alpha channel mask extraction unit 150 divides (Trimap segmentation) a definite background, a reliable foreground, and an uncertain boundary region for each local window using the color information and the geometric information in the MRF (alpha-expansion) Create a map. The key assumption here is that the foreground and background samples of the local part can be represented linearly, thereby classifying the samples that are the most distant from these two groups into samples that require matting. 9 (b), after the tri-map optimization, the initial tri-map can be corrected by a post-processing operation that increases the uncertain area.

)을 구할 수 있다. The alpha channel mask extraction unit 150 mats an uncertain region of the tri-map using a matting expression including a restriction condition (m _g ) for keeping the binary division result obtained from the geometric model shared by the viewpoint images (S230). The alpha channel mask extraction unit 150 extracts the matting expression of Equation (2) to obtain an alpha channel value

) Can be obtained.

The alpha channel mask extraction unit 150 outputs an alpha channel mask in which the matting area of the triam is precisely filled (S240). Referring to FIG. 9C, the alpha channel mask 70 extracted from the tri-map can be confirmed. After matting, foreground and background color values can be deduced more accurately depending on the application.

10 is a view for explaining a method of detecting a matching area through local window optimization according to an embodiment of the present invention.

Referring to FIG. 10 (a), the alpha channel mask extraction unit 150 sets windows having various shapes and sizes. 10B, the alpha channel mask extraction unit 150 calculates the Kullback-Leibler divergence using various windows along a constrained path, and then calculates a Kullback- Determine shape and size. At this time, the alpha channel mask extraction unit 150 can smoothly change the window shape and size on the path.

11 is a view for explaining an initial foreground area setting by a user according to an embodiment of the present invention.

Referring to FIG. 11, the cooperative method of sharing information between the viewpoints can effectively propagate the manual input information of the user. For example, FIG. 11 (a) shows a result of convergence of the coarse division into the ambiguous initial foreground region.

The object extracting apparatus 100 receives a box 80 containing an object for one chapter at a specific time point (for example, a front view point) from the user, as shown in (b) of FIG. The region of interest in space is further limited according to the given box 80, and this information is propagated to all points of view and recalculated according to the area in which the MRF data terms are updated. Finally, converge to the correct foregrounds at all points (c) and (d).

12 is an example of a result of extracting a multi-view object according to an embodiment of the present invention.

Referring to Fig. 12, the data sets used for verification are a sofa, a teddy bear, a tree, a lion 1, and a lion 2. The Tree and Lion 1 includes 12 view images collected with a turntable (SNRT400-Solutionix) and two cameras (Flea2-PGR) with a resolution of 1280 × 960. Lion 2 includes images taken by moving a DSLR camera (Canon DSLR Mark 3) with a resolution of 5184 × 3456. Eight of the calibrated camera inputs are selected to apply the algorithm of the present invention, and the rest is shared by correct geometric models.

In FIG. 12, the first column is the input image of the front view of each object. The second column is the result of the binary division which is the intermediate result of the object extracting apparatus 100. The third column represents the alpha channel mask estimated from the intermediate output of the object extraction apparatus 100 and the object extracted using the alpha channel mask. Referring to FIG. 12, the present invention can obtain a cooperative result sharing information between multi-view images, and can extract the alpha channel mask of the object very precisely.

The embodiments of the present invention described above are not implemented only by the apparatus and method, but may be implemented through a program for realizing the function corresponding to the configuration of the embodiment of the present invention or a recording medium on which the program is recorded.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, It belongs to the scope of right.

Claims (17)

CLAIMS What is claimed is: 1. An operating method of an object extraction device operated by at least one processor,
Extracting a volume of interest including an object in a three-dimensional space on the basis of a plurality of viewpoint images obtained by capturing an object at a re-point,
Projecting the volume of interest to each viewpoint image to obtain an initial foreground area containing the object in each viewpoint image,
Extracting a geometric energy model initialized from the initial foreground region and a foreground region of each viewpoint image converged from the initial foreground region while updating an external energy model associated with color and texture,
Outputting a binary mask that divides the foreground and background of each viewpoint image
Lt; / RTI >
The step of extracting the volume of interest
Extracting, into the volume of interest, a three-dimensional space in which frustums of camera views related to each of the plurality of viewpoint images overlap. delete The method of claim 1,
The geometric energy model
The geometric consistency between the viewpoints, which indicate whether the viewpoint image is inferred as the foreground or background in other warped viewpoint images, where the viewpoint image is inferred as foreground or background in the viewpoint image, geometric coherence). 4. The method of claim 3,
The step of extracting the foreground region of each viewpoint image
Connecting the super-pixels of each viewpoint image to the corresponding three-dimensional point in the current volume of interest based on the geometric information of each viewpoint image to determine geometric consistency between the viewpoints,
Wherein the super-pixel is a unit for dividing an image. 5. The method of claim 4,
The step of extracting the foreground region of each viewpoint image
Updating the current volume of interest by adding a three-dimensional point around a point at which a geometric coherence score is higher than a reference value in the current volume of interest, deleting invisible three-dimensional points in the plurality of images from the current volume of interest,
And updating the geometric energy model and the cosmetic energy model related to the color and texture based on the updated volume of interest. The method of claim 1,
The external energy model related to the texture
Modeled to calculate energy using texture information shared in the plurality of view-point images,
The external energy model associated with the color
Is modeled to calculate energy using color information of foreground and background inferred from each viewpoint image. The method of claim 6,
The external energy model related to the texture
And calculating a classification score related to the texture of each superpixel based on the classifier learned by the texture characteristics of the foreground and background deduced from the plurality of viewpoint images,
Wherein the super-pixel is a unit for dividing an image. The method of claim 1,
Generating a trimap that divides the perimeter of the boundary into a background, a foreground, and an uncertain boundary region based on color information and geometric information around the perimeter of the binary mask; and
Matting the uncertain boundary region of the tri map to generate an alpha channel mask
≪ / RTI > CLAIMS What is claimed is: 1. An operating method of an object extraction device operated by at least one processor,
Extracting a volume of interest in which the frustums of the plurality of view-point images that have photographed the object at the multiple points overlap,
Projecting the volume of interest to an arbitrary viewpoint image of the plurality of viewpoint images to divide the arbitrary viewpoint image into a foreground and a background including the object;
Obtaining a binary mask of the arbitrarily-point-separated image,
Calculating entropy of foreground and background in the pixels sampled around the perimeter of the binary mask to dynamically determine the local window size corresponding to each pixel,
Dividing the region of each local window into a background, a foreground, and an uncertain boundary region to create a trimap; and
Extracting an alpha channel value of the tri-map in a matting manner to generate an alpha channel mask
≪ / RTI > The method of claim 9,
The step of dynamically determining the local window size
Calculating a Kullback-Leibler divergence in a plurality of window sizes at the sampled pixels, and extracting a window size that maximizes the Kullback-Leibler divergence. The method of claim 9,
The step of generating the tri-map
A method of dividing each local window region into a background, a foreground, and an uncertain boundary region using a Markov random field (MRF). delete The method of claim 9,
The matting equation
And constraints for geometrically sharing the matting results of the plurality of viewpoint images. 1. An object extraction device operated by at least one processor,
Extracting a volume of interest including an object in a three-dimensional space on the basis of a plurality of viewpoint images obtained by capturing an object at a plurality of viewpoints, and using geometric constraints on the volume of interest and color and texture information of the plurality of viewpoint images A co-division unit for co-splitting the foreground and the background in the plurality of viewpoint images, and
A binary mask as a division result of foreground and background for each viewpoint image is input from the multi-point object covariance unit, a region around the boundary line of the binary mask is generated as a trimap, An extraction unit for extracting an alpha channel value of the tri-map and generating an alpha channel mask,
Lt; / RTI >
The co-
And extracts a three-dimensional space in which the frustums of the camera view related to each of the plurality of viewpoint images overlap with the volume of interest. The method of claim 14,
The co-
Projecting the volume of interest into each viewpoint image to obtain an initial foreground region containing the object in each viewpoint image, updating the geometric energy model initialized from the initial foreground region and the exterior energy model associated with color and texture, An object extraction apparatus for extracting a foreground region of each viewpoint image converged from a foreground region. 16. The method of claim 15,
The geometric energy model
The geometric consistency between the point-in-time and the point-in-time points indicating whether the super-pixel inferred from the point-in-time image as the foreground or background is inferred as foreground or background in the warped other point-in- lt; / RTI > is modeled to measure geometric coherence,
The external energy model related to the texture
And calculating a classification score related to the texture of each superpixel based on the classifier learned by the texture characteristics of the foreground and background deduced from the plurality of viewpoint images,
Wherein the super-pixel is a unit for dividing an image. The method of claim 14,
The extracting unit
The entropy of the foreground and background in the pixels sampled around the boundary line of the binary mask is calculated to dynamically determine the size of the local window corresponding to each pixel, and the area of each local window is divided into the background, foreground, and uncertain boundary region And generating the triimage by dividing the object.

Images