KR100953738B1 - Apparatus for segmenting multi-view images - Google Patents

Abstract

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to image processing technology, and to an image segmentation apparatus for localizing a multiview image set.

An image segmentation apparatus according to an embodiment of the present invention includes an approximation unit for approximating a foreground and a background of a multi-view image set by using segmentation information of one image, an approximated foreground and It includes a localization unit for localizing a multi-view image set using the background information.

Multiview Image Set, Segmentation

Description

Image segmentation device {APPARATUS FOR SEGMENTING MULTI-VIEW IMAGES}

The present invention relates to an image processing technology, and more particularly, to an image segmentation apparatus for localizing an image set obtained by photographing an object at various viewpoints.

Image segmentation is a technique of classifying images by grouping similar parts. Such image segmentation is used as a preprocessing process such as recognition and 3D structure reconstruction in computer vision and graphics fields, and is used for intermediate processing such as image synthesis and virtual reality, including background substitution.

In recent years, there is an increasing demand for collectively editing image sets (hereinafter, referred to as multi-view image sets) photographed at various viewpoints on one object. This is because the demand for the 3D structure reconstruction of the object is improved through the multi-view stereo technique using a multi-view image set as input data.

Such a multi-view image set segmentation technique includes a technique of receiving the entire viewpoint information of the multi-view image set from the user and terminating it using a boundary propagation information through distance transformation, There is a technique of territorialization through the 3D structure restoration process of the foreground object.

However, a technique for receiving and visualizing entire viewpoint information on a multiview image set from a user is inconvenient for the user to input all viewpoint information. The technique of segmentation using boundary propagation information through distance transformation treats a set of images in the same way as a video image, so it is difficult to apply them to images with various viewpoint changes. The technology of territorialization through the 3D structure restoration process becomes complicated by the system configuration through the 3D structure restoration process.

Accordingly, an object of the present invention is to provide an image segmentation apparatus that can simplify the segmentation of a multi-view image set.

According to an aspect of the present invention, there is provided an apparatus for approximating a foreground and a background of a multi-view image set using segmentation information of one image of an object. part; And a localization unit for localizing the multi-view image set using the approximated foreground and background information.

The segmentation information of the one image may include at least one of a color distribution histogram of a foreground and a background, a foreground region, and outline information of the foreground region of one image of the multiview image set.

The approximation unit may approximate the foreground and the background between the multiview image sets using the feature points of the multiview image sets.

The approximation unit extracts a feature point of a neighboring image to receive the background region and the approximation information of the current image, matches it using nearest neighbor matching and random sample consensus (RANSAC), and then uses the corresponding feature points to transform the background change into a transformation matrix. Can be approximated

The approximation unit may approximate the foreground change of the multi-view image set through registration using the feature points of the image.

The approximator may register the target image by transforming the image using a free-form doformation model of a regular mesh based on cubic B-splines.

The foreground and background approximation information may include at least one of a region estimated as a foreground and a background in a neighboring image, a pixel set corresponding to the region estimated as the foreground and a background, and a pixel probability distribution of each of the foreground and background of a previous image. Can be.

The localization unit may re-approximate the prediction probability distribution of the foreground pixels and the background prediction pixels of the neighboring pixels using the pixels estimated as the foreground in the adjacent image.

The segmentation unit may include a prediction probability distribution of foreground pixels and a background prediction pixel, a probability distribution of foreground pixels of a previous image, and a weighted average of probability distributions of background pixels of a neighboring image re-approximated by damping. For example, among the pixels used for re-approximation, a large value among the pixels overlapping the foreground and the background and the possibility of the pixel probability distribution of the previous image may be excluded from the label and pixels different from the labels of the current area pixels. .

The localization unit may localize the image by allocating a label indicating each area of the foreground and the background to each pixel.

As described above, the apparatus for image segmentation according to an embodiment of the present invention sequentially approximates the foreground and the background with respect to a multi-view image set based on a minimum user input, and makes a region by using the approximated information. Segmentation of a multiview image set can be performed simply.

Hereinafter, with reference to the accompanying drawings will be described a preferred embodiment of the present invention; In the following description of the present invention, if it is determined that detailed descriptions of related well-known functions or configurations may obscure the gist of the present invention, the detailed description will be omitted. In addition, terms to be described below are terms defined in consideration of functions in the present invention, which may vary according to intention or custom of a user or an operator. Therefore, the definition should be made based on the contents throughout the specification.

1 is a diagram illustrating a configuration of an image segmentation apparatus according to an embodiment of the present invention.

As shown in FIG. 1, an image segmentation apparatus according to an embodiment of the present invention includes an approximation unit 10 and a regionization unit 20.

The approximation unit 10 approximates the foreground and the background of the multi-view image set by using the segmentation information of one image input from the user. Accordingly, the localization unit 20 localizes and outputs a multiview image set by using the foreground and background information approximated by the approximation unit 10.

This will be described in detail.

The approximator 10 sequentially approximates the background and the foreground of the multiview image set by using the segmentation information of one image input from the user and transmits the approximation to the regionr 20. In an embodiment, the foreground and background designated by the user are shown in FIG. 2. FIG. 2A illustrates a case where a part of the foreground 25 and the background 26 are designated by the user, and FIG. 2B illustrates a case where the user schematically designates an area of the foreground 27.

In this case, the localization information input from the user includes at least one of a foreground color and a color distribution histogram of one of the multi-view image sets, a foreground area, and outline information of the foreground area. The segmentation information input from the user may be segmentation information of the first image photographed by the user during the multiview image set.

That is, the approximation unit 10 approximates the background and the foreground of the second image captured by using the area information of the first image photographed by the user, and transmits the approximation to the localization unit 20. The third photographed image is approximated by using the approximated information about the third photographed image. This approximation process is performed sequentially for a multiview image set. To this end, the multi-view image sets should be arranged in the order in which they were taken according to the flow of viewpoints.

In this case, the approximation unit 10 approximates the foreground and the background between the multiview image sets by using feature points of the images of the multiview image sets. The reason for separating and approximating the foreground and the background between the multi-view image sets is to lower the possibility of outliers that are incorrect registration. Here, the feature point represents a point having special properties compared to surrounding pixels in the image. According to an embodiment, the feature point may be a point where the degree of color change is maximized compared to the surroundings. The feature points of the image illustrated in FIG. 3A may be extracted and displayed as rectangles as illustrated in FIG. 3B.

The approximation unit 10 approximates the background change of the multi-view image set with a deformation matrix based on the affine change. That is, the approximation unit 10 extracts the feature points of the neighboring image to receive the background region and the approximation information of the current image, matches them using nearest neighbor matching and RANSAC (RANdom SAmple Consensus) The background change is approximated to the deformation matrix using the corresponding feature points and transmitted to the localization unit 20. At this time, the background is assumed to have no independent movement.

Meanwhile, the approximation unit 10 approximates the change in the foreground by using a registration method using a feature point of the image and transmits the approximation to the areaization unit 20. In this case, registration is a process of displacing another image on a reference image to make the two images as similar as possible. As an example, FIG. 4A shows a reference image, FIG. 4B shows another image to be transformed, and FIG. 4C shows a registered image. As illustrated in FIG. 4, the image illustrated in FIG. 4B may be distorted and overlapped with the reference image of FIG. 4A to obtain the image of FIG. 4C.

Hereinafter, a registration method using feature points of an image will be described.

The approximation unit 10 deforms an image using a free-form deformation model of a regular mesh based on cubic B-splines and registers the target image.

, 너비

인 경우,

의 영역을 각 축에 따라 N 개의 구간으로 나누어,

,

의 간격으로 벌어진 일정한 점을 이용하는 일정한 메쉬

의 변화에 대한 에너지를 아래의 수학식 1을 이용하여 계산한다. In other words, the size of the image

, width

If is

Divide the region of into N intervals along each axis,

,

Mesh using constant points spaced at intervals of

The energy for the change of is calculated using Equation 1 below.

는 메쉬

의 변화상태를 나타내는 벡터이고,

는 대응점의 집합,

는 메쉬

의 점들의 변형(deformation)을 나타내는 에너지, 그리고

는 대응관계로 나타나는 에너지를 나타내며,

이다.here,

Mesh

Is a vector representing the state of change of,

Is a set of corresponding points,

Mesh

Energy representing the deformation of the points of, and

Represents the energy represented by the correspondence,

to be.

는

의 점들에 대한 색인(index)을 나타내며,

은 연속되는 세 점들의 색인의 집합을 나타낸다.At this time,

Is

Index of the points in,

Denotes a set of indices of three consecutive points.

으로

,

,

는 강인한 측정기(robust estimator),

은 강인한 측정기의 입력으로 대입되는 신뢰 반경(confidence radius)을 나타낸다.

는 영역화된 전경에서 추출된 특징점을,

은 인접 영상에서 추출된

의 대응점을, 그리고

는

와 변화상태

를 이용하여

가 변환된 점을 나타내는데, B-스플라인을 이용한 보간(interpolation)을 통해 계산된다. 즉, B-스플라인을 이용한 보간은 아래의 수학식 2을 이용하여 계산된다. Furthermore,

to

,

,

Is a robust estimator,

Denotes a confidence radius that is substituted into the input of a robust meter.

Is the feature point extracted from the areaized foreground,

Is extracted from adjacent images

And the corresponding points of

Is

And state of change

Using

Denotes the transformed point, which is calculated through interpolation using B-splines. That is, interpolation using a B-spline is calculated using Equation 2 below.

는 버림 함수(floor function)를,

,

,

,

, 그리고

과

은 각각

과

차 큐빅 B-스플라인의 기저(basis) 함수를 나타낸다.From here,

Is a float function,

,

,

,

, And

and

Are each

and

Represent the basis function of the cubic B-spline.

는 영상의 변환을 조절하는 일정한 메쉬의 점들의 좌표를 의미한다. Based on the above equations, a state transition S that minimizes energy can be obtained. In this case, a nonlinear conjugate gradient technique is used to obtain a state transition S that minimizes energy. State transition

Denotes the coordinates of the points of a certain mesh that controls the transformation of the image.

When the localization unit 20 receives the approximation information of the foreground and the background of the adjacent image from the approximation unit 10, the localization unit 20 localizes the adjacent image based on the received approximation information. In this case, the approximation unit 10 receives an adjacent image in the foreground and the approximation information, pixel set corresponding to the area which is estimated as a foreground and a background in an adjacent image of a background on from, and the pixel distribution of the foreground and background from the previous image h Contains _seg (O) and h _seg (B) .

The regionr 20 reapproximates the predicted probability distribution h _next (O) of the foreground pixels of the adjacent images using the pixels estimated as the foreground in the adjacent image, and predicts using the pixels estimated as the background as well for the background. Reapproximate the probability distribution h _next (B) . In this case, h means a color distribution histogram of each region.

및

중 큰 값을 갖는 레이블과 현재 해당하는 영역 픽셀의 레이블과 다른 픽셀을 제외한다.

및

은 후술될 수학식 8에 정의된다.On the other hand, the predicted foreground and background areas may not be accurate. However, the color distribution of the foreground and background pixels of the previous image and the adjacent image may be similar. Accordingly, the localization unit 20 obtains a weighted average of the reapproximated probability distribution and the probability distribution of the previous image by applying damping. At this time, the pixels used for the reapproximation are likely to be the pixels overlapping the foreground and the background, and the probability of the pixel probability distribution of the previous image.

And

Exclude the label with the larger value and the pixel that is different from the label of the current area pixel.

And

Is defined in Equation 8 to be described later.

, 재근사에 이용되는 픽셀 수를

이라 하면, 재근사된 전경과 영역화된 이전 영상의 전경의 가중치는 아래의 수학식 3 및 4를 이용하여 구할 수 있다. Each weight of the color probability distribution of the foreground and background pixels of the neighboring image predicted through the reapproximation and the pixel probability distribution of the foreground and background of the previous image will be described. The number of pixels in the regioned background area

, The number of pixels used for reapproximation

In this case, the weights of the re-approximated foreground and the foreground of the localized previous image may be obtained by using Equations 3 and 4 below.

,

로 구해짐을 알 수 있다. 나아가, 배경에 대해서도 수학식 3 및 수학식 4를 이용하여 가중치를 계산할 수 있다.That is, the weights of the re-approximated foreground and the foreground of the localized previous image are respectively

,

It can be seen that. Furthermore, the weight may be calculated for the background using equations (3) and (4).

을 각 노드에 할당하여 영역화 한다. 레이블 할당은 할당된 레이블에 따른 에너지를 최소화시키는 형태로 이루어질 수 있다. 이때, 에너지는 아래의 수학식 5를 이용하여 구해질 수 있다. The localization unit 20 checks whether the pixel belongs to the foreground or the background. Let's take a closer look at this. Each pixel of the image is generated as a node, and a pair of pixels is generated to view a relationship with adjacent pixels as an edge. The set of all nodes is called P, and a label indicating which area corresponds to each pixel

To each node. Label assignment may be in the form of minimizing the energy associated with the assigned label. In this case, energy may be obtained using Equation 5 below.

과

는 각각 에너지의 영역 항(region term)과 경계 항(boundary term)을 나타낸다. 에너지의 영역 항은 각 픽셀이 전경 또는 배경에 속할 가능성을 모든 픽셀에 대해 종합한 값이다. 경계 항은 특정 지점에 경계가 위치하는 에너지를 나타낸다. 이러한 영역 항과 경계 항은 각각 수학식 6 및 수학식 7을 이용하여 구해질 수 있다. 또한,

는

과

의 실험적인 가중치를 나타낸다.From here,

and

Denote regions of energy and boundary terms, respectively. The area term of energy is the sum of the possibilities for each pixel that each pixel belongs to the foreground or background. The boundary term represents the energy at which the boundary is located at a particular point. These region terms and boundary terms may be obtained using Equations 6 and 7, respectively. Also,

Is

and

Represents the experimental weight of

이다.Where N represents a set of all pairs of adjacent pixels,

to be.

는 픽셀

에 레이블

를 할당할 때의 에너지로, 픽셀

의 색 또는 밟기 값

에 따라 근사된 전경 및 배경의 픽셀 확률분포를 이용하여 계산되며, 아래의 수학식 8을 이용하여 구해진다.

Is a pixel

Labels on

Is the energy at which to assign

Color or tread value

It is calculated using the approximate pixel probability distribution of the foreground and background according to the equation, and is obtained using Equation 8 below.

,

,

와 전파된 예상 경계의 거리 변환에 의해 결정되는

로 이루어진다. 이때,

는 아래의 수학식 9에 의해 구해질 수 있다.The boundary term is determined by the gradient value, which is the difference in color or brightness of adjacent pixels.

Determined by the distance transformation of

Is made of. At this time,

Can be obtained by Equation 9 below.

이고,

는

와

의 거리를 나타내며,

는 벡터의

노름(norm)을 나타내며,

는 전체 영상 내의 평균을 의미하며, 영역 간의 경계는 차이가 큰 곳에 위치한다는 가정하에 두 픽셀 값의 차이가 클수록 작은 값을 가지게 된다. At this time,

ego,

Is

Wow

Indicates the distance of,

Of the vector

Represents norm,

Means the average in the entire image, and the larger the difference between two pixel values is, the smaller the value is, assuming that the boundary between regions is located at a large difference.

는 아래의 수학식 10에 의해 구해진다.Meanwhile,

Is obtained by the following equation (10).

는 가장 가까운 예상 경계 지점으로부터

까지 거리를 의미하며, 예상 경계 지점과 멀리 떨어진 지점에 경계가 있을수록 에너지가 커지게 된다. At this time,

From the nearest expected boundary point

The distance to the distance, the farther away from the expected boundary point, the greater the energy.

Subsequently, the localization unit 20 calculates an energy value and then derives a label of each node using a grab cut technique. At this time, the grab cut technique extends the black and white image segmentation technique to the color image using the graph cut technique, which is a theory of modeling various phenomena using a graph composed of nodes and edges connecting the nodes. One technique.

As an example of the graph cut technique, when modeling energy or flows, a numerical value representing the maximum flow capacity between nodes at an edge is specified, and a special node source in which the flow occurs and a special flow out flow flow. Define and model node sinks. In this flow graph we find the max-flow through the graph from the source to the sink. In this case, the maximum flow can be obtained through a method of cutting the entire graph at the lowest cost by cutting the edges between the nodes when the edges are replaced to indicate the strength of the connection between the nodes.

In the segmentation using a grab cut, pixels of an image may be defined as nodes of a graph, and edges between adjacent nodes may be defined such that the numerical value is smaller as the color difference between adjacent pixels increases according to the color of each pixel. In addition to the edges between nodes, all nodes are connected to sources and sinks and edges. Source and sink respectively mean foreground and background, and each node reflects how much the color of the corresponding pixel is similar to the foreground color distribution at the edge of the source, and how closely it is similar to the background color distribution. Reflect on.

In this way, the value of the edge with the source or sink node is a value that indicates which area the pixel corresponds to and is called an area term. The edge between nodes is a value indicating the possibility that there is a boundary where the area is divided. The lowest cost split is the sum of the edge numbers that are truncated when splitting the graph by truncating one of the edges with the sink corresponding to the source and background for the foreground, respectively, for all nodes, and cutting the edges between adjacent nodes where the boundary occurs. Indicates the division to be made. By finding the lowest cost division based on the edge values defined in this way, the image can be divided into two regions, a foreground and a background.

As described above, the apparatus for image segmentation according to an embodiment of the present invention sequentially approximates the foreground and the background with respect to a multi-view image set based on a minimum user input, and makes a region by using the approximated information. Segmentation of a multiview image set can be performed simply.

So far I looked at the center of the preferred embodiment for the present invention. Those skilled in the art will understand that the present invention may be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered in descriptive sense only and not for purposes of limitation. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the scope will be construed as being included in the present invention.

1 is a diagram illustrating a configuration of an image segmentation apparatus according to an embodiment of the present invention.

2 is a diagram illustrating a foreground and a background designated by a user according to an embodiment of the present invention.

3 illustrates feature points in accordance with an embodiment of the present invention.

4 is a diagram illustrating a registration image according to an embodiment of the present invention.

Claims (11)

The foreground and background of the multi-view image set are approximated using the segmentation information of one image input from the user, but the background change of the multiview image set is based on the affine change. Is approximated by a deformation matrix, and the image is transformed using a free-form deformation model of a regular mesh based on cubic B-splines to register the target image. An approximation unit for approximating the change of the foreground by the method and transmitting the approximation information of the foreground and the background to the areaizer; And Localize adjacent images based on the approximation information of the foreground and background received from the approximation unit, and segment the corresponding image by assigning each pixel of the image a label indicating which area of the foreground and background is the pixel of the image. An image segmentation apparatus comprising the regionr. delete delete delete delete delete The method of claim 1, Approximation information of the foreground and background, And at least one of a pixel set corresponding to the foreground and the background estimated from a neighboring image, a pixel set corresponding to the foreground and the background estimated, and a pixel probability distribution of each of the foreground and the background of a previous image. delete The method of claim 7, wherein The localization unit, Prediction probability distribution of the foreground pixels of the neighboring image and prediction probability distribution of the background pixels are reapproximated by using the pixels estimated as the foreground in the neighboring image, and the prediction probability of the foreground pixels of the neighboring image reapproximated by damping An image segmentation apparatus for calculating a predicted probability distribution of distribution and background pixels, a probability distribution of foreground pixels of a previous image, and a weighted average of probability distributions of background pixels. delete The method of claim 9, The segmentation unit, the energy according to the assigned label

To obtain using the equation

and

Represents the region term and boundary term of energy, respectively, and the region term of energy is the sum of the probability of each pixel belonging to the foreground or background for all pixels, and the boundary term at a specific point. Represents the energy where the boundary is located,

Is

and

Image segmentation device showing the weight of the.

Images