KR20090111939A - Method and apparatus for separating foreground and background from image, Method and apparatus for substituting separated background - Google Patents

Abstract

PURPOSE: A method for dividing an image into a foreground and a background, an apparatus thereof, a method for substituting the divided background and an apparatus thereof are provided to use a foreground probability model generated by accumulating divided images. CONSTITUTION: An apparatus(10) for dividing an image into a foreground and a background receives an image of frame unit obtained from a camera. A background model generating unit(100) generates a background model which will be compared with the received image. A pixel unit dividing unit(110) calculates similarity between the received image and background model in each pixel belonging to the image, and classifies pixels into foreground pixels and background pixels according to the similarity.

Description

Method and apparatus for separating foreground and background from image, method and apparatus for substituting separated background}

The present invention relates to image processing, and more particularly, to a method and apparatus for separating an image into a foreground and a background, a method and apparatus for replacing a background by separating an image into a foreground and a background, and a program for executing the method. The invention relates to a computer-readable recording medium having recorded thereon.

Techniques for separating images into foreground and background are used in a variety of visual systems, including video surveillance, human counting, and video editing. In particular, with the development of the internet network and the emergence of web cameras, video conversations using both voice and video have been commercialized in recent years. Therefore, a method of protecting an individual's privacy by separating a background from a person who is the subject of communication and transmitting a new background is replaced by a background that is not exposed. In such a background replacement technique, the separation of the image into the background and the person corresponding to the foreground should be preceded.

There is a method of separating a moving object through the background difference with conventional image separation technology. Background difference is a method of difference of the same part from the reference image obtained from the static background for a certain period of time. After this removal process, only moving objects or new objects remain on the screen. Although it is possible to process in real time, it has a disadvantage of being sensitive to changes in lighting, and there is a problem that image separation is not accurate when a person dressed in a color similar to the background appears.

Model-based image separation technology detects the face region using color information to estimate the approximate location of a person, and then separates the image using the human head and torso model assuming that only the upper body of the person appears. Technology. Because of the method of using a model, image separation is not possible when various postures such as the whole body appear or the hand is raised, not the upper body of the person.

Motion-based image separation technology extracts motion information such as optical flow from an input image and extracts the movement of previous and current frames using an algorithm. The extracted motion information is a method of separating images by extracting only the motion of a moving person because the background and the moving person appear differently. According to this technology, if the person does not move, the motion information does not appear, so only the moving person can separate it, and the weak information because the motion information can be extracted accurately only by assuming that there is no light change of the previous frame and the current frame. The disadvantage is that it is sensitive to change.

In addition, there is a method of distinguishing the object and the background by measuring the distance from the camera to the object by using a stereo camera. This method requires a stereo camera, that is, two cameras. There is a significant disadvantage and a disadvantage in terms of cost.

The present invention provides a method and apparatus for separating an image into a foreground and a background, which can accurately separate an image into a foreground and a background in real time using a single camera, and a computer-readable method for executing the method. To provide a recording medium.

Another technical object of the present invention is to accurately separate an image into a foreground and a background in real time using a single camera, and to replace the background by separating the image into the foreground and the background, which can replace the background of the image with another background. A method and apparatus, and a computer readable recording medium for executing the method.

In order to solve the above technical problem, a method of separating an image into a foreground and a background according to the present invention includes (a) comparing an image of an input current frame with a background model, and separating the image into a foreground and a background by a pixel unit. Making; (b) dividing the image into a plurality of regions and extracting unclear regions of the divided regions that are foreground or background regions; (c) classifying the extracted areas into a foreground area and a background area; And (d) correcting the foreground background separation result of the pixels belonging to the extracted areas according to the results classified in step (c).

The method may further include generating the background model, wherein the background model may follow a learned model every frame up to a frame before an object or a person corresponding to the foreground appears.

In operation (a), the image may be separated into a foreground and a background by calculating a similarity between the image and the background image in pixel units and comparing the calculated similarity with a predetermined first threshold.

Further, in step (b), for each of the divided regions, the probability of uncertainty as to whether it is a foreground area or a background area is calculated by using the result separated into the foreground and the background by the pixel unit in step (a), The areas to be uncertain may be extracted by comparing the calculated probability with a predetermined second threshold.

In addition, the step (c), (c1) for each of the extracted areas, calculating the probability of the foreground area; And (c2) classifying the extracted areas into a foreground area and a background area by comparing the calculated probability with a predetermined third threshold.

Further, in the step (a), the similarity between the image and the background image is calculated in pixel units, and the image is separated into the foreground and the background by comparing the calculated similarity with a predetermined first threshold value, and (c1). In step), the probability of the foreground area may be calculated using the calculated similarity.

In addition, the probability of the foreground area may be calculated according to the following equation.

Where P _Fg (R _i ) is the foreground area of the uncertainty region R _i , N _Fg (R _i ) is the number of foreground pixels in the uncertainty area R _i , and N _Bg (R _i ) is the uncertainty area R _i Means the number of background pixels.

The method may further include generating a foreground probability model by accumulating a resultant image separated into a foreground and a background in units of pixels from a predetermined past frame to the current frame in step (a). In step), the probability of the foreground area may be calculated using the generated foreground probability model.

In addition, the foreground probability model may be expressed according to the following equation.

는 프레임 t의 영상에서 화소 p가 전경 화소일 확률을,

는 프레임 t-1의 영상에서 화소 p가 전경 화소일 확률을,

는 프레임 t의 영상에서 화소 p의 전경 분리 결과값을 의미하며,

는 소정 값이다.here,

Is the probability that pixel p is a foreground pixel in an image of frame t,

Is the probability that pixel p is a foreground pixel in the image of frame t-1,

Represents the foreground separation result of the pixel p in the image of the frame t,

Is a predetermined value.

In addition, the probability of the foreground area may be calculated according to the following equation.

Here, P _Fg ^t (R _i ) is the probability of the foreground area of the uncertainty region R _i in the t-th frame that is the current frame, and P _Fg ^t-1 (R _i ) is the uncertainty area in the t-1 th frame that is the previous frame. For the probability that it is the foreground region of R _i , N _Fg (R _i ) and N _Bg (R _i ) represent the number of foreground pixels and background pixels of uncertain region R _i in the t-th frame.

In order to solve the above technical problem, there is provided a computer-readable recording medium having recorded thereon a program for executing a method for separating an image according to the present invention into a foreground and a background.

In order to solve the above technical problem, an apparatus for separating an image into a foreground and a background according to the present invention includes comparing an image of an input current frame with a background model, and separating the image into a foreground and a background by a pixel unit. Separator; An uncertain region extraction unit for dividing the image into a plurality of regions and extracting regions of uncertainty whether the image is a foreground region or a background region; An area unit classifying unit classifying the extracted areas into a foreground area and a background area; And a correction unit configured to correct a foreground background separation result of pixels belonging to the extracted areas according to the result classified by the area unit classifying unit.

In order to solve the above other technical problem, a method of replacing a background by separating an image into a foreground and a background according to the present invention includes (a) comparing the input image with a background model and displaying the image in the foreground and background in units of pixels. Separating into; (b) dividing the image into a plurality of regions, and extracting regions of uncertainty whether the foreground region or the background region is one of the divided regions; (c) classifying the extracted areas into a foreground area and a background area; (d) correcting a foreground background separation result of pixels belonging to the extracted areas according to the results classified in step (c), and outputting a foreground image; And (e) synthesizing a background image prepared in advance in the foreground image.

Here, the step (e) may include correcting the color of the background image according to each color channel value of the foreground image and the background image.

In addition, the step (e) is to analyze the color information of the foreground image, and optionally depending on whether or not to be biased in a particular color channel, the color of the background image in accordance with each color channel value of the foreground image and the background image Or correcting the brightness of the background image.

In order to solve the above other technical problem, there is provided a computer-readable recording medium having recorded thereon a program for executing a method of replacing a background by separating an image according to the present invention into a foreground and a background.

In order to solve the above other technical problem, an apparatus for replacing a background by separating an image according to the present invention into a foreground and a background includes: comparing the input image with a background model, and separating the image into a foreground and a background in pixel units. Unit separator; An uncertain region extraction unit for dividing the image into a plurality of regions and extracting regions of uncertainty whether the image is a foreground region or a background region; An area unit classifying unit classifying the extracted areas into a foreground area and a background area; A correction unit for correcting a foreground background separation result of pixels belonging to the extracted areas according to the result classified by the area unit classifying unit, and outputting a foreground image; And a background image synthesizer configured to synthesize a background image prepared in advance in the foreground image.

According to the present invention described above, by dividing the image into a plurality of areas, and by classifying and correcting the uncertain areas of the divided areas into the foreground and the background, even if there is a change in lighting or similar colors in the foreground and background accurately in real time You can separate foreground and background. In addition, by using the foreground probability model generated by accumulating the resultant image separated into the foreground and the background, the foreground and the background can be separated more accurately.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description and the accompanying drawings, the substantially identical components are represented by the same reference numerals, and thus redundant description will be omitted. In addition, in the following description of the present invention, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted.

1 is a block diagram of an apparatus for separating an image into a foreground and a background according to an embodiment of the present invention. The foreground background separator 10 according to the present exemplary embodiment may include a background model generator 100, a pixel unit separator 110, an uncertain region extractor 120, an area unit classifier 130, and a corrector 140. ) The foreground background separating apparatus 10 receives an image of a frame unit obtained from a camera (not shown).

The background model generation unit 100 generates a background model to be compared with the input image when the pixel unit separation unit 110 to be described later separates the input image into the foreground and the background. In this case, the background model generator 100 learns a model every frame up to a frame before an object or a person corresponding to the foreground appears in the image, and generates and stores the learned model as a background model. Although not shown, the foreground background separation apparatus 10 according to the present embodiment may include a storage means (not shown) for storing the background model.

로 모델링된다. 여기서,

는 화소 p에서의 컬러 기댓값을,

는 컬러값의 표준편차를,

는 밝기 왜곡의 변화(variation of brightness distortion),

는 색도 왜곡의 변화(variation of chromaticity distortion)를 의미한다. 밝기 왜곡의 변화와, 색도 왜곡의 변화에 관하여는 논문 [T. Horprasert, D. Harwood, L.S. Davis "A Statistical Approach for Real-time Robust Background Subtraction and Shadow Detection", IEEE Frame Rate Workshop, pp. 1-19, 1999.]에 개시되어 있으므로, 상세한 설명은 생략하기로 한다.Hereinafter, a background model according to an embodiment of the present invention will be described in detail. According to the background model according to the present embodiment, any pixel p in the input image has four elements,

Is modeled as: here,

Is the color expected value at pixel p ,

Is the standard deviation of the color values,

Is the variation of brightness distortion,

Means variation of chromaticity distortion. Changes in brightness distortion and changes in chromatic distortion are described in the paper [T. Horprasert, D. Harwood, LS Davis, "A Statistical Approach for Real-time Robust Background Subtraction and Shadow Detection", IEEE Frame Rate Workshop, pp. 1-19, 1999., detailed description thereof will be omitted.

을 계산한다. t번째 프레임에서 계산되는

를

라 표현하면, 예를 들어 N번째 프레임에서 사람이 등장했다면, N-1 번째 프레임까지 계산된

가 배경 모델로서 생성된다. According to the present exemplary embodiment, the background model generator 100 may perform every frame until an object or a person appears in an image.

Calculate computed on the t frame

To

For example, if a person appeared in the Nth frame, the N-1th frame is calculated.

Is generated as the background model.

와 표준편차의 벡터

는 다음 수학식과 같이, r, g, b 컬러값 각각에 대한 평균 및 표준편차의 성분을 가진다.vector of color expected values in pixel p for the t-th frame

Vector of standard deviations

Has the components of the mean and standard deviation for each of the r, g and b color values, as shown in the following equation.

및 표준편차

는 다음 수학식에 따라 이전 프레임인 t-1 번째 프레임의 평균

및 표준편차

를 이용하여 구해질 수 있다.According to the present exemplary embodiment, the background model is trained every frame, and the average of the color value i at pixel p in the image of the t-th frame is obtained.

And standard deviation

Is the average of the previous frame, the t-1 th frame, according to the following equation

And standard deviation

Can be obtained using

는 화소 p에서의 컬러 i의 컬러값을 의미한다. Where the subscript i represents any one of r, g, b,

Denotes the color value of the color i in the pixel p .

와 색도 왜곡

는 다음 수학식에 따라 구해질 수 있다.Also, the brightness distortion at the pixel p in the image of the t-th frame

And chromatic distortion

Can be obtained according to the following equation.

와 색도 왜곡의 변화도

는 다음 수학식에 따라 이전 프레임인 t-1번째 프레임의 밝기 왜곡 의 변화

과 색도 왜곡의 변화

를 이용하여 구해질 수 있다. In this case, the variation of the brightness distortion in the pixel p in the image of the t-th frame

And chromaticity distortion

Is the change in brightness distortion of the previous frame, the t-1th frame,

Of color and chromatic distortion

Can be obtained using

In the background model generation unit 100, the background model learned as described above up to the frame before the frame in which the object or the person corresponding to the foreground appears is stored in the storage means (not shown).

When the background model is generated by the background model generator 100, the pixel unit separator 110 compares an image of an input frame with a background model generated by the background model generator 100, and compares the image in pixel units. Separate into foreground and background. Herein, the input frame refers to a frame in which an object or a person corresponding to the foreground appears and each subsequent frame, and the pixel separating unit 110 compares the image of each frame with the background model in pixel units. Separate into foreground and background. Separating the foreground and the background in pixel units, in other words, distinguishes whether each pixel in an image corresponds to the foreground or the background.

The pixel unit separator 110 calculates a similarity degree between the input image and the background model for each pixel belonging to the image to compare the input image with the background model. According to this similarity, each pixel is classified into either a foreground pixel or a background pixel. For example, each pixel is classified into a background pixel if the similarity is high and a foreground pixel if the similarity is low.

The similarity l (p) for any pixel p in the image may be obtained according to the following equation using the brightness distortion distribution and the chromatic distortion distribution.

는 화소 p에 대한 밝기 왜곡 분포를,

는 화소 p에 대한 색도 왜곡 분포를 의미하며,

는 실제적으로 색도 유사도가 밝기 유사도보다 상대적으로 적음을 반영하기 위한 값으로서, 1 이상의 소정 값이다. here,

Is the brightness distortion distribution for pixel p ,

Denotes the chromatic distortion distribution for the pixel p ,

Is a value for reflecting that the chromaticity similarity is actually less than the brightness similarity, and is one or more predetermined values.

와 색도 왜곡 분포

는 다음 수학식에 따라 구해질 수 있다. The brightness distortion distribution

And chromatic distortion distribution

Can be obtained according to the following equation.

과 비교하고, 비교 결과에 따라 전경 화소 또는 배경 화소로 분류한다. 이는 다음 수학식과 같이 표현될 수 있다.In addition, the pixel unit separator 110 may set the similarity to a predetermined threshold value.

And classify the image into a foreground pixel or a background pixel according to the comparison result. This can be expressed as the following equation.

는 임의의 화소 p가 전경 화소 또는 배경 화소로 분리된 결과를 나타내고, 1은 전경 화소로 분류되었음을, 0은 배경 화소로 분류되었음을 의미한다. here,

Denotes a result in which an arbitrary pixel p is divided into a foreground pixel or a background pixel, 1 is classified as a foreground pixel, and 0 is classified as a background pixel.

2 illustrates a result of splitting an input image into a foreground and a background according to the above-described embodiment. In FIG. 2, (a) shows an image according to a background model generated according to the above-described embodiment, (b) shows an input image, and (c) shows a result image in which the foreground and background are separated in units of pixels. Indicates. In (c), the part shown in white represents the foreground pixel, and the part shown in black represents the background pixel.

The uncertainty region extractor 120 divides the input image into a plurality of regions and extracts regions of uncertainty that are a foreground region or a background region from among the divided regions. The area unit classifier 130 separates the regions extracted by the uncertainty region extractor 120, that is, the uncertainty regions into a foreground region and a background region.

Referring to (c) of FIG. 2, it can be seen that the result image obtained by separating the foreground and the background obtained by the pixel unit separator 110 has a portion determined as the foreground as the foreground or the background as the foreground. have. This result is more prominent when the color of the foreground and background changes due to the influence of lighting or natural light, or when the color of the foreground and background is similar. Therefore, the uncertainty region extracting unit 120 and the region unit classifying unit 130 determine whether the foreground or the background is uncertain, and the correction unit 140 which will be described later according to the result is the pixel unit separating unit 110. The resultant image obtained by separating the foreground and background is corrected.

The uncertainty region extractor 120 first divides the input image into a plurality of regions, for example, using a mean-shift algorithm. Mean-Shift algorithm is an algorithm that clusters similar data while moving the center toward the average of the surrounding data existing in the vicinity of specific data.

In addition, the uncertainty region extractor 120 extracts uncertain regions (hereinafter, referred to as uncertainty regions) from among the divided regions, whether the foreground region or the background region, wherein each pixel is divided into the foreground or the background in the pixel unit separator 110. Use separate results. In detail, the uncertainty region extractor 120 may determine whether each of the divided regions is a foreground region or a background region by using a foreground background separation result of each pixel belonging to the region (hereinafter, referred to as uncertainty probability). ) And classify the area as an uncertainty area if the uncertainty probability is high.

Here, the uncertainty probability P _U (R _i ) for any region R _i may be obtained according to the following equation.

는 분할된 영역들의 집 합을, N _Fg ( R _i )는 영역 R _i 에서의 전경 화소의 개수를, N _Bg ( R _i )는 영역 R _i 에서의 배경 화소의 개수를 의미한다. Where n is the number of divided regions,

_{Denotes the set} of divided regions, N _Fg ( R _i ) denotes the number of foreground pixels in the region R _i , and N _Bg ( R _i ) denotes the number of background pixels in the region R _i .

와 비교하고, 비교 결과에 따라 불확실 영역인지 여부를 결정한다. 이는 다음 수학식과 같이 표현될 수 있다.The uncertainty region extractor 120 sets the uncertainty probability P _U (R _i ) to a predetermined threshold value.

And determine whether the area is uncertain according to the comparison result. This can be expressed as the following equation.

보다 크면 불확실 영역으로, 그렇지 않으면 불확실 영역이 아닌 것으로 결정한다.Here, UR indicates that the region is an uncertain region, and NON_UR indicates that the region is not an uncertain region. According to the above equation, the uncertainty probability P _U (R _i ) is a predetermined threshold

If greater, it is determined as the uncertainty area, otherwise it is determined as not the uncertainty area.

The area unit classifier 130 classifies the uncertainty regions extracted by the uncertainty region extractor 120 into a foreground region and a background region. To this end, the area unit classifier 130 first calculates the probability that the corresponding area is the foreground area for each of the uncertain areas. This probability may be calculated according to the following equation using the similarity calculated by the pixel unit separator 110 of pixels belonging to the corresponding area.

Where P _Fg (R _i ) is the foreground area of the uncertainty region R _i , N _Fg (R _i ) is the number of foreground pixels in the uncertainty area R _i , and N _Bg (R _i ) is the uncertainty area R _i Means the number of background pixels.

와 비교하고, 비교 결과에 따라 전경 영역인지 배경 영역인지를 결정한다. 이는 다음 수학식과 같이 표현될 수 있다.The area unit classifier 130 may determine the calculated probability as a predetermined threshold value.

And compares the result to the foreground area and the background area. This can be expressed as the following equation.

는 불확실 영역 R _i 가 전경 영역 또는 배경 영역으로 분류된 결과를 나타내고, 1은 전경 영역으로 분류되었음을, 0은 배경 영역으로 분류되었음을 의미한다. 상기 수학식에 따르면, 전경 영역일 확률 P _Fg (R _i )가 소정 임계값

보다 크면, 전경 영역으로, 그렇지 않으면 배경 영역으로 분류한다. here,

Denotes a result in which the uncertainty region R _i is classified as a foreground region or a background region, where 1 is classified as a foreground region and 0 is classified as a background region. According to the above equation, the probability P _Fg (R _i ) in the foreground region is a predetermined threshold value.

If greater, it is classified as a foreground area, otherwise it is classified as a background area.

In the present embodiment, the area unit classifier 130 calculates the probability of being the foreground area and compares the probability with a predetermined threshold value, but calculates the probability of the background area in the same manner, and calculates the probability by the predetermined threshold. Of course, it can also be implemented in the form of comparing with a value.

The corrector 140 corrects the foreground background separation result of the pixels belonging to the uncertain regions among the pixels in the image according to the result classified by the area unit classifier 130. That is, if the pixel classified as the foreground in the pixel separator 100 is included in the background area classified by the area unit classifier 130, the pixel is classified as the background pixel, and the pixel classified as the background is classified by the pixel divider 110. If is included in the foreground area classified by the area unit classifying unit 130, the correction is made to the foreground pixel. The correction unit 140 outputs the foreground image by removing the separated background.

3 illustrates a process of separating an input image into a foreground and a background according to the above-described embodiment. In FIG. 3, (a) shows an input image and (b) shows an image divided into a plurality of regions using a Mean-Shift algorithm. (c) shows a result image in which the foreground and the background are separated on a pixel basis, wherein a portion displayed in white represents a foreground pixel and a portion displayed in black represents a background pixel. In (d), the uncertain areas are shown in white, and (e) is a result of the uncertain areas being classified into the foreground area and the background area, and the foreground area is red and the background area is blue. (f) shows an image corrected according to the result (e) of the resultant image (c), which is divided into the foreground and the background by area, in the pixel unit.

Referring to (c) and (f) of FIG. 3, the color of the clothes of the person corresponding to the foreground is similar to the background, so that the phenomenon that is incorrectly judged as the background appears in the foreground in (c), whereas the corrected image In (f), it can be seen that such a problem rarely occurs.

4 is a block diagram of an apparatus for separating an image into a foreground and a background according to another embodiment of the present invention. The foreground background separator 20 according to the present exemplary embodiment may include a background model generator 100, a pixel unit separator 110, an uncertain region extractor 120, an area unit classifier 135, and a corrector 140. ), The foreground probability model generator 145 is included. Like the foreground background separation apparatus shown in FIG. 1, an image in a frame unit obtained from a camera (not shown) is also input to the foreground background separation apparatus 10 according to the present embodiment.

In the foreground background separator 20 according to the present exemplary embodiment, operations of the background model generator 100, the pixel unit separator 110, the uncertainty region extractor 120, and the corrector 140 are described with reference to FIG. 1. Since the description is the same, the description will be omitted.

First, the foreground probability model generator 145 will be described. In general, an important feature in a video is that the image changes with time, but there is a feature that the difference between the adjacent frames is not large. Therefore, the pixels classified as the foreground in the previous frame have a high probability of being classified as the foreground in the current frame. The foreground probability model generator 145 generates a foreground probability model indicating the probability that the pixels of the image of the current frame are foreground pixels by reflecting the information of the past frame. To this end, the foreground probability model generator 145 generates a foreground probability model by accumulating the resultant image separated into the foreground and the background in the pixel unit separator 110. This foreground probability model will be referred to as a temporal foreground probability model (TPM) for convenience. The temporal foreground probability model may be expressed as a weighted sum of the foreground probability of the previous frame and the foreground separation result of the current frame, as shown in the following equation.

는 현재 프레임인 t번째 프레임의 영상에서 화소 p가 전경 화소일 확률을,

는, t-1번째 프레임의 영상에서 화소 p가 전경 화소일 확률을,

는 t번째 프레임의 영상에 대한 화소 단위 분리부(110)에서 화소 p의 전경 분리 결과값을 의미하며, 0번째 프레임은 영상에 사람이 등장한 프레임이다.here,

Is the probability that pixel p is a foreground pixel in the image of the t-th frame, which is the current frame,

Is the probability that the pixel p is the foreground pixel in the image of the t-1th frame,

Denotes a result of foreground separation of the pixel p in the pixel unit separator 110 for the image of the t-th frame, and the 0-th frame is a frame in which a person appears in the image.

는 미리 정의되는 소정 값으로서 0에서 1 사이의 값으로 부여되며, 전경 변화의 정도에 따라 그 값을 달리 정의할 수 있다. 예를 들어, 사람의 움직임이 작은 경우에는 바로 이전 프레임의 영상에 가중치를 비교적 높게 주어야 하므로,

를 상대적으로 큰 값으로 정의하고, 사람의 움직임이 큰 경우에는 이전 프레임과 현재 프레임의 차이가 크기 때문에 사람의 움직임이 작은 경우에 비해 이전 프레임의 영상에 가중치를 비교적 낮게 주어야 하므로

를 상대적으로 작은 값으로 정의할 수 있을 것이다. 화상 대화나 화상 회의에서는 사람이 급격하게 움직이는 경우가 적다고 본다면, 본 실시예를 화상 대화나 화상 회의에 적용하고자 할 때

를 상대적으로 큰 값으로 정의할 수 있을 것이다. Also,

Is a predefined value which is given as a value between 0 and 1, and may be defined differently according to the degree of foreground change. For example, if the human motion is small, the weight of the image of the previous frame should be relatively high,

Is defined as a relatively large value, and if the human motion is large, the difference between the previous frame and the current frame is large, so that the weight of the image of the previous frame should be relatively low compared to the case where the human motion is small.

Can be defined as a relatively small value. Given that there are fewer people moving rapidly in a video chat or video conference, when the present embodiment is to be applied to a video chat or video conference.

Can be defined as a relatively large value.

FIG. 5 is a diagram illustrating such a temporal foreground probability model, wherein (a) shows an input image for each frame and (b) shows an image based on a foreground probability obtained according to the above-described temporal foreground probability model. In the temporal foreground probability model, the result of foreground separation is continuously accumulated. Referring to (b), it can be seen that the portion that is continuously separated by the foreground appears to have a high density of white. can do.

Using the temporal foreground probability model described above has the advantage of solving a problem such as the occurrence of a hole in the body of a person in the foreground. Even if a hole is formed in the human body, even if the hole is generated in the human body or the head part, when the background is synthesized into another image, the image is clearly visible to the viewer. However, according to the temporal foreground probability model described above, since the foreground foreground probability is obtained by accumulating past foreground separation results, a problem in which a hole is generated can be solved.

Like the area unit classifier 130 described with reference to FIG. 1, the area unit classifier 135 calculates a probability of being the foreground area with respect to the uncertain areas extracted by the uncertain area extractor 120, Therefore, the uncertainty areas are classified into a foreground area or a background area. However, in the present exemplary embodiment, the temporal foreground probability model generated by the foreground probability model generator 145 is used to calculate the probability of the foreground region with respect to the uncertainty regions. According to the present exemplary embodiment, the probability that the corresponding area is the foreground area for each of the uncertainty areas in the area unit classifying unit 135 may be calculated according to the following equation.

는 상기된 수학식 10에 따른 현재 프레임인 t번째 프레임의 영상에서 화소 p가 전경 화소일 확률을 나타낸다. Here, P _Fg ^t (R _i ) is the probability of the foreground area of the uncertainty region R _i in the t-th frame that is the current frame, and P _Fg ^t-1 (R _i ) is the uncertainty area in the t-1 th frame that is the previous frame. For the probability that it is the foreground region of R _i , N _Fg (R _i ) and N _Bg (R _i ) mean the number of foreground pixels and background pixels of uncertain region R _i in the t-th frame. And

Denotes the probability that the pixel p is the foreground pixel in the image of the t-th frame which is the current frame according to Equation 10 described above.

FIG. 6 is a block diagram of an apparatus for replacing an image by separating an image into a foreground and a background according to an embodiment of the present invention. Background replacement apparatus according to the present embodiment comprises a foreground background separator 30 and the background image synthesizer 40.

The foreground background separator 30 separates the input image into the foreground and the background, and outputs the foreground image from which the background is removed. The foreground background separator 30 is the same as the configuration and operation of the foreground background separator 10 described with reference to FIG. 1 or the foreground background separator 20 described with reference to FIG. 4, and thus description thereof will be omitted. .

The background image synthesizer 40 synthesizes the background image to be replaced in advance with the foreground image output from the foreground background separator 30, and outputs an image in which the background is replaced from the input image.

If a new background image is simply replaced with a separate human image, the two images are created under completely different conditions, so that the human area can be cut with scissors and attached to the new background. That is, since the new background image to be replaced with the foreground image is an image photographed under completely different lighting conditions, when synthesized into one image, the overall brightness and color of each image are completely different and thus may not be viewed as a natural image. To solve this problem, it is necessary to correct brightness and color to make the background image appear to be taken under the same lighting conditions as the foreground image. Therefore, in the present embodiment, the background image synthesizer 40 corrects the color or brightness of the synthesized background image in consideration of the color information of the foreground image.

First, the color correction of the background image will be described. In this embodiment, a color transform method using a diagonal model is used to correct a color difference between the foreground image and the background image. Color correction using the diagonal model has the advantage that color correction is possible even if there is no corresponding pixel in the two images, and the speed is very fast.

The color correction of the background to be replaced may be represented by the product of the diagonal model M and the background to be replaced as in the following equation.

대체될 배경 x M = 보정된 배경,

Background to be replaced x M = corrected background,

Here, α, β, and γ denote ratios of average intensity of R, G, and B color channels of the foreground image and the background image to be replaced, respectively, and are obtained according to the following equation.

Here, mean ( R _F ) and mean ( R _B ) represent the average of the R color channel values of the foreground image and the R color channel values of the background image to be replaced, respectively. The same applies to the G color channel and the B color channel.

However, when the color of the background image is corrected, if the color distribution of the foreground image is too biased in a specific color channel, it is not a good idea to correct the color of the background image in consideration of the color channel value of the foreground image as described above. May result. For example, the background area may be corrected to be biased to a particular color channel. In this case, it is better to correct only the brightness than to correct the color of the background area. Therefore, in the present embodiment, the color information of the foreground image is analyzed, and optionally, the color of the background image is corrected or the brightness of the background image is corrected depending on whether the color is excessively biased in a specific color channel.

7 is a flowchart illustrating a method of selectively correcting color or brightness of a background image according to an embodiment of the present invention.

First, in step 610, the average of the R, G, and B color channel values of the foreground image and the background image to be replaced are respectively calculated.

In operation 620, it is determined whether the color distribution of the foreground image is excessively biased in a specific color channel. Here, whether or not the specific color channel is excessively biased may be determined depending on whether the average of the specific color channel value among the averages of the color channel values of the foreground image is greater than a predetermined threshold value. Alternatively, the average of each color channel value may be compared with each other to determine whether the difference is greater than a predetermined threshold.

If it is determined in step 620 that it is excessively biased, the process proceeds to step 630 to correct the brightness of the background image. The brightness correction of the background image may be represented by replacing the diagonal model M of Equation 12 with a diagonal model according to the following equation, that is, a diagonal model in which all of the components α, β, and γ have the same value.

As described above, the brightness correction or the color correction is optionally performed to solve the unnatural problem that occurs when the background image is dark and the foreground is bright, or the unnatural problem that occurs when the color ratio between the background and the foreground is different. This allows for more natural background synthesis.

On the other hand, even if the color and brightness are well corrected, if the boundary between the background and the person is not smoothly processed, the human area can be cut with scissors to give a feeling of being attached to another background. In order to solve this problem, the background image synthesizer 40 in this embodiment minimizes the separation between the two images by blurring the boundary area between the foreground image and the background image. For the blurring effect, a Gaussian smoothing filter with a 5 ㅧ 5 mask is applied, and the type of Gaussian smoothing filter can be defined according to the following equation.

For real-time application of the Gaussian smoothing filter, a pre-calculated filter may be used without calculating each time, and the shape of such a filter is illustrated in FIG. 8. The application range of the filter limits the application range by obtaining an edge from an image divided into a foreground and a background, and performing an expansion operation to generate a blur mask.

FIG. 9 is a diagram illustrating a state in which a blurring mask and a Gaussian smoothing filter are applied. In FIG. 9, (a) shows an image according to a background replacement result without using a filter, (b) shows a blurring mask for applying the filter, and (c) shows a Gaussian smoothing filter using the blurring mask. The image according to the applied result is shown.

10 illustrates a result of replacing the background of the input image with a new background according to the embodiment of the present invention described above. In FIG. 9, (a-1), (a-2), and (a-3) denote input images, and (b-1), (b-2) and (b-3) denote new backgrounds of the input image. Replace with

Meanwhile, the above-described embodiments of the present invention can be written as a program that can be executed in a computer, and can be implemented in a general-purpose digital computer that operates the program using a computer-readable recording medium. The computer-readable recording medium may be a magnetic storage medium (for example, a ROM, a floppy disk, a hard disk, etc.), an optical reading medium (for example, a CD-ROM, DVD, etc.) and a carrier wave (for example, the Internet). Storage medium).

So far I looked at the center of the preferred embodiment for the present invention. Those skilled in the art will appreciate that the present invention can be implemented in a modified form without departing from the essential features of the present invention. Therefore, the disclosed embodiments should be considered in descriptive sense only and not for purposes of limitation. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the scope will be construed as being included in the present invention.

1 is a block diagram of an apparatus for separating an image into a foreground and a background according to an embodiment of the present invention.

2 illustrates a result of splitting an input image into a foreground and a background according to an embodiment of the present invention.

3 illustrates a process of separating an input image into a foreground and a background according to an embodiment of the present invention.

4 is a block diagram of an apparatus for separating an image into a foreground and a background according to another embodiment of the present invention.

5 is a diagram illustrating a temporal foreground probability model according to the present invention.

FIG. 6 is a block diagram of an apparatus for replacing an image by separating an image into a foreground and a background according to an embodiment of the present invention.

7 is a flowchart illustrating a method of selectively correcting color or brightness of a background image according to an embodiment of the present invention.

8 shows the form of a filter calculated in advance for real-time application of the Gaussian smoothing filter.

FIG. 9 is a diagram illustrating a state in which a blurring mask and a Gaussian smoothing filter are applied.

10 illustrates a result of replacing a background of an input image with a new background according to an embodiment of the present invention.

Claims (26)

In the method for separating the image into the foreground and background, (a) comparing the image of the input current frame with the background model and separating the image into a foreground and a background in pixel units; (b) dividing the image into a plurality of regions, and extracting regions of uncertainty whether the foreground region or the background region is one of the divided regions; (c) classifying the extracted areas into a foreground area and a background area; And and (d) correcting the foreground background separation result of the pixels belonging to the extracted regions according to the results classified in step (c). The method of claim 1, Generating the background model, wherein the background model follows a learned model every frame up to a frame before an object or person corresponding to the foreground appears. The method of claim 1, In the step (a), the foreground background may be divided into a foreground and a background by calculating a similarity between the image and the background image in pixel units and comparing the calculated similarity with a predetermined first threshold. Separation method. The method of claim 1, In the step (b), for each of the divided areas, the probability of uncertainty as to whether it is a foreground area or a background area is calculated by using the result separated into the foreground and the background by the pixel unit in the step (a), and the calculation is performed. Extracting the areas to be uncertain by comparing the estimated probability with a predetermined second threshold. The method of claim 1, wherein step (c) comprises: (c1) for each of the extracted regions, calculating a probability of being a foreground region; And (c2) classifying the extracted areas into a foreground area and a background area by comparing the calculated probability with a predetermined third threshold value. The method of claim 5, In the step (a), the similarity between the image and the background image is calculated in pixel units, and the image is separated into the foreground and the background by comparing the calculated similarity with a predetermined first threshold value. In the step (c1), the foreground background separation method may be calculated using the calculated similarity. The method of claim 6, The probability of the foreground area is calculated according to the following equation.

Where P _Fg (R _i ) is the foreground area of the uncertainty region R _i , N _Fg (R _i ) is the number of foreground pixels in the uncertainty area R _i , and N _Bg (R _i ) is the uncertainty area R _i Means the number of background pixels. The method of claim 5, Generating a foreground probability model by accumulating a resultant image separated into a foreground and a background in pixel units from a predetermined past frame to the current frame in step (a); In the step (c1), the foreground background separation method may be calculated using the generated foreground probability model. The method of claim 8, The foreground probability model is a foreground background separation method, characterized in that represented by the following equation.

here,

Is the probability that pixel p is a foreground pixel in an image of frame t,

Is the probability that pixel p is a foreground pixel in the image of frame t-1,

Represents the foreground separation result of the pixel p in the image of the frame t,

Is a predetermined value. The method of claim 9, The probability of the foreground area is calculated according to the following equation.

Here, P _Fg ^t (R _i ) is the probability of the foreground area of the uncertainty region R _i in the t-th frame that is the current frame, and P _Fg ^t-1 (R _i ) is the uncertainty area in the t-1 th frame that is the previous frame. For the probability that it is the foreground region of R _i , N _Fg (R _i ) and N _Bg (R _i ) represent the number of foreground pixels and background pixels of uncertain region R _i in the t-th frame. A computer-readable recording medium having recorded thereon a program for executing the method of separating an image into a foreground and a background, according to any one of claims 1 to 10. In the device for separating the image into the foreground and background, A pixel unit separator which separates the image into a foreground and a background by pixel by comparing an image of an input current frame with a background model; An uncertain region extraction unit for dividing the image into a plurality of regions and extracting regions of uncertainty whether the image is a foreground region or a background region; An area unit classifying unit classifying the extracted areas into a foreground area and a background area; And And a correcting unit configured to correct a foreground background separation result of pixels belonging to the extracted areas according to the result classified by the area unit classifying unit. The method of claim 12, And a background model generator for generating the background model, wherein the background model follows a learned model every frame up to a frame before an object or a person corresponding to the foreground appears. The method of claim 12, The pixel unit separator may separate the image into a foreground and a background by calculating a similarity degree between the image and the background image in pixel units and comparing the calculated similarity with a predetermined first threshold value. Separation method. The method of claim 12, The uncertainty region extractor calculates a probability of uncertainty whether the image is a foreground region or a background region by using a result obtained by dividing the foreground and the background by the pixel unit in each of the divided regions. And extract areas of uncertainty by comparing a probability with a predetermined second threshold. The method of claim 12, The area unit classifying unit classifies the extracted areas into a foreground area and a background area by calculating a probability of being a foreground area and comparing the calculated probability with a predetermined third threshold value for each of the extracted areas. Featuring foreground background separation devices. The method of claim 16, The pixel unit separator may divide the image into the foreground and the background by calculating a similarity between the image and the background image in pixel units and comparing the calculated similarity with a predetermined first threshold value. And the area unit classifying unit calculates a probability of the foreground area using the calculated similarity. The method of claim 17, The foreground background separation apparatus, wherein the probability of the foreground region is calculated according to the following equation.

Where P _Fg (R _i ) is the foreground area of the uncertainty region R _i , N _Fg (R _i ) is the number of foreground pixels in the uncertainty area R _i , and N _Bg (R _i ) is the uncertainty area R _i Means the number of background pixels. The method of claim 16, And a probability model generator configured to accumulate a resultant image separated into a foreground and a background by a pixel unit from a predetermined past frame to the current frame to generate a foreground probability model. And the area unit classifier calculates a probability of the foreground area by using the generated foreground probability model. The method of claim 19, The foreground probability model is the foreground background separation apparatus, characterized in that represented by the following equation.

here,

Is the probability that pixel p is a foreground pixel in an image of frame t,

Is the probability that pixel p is a foreground pixel in the image of frame t-1,

Represents the foreground separation result of the pixel p in the image of the frame t,

Is a predetermined value. The method of claim 20, The foreground background separation apparatus, wherein the probability of the foreground region is calculated according to the following equation.

Here, P _Fg ^t (R _i ) is the probability of the foreground area of the uncertainty region R _i in the t-th frame that is the current frame, and P _Fg ^t-1 (R _i ) is the uncertainty area in the t-1 th frame that is the previous frame. For the probability that it is the foreground region of R _i , N _Fg (R _i ) and N _Bg (R _i ) represent the number of foreground pixels and background pixels of uncertain region R _i in the t-th frame. In the method to replace the background by separating the image into the foreground and background, (a) comparing the input image with a background model and separating the image into a foreground and a background in pixel units; (b) dividing the image into a plurality of regions, and extracting regions of uncertainty whether the foreground region or the background region is one of the divided regions; (c) classifying the extracted areas into a foreground area and a background area; (d) correcting a foreground background separation result of pixels belonging to the extracted areas according to the results classified in step (c), and outputting a foreground image; And and (e) synthesizing the background image prepared in advance on the foreground image. The method of claim 22, The step (e), the background replacement apparatus comprising the step of correcting the color of the background image according to the value of each color channel of the foreground image and the background image. The method of claim 22, In the step (e), the color information of the foreground image is analyzed, and the color of the background image is selectively corrected according to each color channel value of the foreground image and the background image, depending on whether or not a particular color channel is biased. Or correcting the brightness of the background image. A computer-readable recording medium according to any one of claims 22 to 24, wherein a program for recording a program for executing a method of replacing a background by separating an image into a foreground and a background is recorded. In the device to replace the background by separating the image into the foreground and background, A pixel unit separator for comparing the input image with the background model and separating the image into a foreground and a background in pixel units; An uncertain region extraction unit for dividing the image into a plurality of regions and extracting regions of uncertainty whether the image is a foreground region or a background region; An area unit classifying unit classifying the extracted areas into a foreground area and a background area; A correction unit for correcting a foreground background separation result of pixels belonging to the extracted areas according to the result classified by the area unit classifying unit, and outputting a foreground image; And And a background image synthesizer configured to synthesize a background image prepared in advance on the foreground image.

Images