KR101137110B1 - Method and apparatus for surveying objects in moving picture images - Google Patents

Abstract

In the object monitoring method according to the present invention, comparing the current background model and the current image, classifying the blocks with relatively small changes to the background and classifying the video class of blocks with relatively large changes as either foreground or ghost. Grouping the blocks whose image class is the foreground and outputting them as the current foreground blobs and outputting the blocks whose image class is ghost as the ghost blobs, searching for the current foreground blobs corresponding to the tracks accumulating information of the previous foreground blobs and the tracks. The method may include generating a next background model by updating a block position corresponding to the accumulated ghost blob in the current background model when the number of reflected and ghost blobs is accumulated more than a predetermined number.

Description

METHOD AND APPARATUS FOR SURVEYING OBJECTS IN MOVING PICTURE IMAGES}

TECHNICAL FIELD The present invention relates to video surveillance, and more particularly, to a technique for tracking a moving object in a video.

In the video security method using equipment such as CCTV or DVR, it is only a level of obtaining captured images, so the monitor should visually monitor the screen and determine the unusual situation in the image. In this type of monitoring, the longer the monitoring time, the lower the monitoring efficiency due to the monitor's physical fatigue and reduced concentration. Thus, there is a need for surveillance technology of objects in the image. Initially, a technique using a simple motion region detection function was developed, but recently, a motion region extraction technique using a background model and an object tracking an object image defined in units of objects in the extracted region in order to secure the accuracy and efficiency of computation. Development of a technology that delivers information analyzed by combining the tracking technology with the user is actively progressing.

Accordingly, various techniques have been proposed for extracting a background model from an image and extracting an object image showing a movement relative to the background model.

Extracting the background model extracts the background model based on the images taken for a certain period of time, compares the background model with the current image, extracts the foreground image, and reflects the change in the background in reality. It consists of a process of updating.

In this case, after an object that has existed as a background on the screen for a long time moves, the background update deadlock problem, which is a phenomenon that is recognized as a moving area (foreground) even after the place where the object is actually the background, is moved. Due to the many visible areas, the user may not be able to initialize the background model, or the object may move slowly or similar to the background.

Being able to accurately extract and update the background model results in significant differences in extracting and tracking object images from it. Accordingly, there is a need for a video surveillance technology capable of solving a background update deadlock, a background model initialization when there are many moving regions, and a model blur.

An object of the present invention is to provide an image surveillance apparatus and method capable of accurately extracting a background model from an image and effectively extracting and tracking an object image therefrom.

Object monitoring method in the image according to an aspect of the present invention,

Comparing the current background model with the current image, classifying blocks with relatively small changes as background and classifying the image class of blocks with relatively large changes as either foreground or ghost;

Grouping blocks in which the image class is the foreground and outputting the current foreground blob, and outputting the bundles in which the image class is ghost as the ghost blob;

Finding current foreground blobs corresponding to tracks accumulating information of previous foreground blobs and reflecting them in the tracks; And

When the number of ghost blobs is accumulated a predetermined number or more, the method may include generating a next background model by updating a block position corresponding to the accumulated ghost blobs in the current background model.

According to one embodiment, the method for monitoring an object in the image,

The method may further include initializing a background model by combining temporal-spatially stable regions extracted from successive images.

According to one embodiment, the step of initializing the background model by combining the time-spatially stable regions extracted from the successive images,

Comparing the temporary background selected as one of the consecutive images with other images for each block, and counting the number of times the blocks having little change are classified as the background for each block; And

And outputting a block having a number of times classified as a background more than a predetermined value as a background sample.

According to one embodiment, the step of initializing the background model by combining the time-spatially stable regions extracted from the successive images,

Comparing the temporary background and the current image selected as one of the continuous images, the temporary background and the past image, and the current image and the past image, respectively; And

For each block, if the number of times that it is determined that there is a change between the temporary background and the current image and between the temporary background and the past image but there is no change between the current image and the past image is greater than or equal to a predetermined value, The method may further include outputting a corresponding block of the current image as a background sample.

According to one embodiment, the method for monitoring an object in the image,

The method may further include deriving one final background sample representing a plurality of background samples at each block position and combining the final background samples derived for each block to generate the background model.

According to one embodiment, the final background sample,

The background sample may correspond to a histogram median of the plurality of background samples.

According to an embodiment, comparing the current background model with a current image, classifying blocks with relatively small changes as a background and classifying blocks with relatively large changes as a foreground or ghost may include:

Classifying the block as a background if the difference in the unit of a block between the current background model and the current image is less than or equal to a predetermined value; otherwise classifying the block as a motion region;

Detecting contours from blocks classified as motion regions in the current image, for the current background model and the current image;

Among the blocks whose difference between the contour of the current background model and the contour of the current image is equal to or greater than a predetermined value, the contour class of the block from which the contour of the current image is detected is classified as the foreground, and the contour of the current image is not detected, but the current Classifying the contour class of the block in which the contour of the background model is detected as a ghost;

Predicting a probability distribution that the block is the foreground from blocks whose contour class is classified as foreground, and predicting a probability distribution that the block is ghost from blocks classified as ghost; And

If the foreground probability of the block is greater than the ghost probability, the image class of the block may be classified as the foreground, otherwise the classification may be included as a ghost.

According to an embodiment, the step of tying blocks with the foreground image class as a foreground foreground blob and outputting the bundle with ghost class as a ghost blob,

Creating a blob by concatenating blocks using concatenation element labeling;

Outputting the corresponding blob as the current foreground blob if the class of the most blocks in each blob is the foreground; And

If the class having the most blocks in each blob is ghost, it may include outputting the ghost blob.

According to an embodiment, the step of finding current foreground blobs corresponding to the tracks accumulating the information of the previous foreground blobs and reflecting the current foreground blobs to the tracks may be performed.

Associating the previous track with a current foreground blob whose similarity between the foreground blob information included in the previous track and the current foreground blob is greater than a predetermined value; And

And updating foreground blob information of the previous track according to the movement of the corresponding current foreground blob.

According to an embodiment, the step of mapping the previous track with the current foreground blob having a similarity between the foreground blob information included in the previous track and the current foreground blob greater than a predetermined value may include:

Defining a candidate region in a current image based on a past position of a foreground blob included in the previous track; And

If the similarity calculated first with respect to the current foreground blob and the previous track included in the candidate area is greater than a predetermined value, the method may include mapping the current foreground blob to the previous track.

According to one embodiment, the step of finding the current foreground blobs corresponding to the tracks accumulating the information of the previous foreground blobs and reflecting on the tracks,

A current foreground blob corresponding to an area occupied by a defined track set by grouping at least one track whose distance between tracks is equal to or less than a predetermined value and a current foreground blob having a similarity calculated between tracks in the track set greater than a predetermined value; Updating the previous track by matching the previous track;

Setting a track as an inactive track whose calculated similarity is less than a predetermined value and does not correspond to any current foreground blobs; And

And generating a new track for the current foreground blob that does not correspond to an area occupied by any track set.

According to one embodiment, the similarity calculated between the tracks in the track set and the current foreground blobs corresponding to the area occupied by the track set defined by grouping at least one track whose distance between the tracks is less than or equal to a predetermined value is predetermined. Updating the previous track by mapping the previous track with a current foreground blob that is greater than a value,

A similarity function set including a feature point for distinguishing each track in the track set, a plurality of similarity functions, is defined with the track set, and the current foreground blobs and the previous track based on the similarity function of one of the similarity functions. Computing the similarity between the.

According to an embodiment, the similarity functions included in the similarity function set may include an absolute error sum (SAD), an error square sum (SSD), a battacharraya function, and a correlation function. .

According to an embodiment, the similarity function having the best performance may be selected based on the performance history of the similarity determination accumulated with respect to the similarity functions.

According to an embodiment, a similarity function having a relatively simple operation may be selected among at least two higher similarity functions having excellent performance based on the performance history of the similarity determination accumulated with respect to the similarity functions.

According to an embodiment of the present disclosure, if the ghost blobs are accumulated more than a predetermined number, generating a next background model by updating a block position corresponding to the accumulated ghost blobs in the current background model.

Accumulating ghost blobs block by block;

Deriving a change occurrence background region that can represent the ghost blobs accumulated in a predetermined number or more; And

The method may include generating a next background model by updating a block position corresponding to the accumulated ghost blob in the current background model as the derived change generation background area.

According to another aspect of the present invention, a video surveillance apparatus,

Compares the current background model with the current image, classifies blocks with less change as background, classifies the image class of blocks with larger changes as either foreground or ghost, and binds blocks with image class as foreground to the current foreground blob A motion class determiner for outputting a block to which the image class is ghosted and outputting the ghost block;

An object tracker which finds current foreground blobs corresponding to tracks accumulating information of previous foreground blobs and reflects the current foreground blobs to the tracks; And

If the number of ghost blobs is accumulated more than a predetermined number, the background model generator may generate a next background model by updating a block position corresponding to the accumulated ghost blobs in the current background model.

According to the apparatus and method of the present invention, it is possible to accurately extract a background model from an image and to effectively extract and track a plurality of object images therefrom. Based on this, an intelligent video security function such as detection of invasion of the surveillance area, object counting, detection of the appearance of an object and deviation from the screen can be implemented.

In addition, according to the video surveillance apparatus and method according to the present invention, since the memory usage and power consumption is small based on the efficient operation, it can be easily applied to various systems.

1 is a conceptual diagram illustrating an image surveillance process including background model initialization, object tracking, and background update through motion class classification according to an embodiment of the present invention.
2 is a diagram illustrating a life cycle of track information used in the video surveillance process according to an embodiment of the present invention.
3 is a flowchart illustrating detailed steps of initializing a background model during an image monitoring process according to an embodiment of the present invention.
4 are image capture screens illustrating a background model initialized according to a background model initialization during an image monitoring process according to an embodiment of the present invention.
5 is a flowchart illustrating the detailed steps of motion region detection and motion class classification during the video surveillance process according to an embodiment of the present invention.
6 is a diagram illustrating a motion region extraction in units of blocks in a video surveillance process according to an embodiment of the present invention.
7 is a conceptual diagram illustrating object tracking using a similarity between a foreground blob and a track in a video surveillance process according to an embodiment of the present invention.
8 is a block diagram illustrating a video surveillance apparatus according to an embodiment of the present invention.

For the embodiments of the invention disclosed herein, specific structural and functional descriptions are set forth for the purpose of describing an embodiment of the invention only, and it is to be understood that the embodiments of the invention may be practiced in various forms, The present invention should not be construed as limited to the embodiments described in Figs.

Hereinafter, with reference to the accompanying drawings, it will be described in detail a preferred embodiment of the present invention. The same reference numerals are used for the same constituent elements in the drawings and redundant explanations for the same constituent elements are omitted.

1 is a conceptual diagram illustrating an image surveillance process including background model initialization, object tracking, and background update through motion class classification according to an embodiment of the present invention.

Referring to FIG. 1, in step S11, a background model (BG) may be initialized by combining time-spatially stable regions extracted from consecutive images (S111), or a change may occur in the background model to generate a ghost ( ghost, GH) is accumulated more than a certain number of parts, by updating the corresponding block position of the current background model with the cumulative change parts to generate the next background model (S112), to exclude the moving object and purely extract only the background Creating a background model.

Here, the background means an entire image without a motion region, whereas the foreground (FG) is a partial image that is a target of temporal and spatial tracking as a motion region. In order to extract the foreground, which is the target of such tracking, from the currently captured image, the synthesized image that is a reference is called a background model.

In the conventional video surveillance process, when there are many movements of objects on the screen, the moving objects are blurred and the background model is not properly generated. However, step S111 of the present invention may initialize the background model by excluding moving objects by extracting time-spatially stable regions and combining them.

On the other hand, in the conventional video surveillance process, when an object in an area that has been recognized as a background moves for a long time and suddenly moves and disappears, in reality, the area of the position should be understood as a new background. Because it is a moving area, it is still recognized as the foreground. Thus, the video surveillance process continues to attempt tracking even if there is no significant object at that location, which is called ghost.

Step S112, together with step S12, continuously determines whether the area in motion is a foreground or ghost, collects the ghost areas into a background model, and updates the new background model in a conventional video surveillance process. The ghost phenomenon that could not be dealt with can be solved.

Conversely, if an object stays stationary for a long time after entering the screen, the conventional video surveillance process can recognize the object as a background. However, the video surveillance process of the present invention does not use such an object to update the background model and therefore does not mistakenly recognize it as a background.

Furthermore, in the conventional video surveillance process, a background blur phenomenon may occur in which information of an object moving slowly across the background is blurred in the background model at the time of updating the background model.

In contrast, step S112 of the present invention accumulates ghost regions and updates the background model at a time, thereby eliminating a phenomenon in which moving objects are mixed with the background model. In detail, the ghost regions are accumulated for each block, and a change generation background region (MVBG) that represents the ghost regions accumulated over a predetermined number is derived. The next background model may be generated by updating a corresponding block position in the current background model as the derived change generation background area.

In this case, even if a plurality of ghost regions are identified, the background model may not be updated until a certain number is accumulated in the corresponding block. However, the ghost-inducing object itself can be identified as the foreground, so there is no problem in tracking the object even if the update of the ghost area is somewhat delayed. In addition, since the screen displayed to the user is the screen currently being shot and the update delay of the background model is short for the user to recognize, the user does not notice the update delay of the background model.

Step S12 compares the current background model with the current image, classifies blocks with relatively small changes or differences as the background, classifies image classes of blocks with relatively large changes or differences as either foreground or ghosts, and then classifies. The combined blocks are combined and output as a motion blob, a foreground blob or a ghost blob.

This step S12 is a step of determining the motion class of each block. First, the background model is compared with the current image to determine whether a change has occurred. There are two cases where changes occur in the current image compared to the background model. First, when the object actually moves, a change occurs in the corresponding block. Second, a change occurs when an object stopped in the part that was recognized as the background moves.

The change below a certain level may be a temporary phenomenon due to image noise during photographing or post-processing. Such sub-level changes classify the block into the background so that it does not affect the background model.

If more than a certain amount of change occurs, such blocks are classified as one of the motion classes of the foreground or the ghost because they are one of the ghosts that have changed in the foreground or the background, which are actual moving objects.

The blocks in which the motion class is classified are grouped according to how spatially the blocks labeled with the motion class have relation with neighboring blocks. For example, the labeled blocks are each blobed in accordance with the Connected Component Labeling technique. This blob is often a mix of blocks labeled with different movement classes. In this case, the motion class of the blob may be determined by the class label having the largest proportion among the class labels of the blocks in the blob.

As a result, the motion block corresponding to the object to be tracked, that is, the foreground blob, may be extracted, and the motion block corresponding to the ghost to be updated of the background model, that is, the ghost blob, may be extracted.

Step S13 is a step of creating, maintaining, updating and managing a track for tracking the extracted foreground blobs for tracking the object. In the field of video surveillance, a track is a unit for defining a tracking state of an object, and in the present invention, a set of similarity functions for comparing a track of an object, features representing an object, and a track of another object in particular. It may include.

Referring to FIG. 2 for a brief description of tracks, FIG. 2 is a diagram illustrating a life cycle of track information used in the video surveillance process according to an embodiment of the present invention.

When there is no foreground motion chunk corresponding to an object, i.e. a track corresponding to the foreground blob, a new track is created. In this case, the newly created track is a temporary track and is not yet secured, and thus is a target for a certain period of time. If the stability is verified by maintaining a one-to-one correspondence with the foreground motion chunk for a certain observation period, the temporary track changes state to the active track.

The tracking of an object is said to be successful when there is a blob of motion, ie a blob, that maintains a one-to-one correspondence with past tracks, and the track can remain active while this successful object tracking state is maintained.

If the track is in the active track state and no object with a one-to-one correspondence is found, the state is changed to an inactive track. If such a phenomenon occurs temporarily and later restores the one-to-one correspondence again, the inactive track is changed back to the active track. However, if a one-to-one correspondence is not found for a period of time, the object is said to disappear from the screen and the track is no longer used.

Returning to FIG. 1, in step S13, the similarity between the tracks tracked so far and the foreground blobs extracted from the current image is calculated to find the foreground blob corresponding to each track.

If there is a foreground blob that does not correspond to any track, create a new track for that foreground blob.

If the foreground blob corresponding to the track is found through the similarity function, the movement trajectory or feature point change of the foreground blob is reflected in the track. In FIG. 1, the similarity calculation is performed once for the track 0, the track 1, and the track 2 with the blob 0, the blob 1, and the blob 2, respectively. As a result, track 0 has the highest similarity to blob 0 (solid blue line) and low similarity to blob 1 and blob 2 (blue dashed line). Similarly, track 1 has the highest similarity to blob 1 (solid red line) and low similarity to blob 0 and blob 2 (red dashed line).

Also, if there is no foreground blob corresponding to a track, that track is converted to an inactive track.

Finally, if an inactive track continues to be inactive, it can be deleted that the corresponding object no longer exists on the screen and is unlikely to reappear. In some embodiments, deletion of tracks may be suppressed for reuse of inactive tracks upon re-appearance of objects.

After the track is updated for the current image, a series of processes are repeated from the updating of the background model (S112) to the tracking and maintenance of the track (S13) for the next image.

3 is a flowchart illustrating detailed steps of initializing a background model during an image monitoring process according to an embodiment of the present invention.

Background image constituting the background model is more stable in time and space than the foreground image. Thus, if a time-spatially stable region is extracted from some successive images, such region is likely to be classified as a background. In FIG. 3, a background region sample is collected from stable regions extracted using an initial image, and an initial background model is generated from the collected background samples.

Time-spatial stable domains can be classified into two types. First, the blocks whose motion class is classified as the background are likely to be the background. Next, the blocks whose motion classes are classified as ghosts are also likely to be actually background. However, at this stage, the motion class is classified based on the temporary background model instead of using the verified background model. Therefore, whether the motion region is ghost can be identified through the stability checking method described below.

In detail, referring to FIG. 3, in step S31, one of the consecutive images, for example, the first image, is defined as a temporary background model.

In operation S32, the blocks of the current image, which are relatively less changed in comparison to the temporary background model, are classified into the background of the motion region class. Whether the blocks of the current image are a background class may be classified through the method described with reference to FIG. 5, or simply sum of absolute difference (SAD) or error per block for the current image and the temporary background model. If the sum of squared difference (SSD) value is smaller than a predetermined allowable value or the correlation function value is larger than the allowable value, it may be immediately classified as a background.

In step S33, a block whose number of times classified as a background at each block position with respect to each image is greater than or equal to a predetermined value is very likely to be a background, and thus is determined as a background sample. This can be formulated as follows.

Here, TBI is a temporary background model and I is image information. TC is a coefficient variable, α _BG is a value in which a count increases when classified as a background, IC is an image class, and BG is an image class corresponding to a background. If TC is greater than or equal to a predetermined value, such block is designated as a background sample.

In step S34, three images, that is, a current image, a past image, and a temporary background model image, as shown in Equation 2, in order to extract a relatively stable region from blocks not classified as a background in preparation for the temporary background model. Perform a stability test using the differences between.

DM _past (x, y, t) is the difference between the temporary background model image TBI (x, y, t) and any past image I (tn), i.e. SSD, SAD, etc. 1, otherwise 0. DM _cur (x, y, t) is 1 when the difference between the temporary background model image TBI (x, y, t) and the current image I (t) is greater than a predetermined value and is a movement area, and 0 otherwise. TD (x, y, t) is the difference between the current image and the past image, and is 1 if the difference is larger than a predetermined value. At this time, if there is no difference between the past image and the current image among the block positions (DM _past = 1 and DM _cur = 1) where the difference between the past image and the current image is compared to the temporary background model, the block position has no difference (TD = 0). If present, that means that the block position is different from the original image, the temporary background model, but there is no change thereafter, and it is likely to be the background. Equation 3 is an expression representing increasing the TC coefficient with respect to such an area. When the TC coefficient exceeds a predetermined number, the corresponding block of the current image is determined as a background sample.

Once the background samples have been collected sufficiently block by block, in step S35, one final background sample representing the background samples and removing foreground images irrelevant to the actual background which may have been included by using the statistical properties of the background samples may be included. To derive Specifically, by calculating one of the average, median, mode, and median estimates of the background samples on a block-by-block basis, the background images corresponding to the actual background are excluded, and the background images corresponding to the actual background are excluded. One sample representative of these can be derived. In step S36, the first background model is generated by combining the final background samples derived for each block.

The median estimation and the generation of the background model using the histogram may be performed as in Equation 4 below.

In Equation 4, when the count TC of the background sample is greater than or equal to a predetermined value for a certain image I (x, y, t), the histogram size BGS of the background sample is increased by one. A median is obtained with respect to such histogram sizes, and the background samples corresponding to the median are combined to generate a background model BG (x, y, t).

4 are image capture screens illustrating a background model initialized according to a background model initialization during an image monitoring process according to an embodiment of the present invention.

Referring to FIG. 4, the initial input images on the left side include vehicles moving on the road, and the background model initialized by the prior art on the upper right side is obtained with the distorted vehicle shape. On the other hand, the background model initialized according to an embodiment of the present invention can be obtained with all of the shape of the moving vehicle, such as the lower right end, classified and removed as a ghost and including the road correctly.

Based on this correctly initialized background model, moving objects can be identified as foreground without exception.

5 is a flowchart illustrating the detailed steps of motion region detection and motion class classification during the video surveillance process according to an embodiment of the present invention.

Referring to FIG. 5, the purpose of this step is to distinguish blocks with or without motion, and in particular to identify whether the blocks with motion are foreground or ghost.

Step S51 calculates the difference between the current background model and the current background model that has been initialized or updated. Specifically, although the difference between the two images may be found in all the pixels of the current image according to the embodiment, the difference between the images is preferably obtained for each block.

In the present invention, the difference between the background model and the current image uses a similarity checking technique such as a sum of absolute difference (SAD), a sum of squared difference (SSD), a correlation function, and the like in a block unit.

In step S52, if a difference between images in units of blocks, for example, the sum of absolute absolute values, is equal to or less than a predetermined value, the corresponding block is classified as a background, and if not, the corresponding block is classified as a motion region.

Referring to FIG. 6 for a while, FIG. 6 is a diagram illustrating a motion region extraction in units of blocks in a video surveillance process according to an exemplary embodiment of the present invention.

The background model was created to not include objects, and the current input image shows a person walking down the hall wearing a bag. As illustrated in the lower left, when the difference image is obtained in pixel units, pixels having a significant difference between the background model and the input image appear. Objects and shadows show a sharp difference compared to the background model, so the outline is raised, but there is also a phenomenon called salt and pepper because of the spatial inconsistency or noise between the input image and the background model.

However, according to the present invention, when extracting a motion region in units of blocks, the salt and pepper phenomenon can be suppressed and computed quickly, while the error is not so large as compared to the unit of pixels.

In general, a difference image by a moving object constantly forms a lump, that is, a blob. On the other hand, the difference image due to noise is scattered irregularly. When the differences of the images are obtained by the unit of block, the difference of the image by the moving object in the unit of block is relatively large, while the difference of the image due to noise is the unit of the unit of block As low. In both cases, the individual values of the sum of the image differences are clearly distinguished by a specific boundary. Thus, a block corresponding to an object with motion can be extracted by comparing the block-by-block sum of the image difference to that particular boundary value.

5, in the present invention, the difference between the background model BGI and the input image I is contrasted and calculated for each block so that the class of the block if the difference between the background model and the input image in the block is greater than a predetermined value. (IC) is classified into the motion region (MV). However, if the difference between the background model and the input image is smaller than a predetermined value, the entire block is classified as a motionless background area BG. Equation 5 shows this.

여기에서,

는 배경모델(BGI)을 기준으로 계산한 차이(DM)를 블록단위의 영상으로 변환한 결과를 의미한다. 즉,

는 배경모델과 차이를 비교하는 대상이 시간상으로 현재의 입력 영상이며 비교하는 위치가 블록단위로 (x,y)임을 의미하게 된다. 그리고

는 블록 영상의 위치 x,y에 해당되는 픽셀 영상의 위치 좌표의 집합을 의미한다. 즉, 블록은 여러 픽셀의 모임인데, 블록의 (x,y)에 해당하는 픽셀들의 좌표를 모은 것이

이다. 파라미터

는 움직임을 구분하기 위한 영상 간의 차이가 보이는 범위를 의미한다. 차이가 해당 파라미터 값 이상 차이가 나는 경우, 즉 범위를 벗어나는 차이를 보이는 경우에 해당 위치에서 움직임이 발생했음을 정의하기 위하여 사용된다. 파라미터의 값은 비교되는 영상의 종류와 상태에 따라서 정의한다.

From here,

Denotes a result of converting a difference DM calculated based on a background model into a block unit image. In other words,

Means that the object to compare the difference with the background model is the current input image in time and the position to be compared is (x, y) in units of blocks. And

Denotes a set of position coordinates of a pixel image corresponding to positions x and y of the block image. In other words, a block is a collection of pixels, and a collection of coordinates of pixels corresponding to (x, y) of a block

to be. parameter

Denotes a range in which differences between images for distinguishing motions are visible. It is used to define that the motion occurred at the position when the difference is more than the parameter value, that is, the difference is out of range. The value of the parameter is defined according to the type and state of the image to be compared.

Subsequently, it is identified whether the movement detected in the block is due to the movement of the real object or the change of the part which was wrongly known in the background. To determine this, the present invention uses the distribution of the contours.

If the motion detected in the block is due to the motion of the real object, the difference in the contours will be strong because there will be a clear difference between the image of the object and the background model. On the other hand, in the case of motion detected as an object stationary in front of the background enters a part of the background model and moves suddenly, the background remaining after the object moves will naturally connect with the adjacent part of the background model, so that the contour The difference will be weak or absent. Therefore, it is possible to determine whether the movement area is the foreground FG or the original background GH.

To this end, in step S53, contours are detected in blocks classified as motion regions within the current image, for the current background model and the current image, respectively. Equation 6 shows the detection of the contour using a sobel edge detection technique.

여기에서,

에는 움직임 영역을 기준으로 블록단위 영상에 윤곽선이 존재하는지 여부를 저장한다. 즉,

는 윤곽선을 검출하는 대상이 배경모델(BGI)임을 의미하고,

는 현재의 입력 영상에서 윤곽선을 검출한 결과를 의미한다. 그리고

는 픽셀단위의 영상을 소벨 윤곽선 검출 기법을 이용하여 윤곽선을 검출한 결과를 블록단위로 변환한 값을 의미한다. 블록(x,y)에 해당하는 픽셀들에 소벨 윤곽선 검출 기법으로 검출한 결과, 윤곽선이 존재한다면

는 1을, 그렇지 않은 경우에는 0을 부여한다.

는 윤곽선을 검출하는 대상이 배경모델임을 의미하며,

는 윤곽선을 검출하는 대상이 현재 입력 영상임을 의미한다. 연산자

는 조건문이 동시에 만족하는 경우 참을 출력한다.

From here,

Stores whether an outline exists in the block unit image based on the movement area. In other words,

Means that the target for detecting the contour is the background model (BGI),

Denotes a result of detecting the contour from the current input image. And

Denotes a value obtained by converting the result of detecting the contour into the unit of block using the Sobel contour detection technique. If the Sobel contour detection technique detects the pixels corresponding to the block (x, y), the contour exists.

Gives 1, otherwise gives 0.

Means that the object to detect the contour is the background model,

Means that the target for detecting the contour is the current input image. Operator

Returns true if the conditional statements are satisfied at the same time.

In step S54, among the blocks whose difference between the contour of the current background model and the contour of the current image is equal to or greater than a predetermined value, the contour class of the block where the contour of the current image is detected is classified as the foreground, and the contour of the current image is not detected. The contour class of the block where the contour of the current background model is detected is classified as a ghost.

In the area where the motion is detected, the contour that does not appear in the background model may appear in the current image, or conversely, the contour that appears in the background model may disappear in the current image. The former is more likely to be the foreground and the latter is more likely to be ghost. Equation 7 formulates this concept.

여기에서,

(Edge Difference Map)에는 배경모델(BGI)을 기반으로 검출한 윤곽선 정보

와 현재 입력 영상을 기반으로 검출한 윤곽선 정보

의 차이를 계산한 값을 저장한다. 파라미터

는 비교하는 윤곽선 간의 차이 범위를 의미한다. 해당 범위를 넘어가는 경우 해당 위치에서 윤곽선의 차이가 발생했음을 정의하기 위하여 사용된다.

From here,

(Edge Difference Map) contains contour information detected based on background model (BGI)

Contour information detected based on the current and current input image

Save the difference between parameter

Means the range of difference between the contours to compare. It is used to define that there is a difference in the contour at the location when it exceeds the range.

여기에서,

는 윤곽선 클래스(Edge Class)를 의미한다.

From here,

Means an edge class.

Subsequently, each block is classified as a foreground or ghost by calculating a probability that the blocks, which are motion regions, are foreground and ghost, respectively, using a non-parametric probability distribution estimation technique based on the distribution of the contour class. do.

In step S55, the probability distribution that the block is the foreground is predicted from the blocks whose contour class is classified as the foreground, and the probability distribution that the block is the ghost is predicted from the blocks whose contour class is classified as ghost.

Equation 8 applies a non-parameter probability distribution prediction technique using a kernel function.

In operation S56, if the foreground probability of the block is greater than the ghost probability, the image class of the block is classified as the foreground FG. Otherwise, the image class is classified as ghost GH.

Blocks classified as foreground are later used for object tracking, and blocks classified as ghost are used for background update.

Next, the blocks must be grouped in the form of chunks corresponding to the object and collectively used for object tracking.

In operation S57, the blocks with the image class as the foreground are bundled and output as the current foreground blob, and the blocks with the image class as the ghost are bundled and output as the ghost blob. For example, the Connected Component Labeling (CCL) technique is used to group related blocks to form chunks, or blobs. Typically, using the CCL technique, identically labeled elements are grouped together to form a cluster. Often, however, elements with different labels can join a group if there is a spatial closeness. In this case, if the class having the most blocks in each blob is the foreground, the corresponding blob may be output as the current foreground blob, and if the class having the most blocks in each blob is the ghost blob, it may be output as a ghost blob.

7 is a conceptual diagram illustrating object tracking using a similarity between a foreground blob and a track in a video surveillance process according to an embodiment of the present invention.

The object tracking is to extract the object from the current image and successfully associate the extracted object with the moving trajectory information of the object, which has been maintained until now, so as to grasp how the object has been moved so far.

In order to efficiently track an object, the present invention proposes a method of first reducing the solution space to be searched and then improving the ability to distinguish between similar objects.

As described above, in order to associate the foreground blob, the foreground blob, with the foreground blob to one of the tracks so far, the similarity is calculated between the foreground blob and each track, and the most similar track and its foreground. Associate blobs Information such as the position of the corresponding foreground blob is updated in the associated track.

In this case, for example, assuming that an object is photographed at 25 frames per second, it will not deviate much from the trajectory of movement so far between the image frame and the frame, and will not show discontinuous movement, and will actually position adjacent blocks between adjacent frames. Will be on. Thus, if the foreground blob is in the vicinity of an area where there are some tracks, the foreground blob is very likely associated with any of the tracks.

In one embodiment, the similarity of the foreground blob information included in the previous track and the current foreground blob is calculated, the current track blob having a calculated similarity greater than a predetermined value corresponds to the previous track, and the previous track is updated.

In this case, if a candidate region is defined in the current image based on the past position of the foreground blob included in the previous track, the space to be searched may be reduced. If the current foreground blob is included in such a candidate region, the similarity is computed first using a specific similarity function with respect to the current foreground blob and the previous track. If the calculated similarity is greater than a predetermined value, the current foreground blob may be corresponded to the previous track and the previous track may be updated.

In another embodiment of reducing the solution space, first, one track set is composed of tracks whose distances between tracks are smaller than a predetermined value. The track set further includes feature points for distinguishing the tracks, as well as adjacent tracks, and similarity functions for comparing the feature points and extracting similarities. Similarity is obtained between the track and the current foreground blob when there are current foreground blobs in the candidate area occupied by the track set.

As explained in Figure 2, the track has a life cycle. In the case of an active track, the similarity calculated between the tracks in the track set and the current foreground blobs corresponding to the area occupied by the defined track set by grouping at least one track whose distance between the tracks is less than or equal to the predetermined value is greater than the predetermined value. Update the previous track by matching the large current foreground blob with the previous track.

If all the calculated similarities are less than a predetermined value, the track does not correspond to any current foreground blobs, so it is set as an inactive track.

Alternatively, if there is a current foreground blob that is not included in the track set area occupied by any track set, a new track for such spaced current foreground blobs can be created.

On the other hand, there are a number of similarity algorithms for distinguishing objects by the feature points of the objects, such as SAD, SSD, Batcharaya, autocorrelation techniques, and the like. These similarity algorithms may vary in computation, speed, and accuracy in some cases. In the present invention, a feature point for distinguishing each track in a track set and a similarity function set including a plurality of similarity functions are defined together with the track set, and the current foreground blobs and the previous one are based on the similarity function of one of the similarity functions. Calculate the similarity between tracks.

In one embodiment, the best performing similarity function is selected and computed for similarity based on the performance history of the similarity decision adaptively managed with respect to the similarity functions.

In another embodiment, the similarity functions are performed in the order of similarity functions, in which the operations are relatively simple, and if similarity results are similar to each other, for example, the similarity performance is greater than that of the similarity operation previously performed. Stops the operation and accepts the similarity result previously performed.

8 is a block diagram illustrating a video surveillance apparatus according to an embodiment of the present invention.

Referring to FIG. 8, the video surveillance apparatus 80 may include a camera 81, a background model generator 82, a motion class determiner 83, an object tracker 84, and a display 85. .

The image captured by the camera 81 is provided to the background model generator 82, the motion class determiner 83, the object tracker 84, and the display 85, respectively.

The background model generator 82 extracts temporally-spatially stable regions from the first photographed images and combines them to provide the first background model, and is subsequently classified as a ghost by the motion class determiner 83. Update the background model based on these.

The motion class determiner 83 extracts a motion area and a background area from the current image in comparison with the background model, and classifies the motion class of the motion area as foreground or ghost. The motion class determiner 83 bundles the areas classified as the foreground into the foreground blob, and transfers them to the object tracker 84. In addition, the motion class determiner 83 bundles the areas classified as ghosts into ghost blobs, and transmits them to the background model generator 82.

The object tracker 84 maintains tracks with movement information of previous foreground blobs. The object tracker 84 associates the foreground blob with the track by calculating the similarity between the foreground blob and the tracks transmitted from the motion class determiner 83, and updates the track, generates a new track, or disables the track according to the association result. To discard. Meanwhile, the object tracker 84 may combine the changed track information with the current video and send the changed track information to the display 85.

The display 85 may display the image captured by the camera 81 as it is, or may display the current image combined with the track information tracked by the object tracker 84 to the user.

As described above, although the present invention has been described by way of limited embodiments and drawings, the present invention is not limited to the above-described embodiments, which can be variously modified and modified by those skilled in the art to which the present invention pertains. Modifications are possible. Accordingly, the spirit of the invention should be understood only by the claims set forth below, and all equivalent or equivalent modifications will fall within the scope of the invention.

In addition, the apparatus according to the present invention can be embodied as computer readable codes on a computer readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data that can be read by a computer system is stored. Examples of the recording medium include ROM, RAM, optical disk, magnetic tape, floppy disk, hard disk, nonvolatile memory, and the like, and also include a carrier wave (for example, transmission through the Internet). The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.

80 Video Surveillance 81 Camera
82 Background Model Generator 83 Motion Class Determination Unit
84 Object tracker 85 Display

Claims (17)

Comparing the current background model with the current image, classifying blocks with relatively small changes as background and classifying the image class of blocks with relatively large changes as either foreground or ghost;
Grouping blocks in which the image class is the foreground and outputting the current foreground blob, and outputting the bundles in which the image class is ghost as the ghost blob;
Finding current foreground blobs corresponding to tracks accumulating information of previous foreground blobs and reflecting them in the tracks; And
And if the number of ghost blobs is accumulated more than a predetermined number, generating a next background model by updating a block position corresponding to the accumulated ghost blobs in the current background model. The method according to claim 1,
And initializing a background model by combining time-spatially stable regions extracted from successive images. The method of claim 2, wherein initializing a background model by combining time-spatially stable regions extracted from the consecutive images comprises:
Comparing the temporary background selected as one of the consecutive images with other images for each block, and counting the number of times the blocks having little change are classified as the background for each block; And
And outputting a block having a number of times classified as a background more than a predetermined value as a background sample. The method of claim 3, wherein the initializing the background model by combining the time-spatially stable regions extracted from the consecutive images comprises:
Comparing the temporary background and the current image selected as one of the continuous images, the temporary background and the past image, and the current image and the past image, respectively; And
For each block, if the number of times that it is determined that there is a change between the temporary background and the current image and between the temporary background and the past image but there is no change between the current image and the past image is greater than or equal to a predetermined value, And outputting a corresponding block of the current image as a background sample. The method according to claim 3 or 4,
And deriving one final background sample representing a plurality of background samples at each block position and combining the final background samples derived for each block to generate the background model. The method of claim 5, wherein the final background sample is
And a background sample corresponding to a median histogram of the plurality of background samples. The method of claim 1, wherein the comparing of the current background model and the current image, classifying blocks with less change as background and classifying blocks with relatively large change as either foreground or ghost,
Classifying the block as a background if the difference in the unit of a block between the current background model and the current image is less than or equal to a predetermined value; otherwise classifying the block as a motion region;
Detecting contours from blocks classified as motion regions in the current image, for the current background model and the current image;
Among the blocks of which the difference between the contour of the current background model and the contour of the current image is equal to or greater than a predetermined value, the contour class of the block from which the contour of the current image is detected is classified as the foreground, and the contour of the current image is not detected, but the current Classifying the contour class of the block in which the contour of the background model is detected as a ghost;
Predicting a probability distribution that the block is the foreground from blocks whose contour class is classified as foreground, and predicting a probability distribution that the block is ghost from blocks classified as ghost; And
If the foreground probability of the block is greater than the ghost probability, classifying the image class of the block as foreground; otherwise, classifying the object into an image. The method of claim 1, wherein the grouping of the blocks of which the image class is the foreground is output as a current foreground blob and the blocks of the image class of the ghost are bundled and output as a ghost blob,
Creating a blob by concatenating blocks using concatenation element labeling;
Outputting the corresponding blob as the current foreground blob if the class of the most blocks in each blob is the foreground; And
And outputting a ghost blob if the class of the most blocks in each blob is a ghost. The method of claim 1, wherein the finding and reflecting the current foreground blobs corresponding to the tracks accumulating the information of the previous foreground blobs and reflecting the tracks are as follows.
Associating the previous track with a current foreground blob whose similarity between the foreground blob information included in the previous track and the current foreground blob is greater than a predetermined value; And
And updating the foreground blob information of the previous track according to the movement of the corresponding current foreground blob. The method of claim 9, wherein the mapping of the previous track with the current foreground blob having a similarity between the foreground blob information included in the previous track and the current foreground blob is greater than a predetermined value,
Defining a candidate region in a current image based on a past position of a foreground blob included in the previous track; And
And associating the current foreground blob with the previous track when the similarity calculated first with respect to the current foreground blob and the previous track included in the candidate area is greater than a predetermined value. The method of claim 1, wherein the finding and reflecting the current foreground blobs corresponding to the tracks accumulated with the previous foreground blobs and reflecting the tracks includes:
A current foreground blob corresponding to an area occupied by a defined track set by grouping at least one track whose distance between tracks is equal to or less than a predetermined value and a current foreground blob having a similarity calculated between tracks in the track set greater than a predetermined value; Updating the previous track by matching the previous track;
Setting a track as an inactive track whose calculated similarity is less than a predetermined value and does not correspond to any current foreground blobs; And
Creating a new track for the current foreground blob that does not correspond to an area occupied by any track set. The method according to claim 11, wherein the similarity calculated between the tracks in the track set and the current foreground blobs corresponding to an area occupied by a track set defined by grouping at least one track whose distance between the tracks is equal to or less than a predetermined value. Updating the previous track by mapping a larger current foreground blob with the previous track,
A similarity function set including a feature point for distinguishing each track in the track set, a plurality of similarity functions, is defined with the track set, and the current foreground blobs and the previous track based on the similarity function of one of the similarity functions. Comprising the step of calculating the similarity between. The method of claim 12, wherein the similarity functions included in the set of similarity functions include absolute error sum (SAD), error square sum (SSD), battacharraya function, and correlation function. How to monitor the object in the image. The method according to claim 13, wherein the similarity function having the best performance is selected based on the performance history of the similarity determination accumulated with respect to the similarity functions. The object monitoring system according to claim 13, wherein a similarity function having a relatively simple operation is selected among at least two higher similarity functions having excellent performance based on the performance history of the similarity determination accumulated with respect to the similarity functions. Way. The method of claim 1, wherein when the ghost blobs are accumulated more than a predetermined number, generating a next background model by updating a block position corresponding to the accumulated ghost blobs in the current background model.
Accumulating ghost blobs block by block;
Deriving a change occurrence background region that can represent the ghost blobs accumulated in a predetermined number or more; And
And updating a block position corresponding to the accumulated ghost blob in the current background model as the derived change occurrence background area to generate a next background model. Compares the current background model with the current image, classifies blocks with less change as background, classifies the image class of blocks with larger changes as either foreground or ghost, and binds blocks with image class as foreground to the current foreground blob A motion class determiner for outputting a block to which the image class is ghosted and outputting the result as a ghost blob;
An object tracker which finds current foreground blobs corresponding to tracks accumulating information of previous foreground blobs and reflects the current foreground blobs to the tracks; And
And a background model generator for generating a next background model by updating a block position corresponding to the accumulated ghost blob in the current background model when the number of ghost blobs is accumulated more than a predetermined number.

Images