KR101460317B1 - Detection apparatus of moving object in unstable camera environment and method thereof - Google Patents

Abstract

A moving object detection apparatus for detecting a moving object from an image photographed in an unstable camera environment is disclosed. The camera motion estimator estimates the motion of the camera by estimating an affine matrix from a plurality of consecutive images. The moving object detection unit detects a moving object from a plurality of consecutive images using the difference of Gaussian based on the estimated camera motion. According to the present invention, movement of a camera can be detected even in an unstable camera environment having non-fixed mobility, thereby accurately detecting a moving object.

Description

[0001] The present invention relates to an apparatus and method for detecting a moving object in an unstable camera environment,

The present invention relates to an apparatus and method for detecting a moving object in an unstable camera environment, and more particularly, to a moving object detecting apparatus and method in an unstable camera environment that captures camera movement in an unstable camera environment, And methods.

Moving object detection is the most fundamental problem in computer vision and video processing, and it is an intelligent type that uses various types of cameras including robot vision, human computer interface and PTZ (Pan-Tilt-Zoom) camera or UVA (Unmanned Aerial Vehicle) And is widely applied to electronic products such as surveillance systems.

Korean Patent Publication No. 2007-0068408 entitled " Camera motion estimation method and video motion identification device ") discloses a method and system for camera motion analysis and moving object analysis, And a method for extracting primarily semantics from camera motion parameters in a video segment.

In addition, Korean Laid-Open Publication No. 2012-0125045 (entitled " object tracking device and method ") describes a method for tracking an object when an object exhibits non-affine transformation rather than affine transformation, Discloses an object tracking method and apparatus for tracking an object with excellent performance utilizing a resilient structure of a local patch and enabling real-time tracking.

An object of the present invention is to provide an apparatus for detecting a moving object in an unstable camera environment capable of detecting a moving object by capturing motion of a camera even in an unstable camera environment having a non-fixed mobility.

Another object of the present invention is to provide a program for causing a computer to execute a method of detecting a moving object in an unstable camera environment capable of detecting a moving object by capturing motion of a camera even in an unstable camera environment having non- And a computer readable recording medium recorded thereon.

According to another aspect of the present invention, there is provided a moving object detection apparatus for detecting a moving object from an image photographed in an unstable camera environment, A camera motion estimator estimating an affine matrix from an image of the camera; And a moving object detection unit detecting a moving object from the plurality of consecutive images using a difference of Gaussian based on the estimated camera motion.

According to another aspect of the present invention, there is provided a method of detecting a moving object in an unstable camera environment, the method comprising: detecting a moving object from an image captured in an unstable camera environment; A method for detecting an object, comprising the steps of: (a) estimating an affine matrix from a plurality of consecutive images to estimate a camera motion; And (b) a moving object detecting step of detecting a moving object from the plurality of consecutive images using a difference of Gaussian based on the estimated camera motion.

According to another aspect of the present invention, there is provided a computer readable medium storing a program for causing a computer to execute any one of the above methods.

The apparatus and method for detecting moving objects in an unstable camera environment according to the present invention can accurately detect a moving object by capturing camera movement even in an unstable camera environment having unfixed mobility.

FIG. 1 is a block diagram showing a configuration of an apparatus for detecting a moving object in an unstable camera environment according to the present invention.
2 is a block diagram illustrating the configuration of a camera motion estimator according to the present invention.
FIG. 3 is a flowchart illustrating an operation procedure of a moving object detection method in an unstable camera environment according to the present invention;
4 is a flowchart illustrating a camera motion estimation process according to an embodiment of the present invention.
FIG. 5 is a diagram illustrating a result of camera motion estimation using an affine model according to an embodiment of the present invention. FIG.
FIG. 6 is a diagram illustrating a Gaussian difference according to an embodiment of the present invention. FIG.
7 is a diagram illustrating a moving object detection process according to an embodiment of the present invention,
FIGS. 8 to 11 are views illustrating results of a moving object detection method in an unstable camera environment according to the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of an apparatus and method for detecting moving objects in an unstable camera environment according to the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a block diagram showing a configuration of a moving object detecting apparatus in an unstable camera environment according to the present invention, and FIG. 2 is a block diagram showing the configuration of a camera motion estimating unit according to the present invention. 3 is a flowchart illustrating an operation procedure of a moving object detection method in an unstable camera environment according to the present invention.

Referring to FIG. 1, a moving object detecting apparatus (hereinafter, referred to as 'moving object detecting apparatus') 100 in an unstable camera environment according to the present invention includes a camera motion estimating unit 200 and a moving object detecting unit 300 .

Unlike a fixed cctv camera, a camera that moves left and right and up and down along a moving object instead of capturing a fixed area moves the camera itself while taking a moving object. Therefore, it is necessary to estimate the motion of the camera before detecting the moving object. That is, as shown in FIG. 3, the moving object detecting apparatus and method according to the present invention comprises a step of estimating camera motion for motion compensation and an object detecting step.

Therefore, the camera motion estimation unit 200 estimates the motion of the camera by estimating an affine matrix using a plurality of consecutive images.

2, the camera motion estimation unit 200 includes a local affine motion estimator 210 and a global affine motion estimator 220. The local affine motion estimator 210 and the global affine motion estimator 220 are the same as those of FIG.

The local affine motion estimator 210 selects an arbitrary region from three or more images in an arbitrary image obtained from the camera and performs a local affine motion using an optical flow algorithm based on an affine motion . As shown in Fig. 4 (a), four regions (dotted line regions) are selected from an image, and local affine motion is estimated in each region. At this time, the area in the image may be preset by the user or may be set by the administrator. It is also preferable that the number of selected regions is three or more.

Based on the estimated local affine motion, the global affine motion estimator 220 estimates the global affine motion by analyzing how far the center points in the previously selected four regions have shifted from the successive multiple images. As shown in FIG. 4 (b), global affine motion can be estimated by analyzing where the center points in the selected four regions have moved.

Further, as shown in FIG. 4 (c), a global affine motion estimation may be performed through a global affine motion estimation and a refine process to minimize an error.

The general affine transformation can be expressed as: < EMI ID = 1.0 >

는 아핀 카메라 모델 매트릭스(affine camera model matrix),

는 선형 아핀 파라미터(linear affine parameter), 그리고,

는 변형된 파라미터(translation parameter)를 의미한다. here,

Is an affine camera model matrix,

Is a linear affine parameter,

Quot; means a translation parameter.

Local affine motion is estimated using an optical flow method based on the affine model. Specifically, assuming that an arbitrary area in the current image is I (x, y, t), the same area in the subsequent image is I (x, y, t + 1) The motion estimation is started using the affine transformation. To estimate the affine matrix, a quadratic error function of the following equation (2) can be minimized.

, 그리고,

는 전체 영역을 의미한다. here,

, And,

Means the entire area.

는

의 비선형 함수이기 때문에 에러값을 최소화하기 위해서는 다음 수학식 3을 이용하여 선형화해야 한다. In Equation (2)

The

, It is necessary to linearize the error value using the following equation (3).

와

는 수평/수직 방향의

의 공간 미분(spatial derivative), 그리고,

는

의 시간 미분(temporal derivative)을 의미한다. here,

Wow

Horizontal / vertical direction

And the spatial derivative of

The

And a temporal derivative of < / RTI >

The error function described in Equation (3) can be rearranged to simplify the following Equation (4).

이다. here,

to be.

는

의 2차 함수이기 때문에 상술한 에러값을 최소화하는 벡터

를 위한 솔루션은

관점에서 미분하여 결과값을 0(zero)으로 만드는 것이다. 따라서 벡터

에 대해 해결되어야 하는 선형 방정식의 집합은 다음 수학식 5와 같다.

The

, The vector minimizing the above error value

The solution for

From the point of view, the result is differentiated to 0 (zero). Therefore,

The set of linear equations to be solved for Equation 5 is as follows.

는 6x6 매트릭스이다. 이때, 매트릭스

는 역변환(invertible)이 된다는 보장은 없지만,

는

영역이 충분히 크고(large) 입력된 이미지가 충분한 콘텐트(content)를 가진다면 일반적으로 역변환될 것이다. here,

Is a 6x6 matrix. At this time,

Is not guaranteed to be invertible,

The

If the region is large enough and the input image has sufficient content, it will generally be inversely transformed.

로 정의되며,

이다.

는 변형 벡터를 나타낸다. 변형에 의해 이동된 점

은 다음 수학식 6에 기재된 바와 같이 정의된다. The affine transformation is performed based on the center axis of the image, and the center point is considered only for the transformation regardless of the affine transformation. The center point of each area is

Lt; / RTI >

to be.

Represents a deformation vector. Point moved by deformation

Is defined as shown in Equation (6) below.

이 글로벌 아핀 모션

에 의해

로 이동된다고 했을 때 추정하고자 하는 모션

는 최소 자승법을 이용하여 다음 수학식 7과 같이 계산할 수 있다.

This global affine motion

By

The motion to be estimated

Can be calculated by the following equation (7) using the least squares method.

를 추정하기 위해 최종 리파인(refine) 과정을 거칠 수 있다. 리파인 단계는 도 4 (c)에 도시된 바와 같이 다음 영상의

를 사용하여 아핀 변환 후 이동된 영상과 현재 영상의 동일한 네 영역만을 가지고 가우시안 피라미드 스킴(Gaussian pyramid scheme)을 수행하지 않고 수학식 5를 이용하여 구한

를

에 보상하여 최종 아핀 매트릭스

를 추정한다. More detailed

A final refinement process may be performed to estimate the total number of users. As shown in Fig. 4 (c), in the refinement step,

Using Equation 5 without performing a Gaussian pyramid scheme with only the same four regions of the shifted image and the current image after affine transformation,

To

Lt; RTI ID = 0.0 > affine matrix

.

FIG. 5 is a diagram illustrating a result of performing affine transformation on an input image using an estimated camera motion. FIG. 5A shows the current image, FIG. 5B shows the next image, FIG. 5C shows the difference image between the current image and the next image, FIG. (E) represents the next inter-image difference image compensated using the current image and the refined affine model.

The moving object detecting unit 300 detects a moving object from a plurality of consecutive images using Difference of Gaussian (DoG) based on the estimated camera motion. That is, in the object detection step, a moving object is detected using a frame difference scheme using a Gaussian difference (DoG) and a non-parametric kernel density estimation algorithm. At this time, it is preferable to use three images for a plurality of images.

The frame difference scheme compensates the next image to the current image using the global affine motion matrix estimated by the global affine motion estimating unit 220 and calculates the global affine motion matrix And inversely transforms the previous image into a current image.

An average image obtained by averaging the current image and the compensated two images (the next image and the previous image) adjacent to the current image is obtained. The average image generation is defined by the following equation (9).

와

은 보상된 이전 영상과 보상된 다음 영상을 의미한다. here,

Wow

Means the compensated previous image and the compensated next image.

When the average image and the continuous image are separated, object detection error occurs due to affine motion estimation error and interpolation used in the affine transformation process. In order to overcome this problem, an average image is generated using the compensated two images adjacent to the current image. In addition, in order to remove noise, the average image and the compensated two images have different sigma (sigma) Perform blur. Next, the difference between the average image and the compensated two images is obtained by using a Difference of Gaussian (DoG). The two generated DoG images are defined as follows.

는 가우시안 블러 커널을 의미한다. 가우시안 블러를 통해 잡음을 제거하면서 동시에 객체의 이동 영역을 검출할 수 있다. 도 6은 DoG를 사용한 결과 영상을 보여준다. 도 6의 (a)는 평균 영상을 나타내고, (b)는 보상된 이전 영상, (c)는 보상된 다음 영상을 나타내며, (d)는 (a)와 (b) 사이 DoG를 나타내며, 그리고, (e)는 (a)와 (c) 사이 DoG를 나타낸다. 즉, 도 6의 (d)와 (e)에 도시된 바와 같이, 객체가 이동하지 않은 영역은 DoG에 의해

과

가 동일한 값을 가지지만, 객체가 이동한 영역에서는

과

의 부호가 반대가 되는 특성을 가지게 된다. here,

Means the Gaussian Blur kernel. Through the Gaussian blur, the moving region of the object can be detected simultaneously while the noise is removed. Figure 6 shows the resulting image using DoG. 6 (a) shows an average image, (b) shows a compensated previous image, (c) shows a compensated next image, (d) shows a DoG between (a) and (b) (e) represents DoG between (a) and (c). That is, as shown in (d) and (e) of FIG. 6, the area in which the object has not moved is determined by DoG

and

Has the same value, but in the area where the object moved

and

The sign of the sign is opposite.

In other words, the difference image obtained by subtraction has the characteristic that the image and the sign are opposite to each other in the region where the object moves, and the moving object can be detected from this characteristic. That is, when the moving region of the object is detected using the above-described characteristic, the following is obtained.

는 입력 값의 부호를 나타내고,

는 잡음 제거를 위한 스레시홀드(threshold)이다. here,

Represents the sign of the input value,

Is a threshold for noise cancellation.

However, the difference image may have a hole in the center of the object, and still noise may remain. Therefore, to solve this problem, non-parametric kernel density estimation is used to detect the final object area.

는

가 1인 총 화소 개수이고,

와

는

가 1인 화소의 좌표를 의미하며, 그리고,

는 밴드위스(bandwidth)를 나타낸다.

로부터 최종 객체 검출은 다음과 같이 정의된다. here,

The

Is the total number of pixels,

Wow

The

Quot; 1 " means the coordinates of a pixel having " 1 &

Represents a bandwidth.

The final object detection is defined as follows.

은 객체 검출을 위한 스레쉬홀드(threshold)로 Otsu threshold를 사용하여 결정된다.

로부터 라벨링을 수행하고 크기가 작은 영역은 제거하여 최종 객체 영역을 검출하게 된다. here,

Is determined using the Otsu threshold as a threshold for object detection.

And the small area is removed to detect the final object area.

와 이동 객체를 검출한 결과를 나타낸다. 즉, 수학식 12에 의해 검출된 객체 이동 영역은 도 7의 (b)에 도시되어 있고, 도 7의 (c)에 도시된 바와 같이 논파라메트릭 커널 덴서티 추정(non-parametric kernel density estimation)을 통해 추정된 밀도를 통해 최종 객체가 검출된 결과를 도 7의 (d)를 통해 확인할 수 있다. FIG. 7 shows a result of detecting a moving object by applying the moving object detecting method according to the present invention. 7 (a) shows an original image, (b) shows a resultant image obtained by detecting a moving region of an object, and (c) and (d)

And the moving object. That is, the object moving region detected by Equation (12) is shown in FIG. 7 (b), and non-parametric kernel density estimation as shown in FIG. 7 (c) The result of the detection of the final object through the density estimated by the method shown in FIG. 7 (d) can be confirmed.

는 0.5를,

는 3으로 설정하였다. 8 to 9 are diagrams illustrating results of a moving object detection method in an unstable camera environment according to the present invention. For camera motion estimation, the size of an arbitrary image selected from the input image is 61x61, and the position is selected in the region excluding the object region detected in the previous image. The size of the Gaussian kernel for DoG is 15x15,

0.5,

Was set to 3.

FIG. 8 shows the results of object detection in an unstable camera environment directly photographed. Referring to FIG. 8, it can be seen that the camera continuously detects the object area while moving. However, in the case of (d) and (f) images, it can be seen that an object misdetection is detected when strong edge components exist in the outline of the image. This error is an error that occurs due to an increase in the affine transformation error toward the outside of the image due to the lens distortion.

FIG. 9 is a result of experiment in a case where there is a zoom in / zoom out function on a camera similar to the shooting environment of FIG. 8, but there is a faster camera movement. Even if the camera movement and zoom exist, the object region is detected. However, if a sudden camera movement such as (d) and (f) is generated, motion blocks are generated in the image and an object detection error is generated.

FIGS. 10 and 11 show the result of detecting an object using an UCF aerial action data set as an image taken from an unmanned aerial vehicle (UAV). It is not suitable to use the moving object detection using the background difference because the unmanned aerial photographing is performed while moving. Therefore, it can be seen that the object can be detected stably by using the moving object detecting method according to the present invention.

The moving object detecting apparatus and the moving object detecting method according to the present invention can quickly estimate a global matrix by estimating global affine motion using local affine motion detected in any of a plurality of regions of an image. Also, when continuously moving images such as a pan-tilt-zoom camera or a UAV (Unmanned Aerial Vehicle) camera, a new background image is input before the background is modeled, It is not desirable to use the method. Therefore, the moving object detection method according to the present invention is more suitable in an environment where the camera moves continuously because the moving object is detected using only the adjacent image without using the background modeling, and memory for background modeling can be reduced, .

The present invention can also be embodied as computer-readable codes on a computer-readable recording medium. A computer-readable recording medium includes all kinds of recording apparatuses in which data that can be read by a computer system is stored. Examples of the computer-readable recording medium include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like, and a carrier wave (transmission via the Internet). In addition, the computer-readable recording medium may be distributed to a computer system connected to a wired / wireless communication network, and a computer-readable code may be stored and executed in a distributed manner.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is clearly understood that the same is by way of illustration and example only and is not to be taken by way of limitation in the embodiment in which said invention is directed. It will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the scope of the appended claims.

100: Moving object detection device
200: camera motion estimation unit
210: local affine motion estimation unit
220: global affine motion estimation unit
300: Moving object detection unit

Claims (13)

delete A moving object detection apparatus for detecting a moving object from an image photographed in an unstable camera environment,
A camera motion estimator estimating an affine matrix from a plurality of consecutive images to estimate a motion of the camera; And
And a moving object detector for detecting a moving object from the plurality of consecutive images using a difference of Gaussian based on the estimated camera motion,
The camera-
A local affine motion estimator for estimating a local affine motion using an optical flow algorithm based on an affine motion by selecting an arbitrary region of at least three from the image; And
And a global affine motion estimator for estimating a global affine motion by comparing center points located in the selected arbitrary region from the plurality of consecutive images. Object detection device. 3. The method of claim 2,
Wherein the global affine motion estimator further performs a process of refining the estimated global affine motion. ≪ RTI ID = 0.0 > 15. < / RTI > 3. The method of claim 2,
Wherein the moving object detection unit compensates the next image with the current image using the global affine motion matrix estimated by the global affine motion estimating unit and inverses the global affine motion matrix estimated from the previous image, Is compensated with the current image. 5. The method of claim 4,
Wherein the moving object detection unit generates an average image obtained by averaging the current image and the compensated two images adjacent to the current image, performs Gaussian blurring on the average image and the compensated two images, Wherein the difference image is generated by subtracting the average image and the compensated two images from each other, thereby generating a difference image. 6. The method of claim 5,
Wherein the generated difference image has a different sign value from an area in which the object has not moved with respect to an area in which the object has moved, in the unstable camera environment. delete A moving object detecting method performed by a moving object detecting apparatus for detecting a moving object from an image photographed in an unstable camera environment,
(a) estimating a motion of a camera by estimating an affine matrix from a plurality of consecutive images; And
(b) a moving object detecting step of detecting a moving object from the plurality of consecutive images using a difference of Gaussian based on the estimated camera motion,
The step (a)
(a1) a local affine motion estimation method for estimating a local affine motion using an optical flow algorithm based on affine motion by selecting an arbitrary region from three or more images in the image, step; And
(a2) a global affine motion estimation step of estimating a global affine motion by comparing the center points located in the selected arbitrary area from the plurality of consecutive images; and A method for detecting moving objects in a mobile terminal. 9. The method of claim 8,
Wherein the step (a2) further performs a process of refining the estimated global affine motion. 9. The method of claim 8,
Wherein the step (b) comprises: compensating the next image using the global affine motion matrix estimated in the step (a2) with the current image, inverse the estimated global affine motion matrix in the previous image, Is compensated with the current image. 11. The method of claim 10,
The step (b) may include: generating an average image based on the current image and an average based on the compensated two images adjacent to the current image, performing Gaussian blurring on the average image and the compensated two images, Wherein the difference image is generated by subtracting the average image and the compensated two images from each other, thereby generating a difference image in the unstable camera environment. 12. The method of claim 11,
Wherein the generated difference image has a different sign value from an area in which an object has not moved with respect to an area in which the object has moved, in the unstable camera environment. A computer-readable recording medium having recorded thereon a program for causing a computer to execute a moving object detection method in an unstable camera environment according to any one of claims 8 to 12.

Images