KR101640563B1 - Method and system for dectecting run - Google Patents

Abstract

A running sensing method and system are disclosed. The running detection system detects moving objects in the input image, detects the foreground edges for the detected objects, extracts the average motion vectors for the foreground edges, and detects the footprint in the detected objects. And the running detection system may be configured to determine a second footprint coincident with the second footprint coordinate using the first footprint coordinate and the average motion vector after obtaining the first footprint co- Estimate the footprint coordinates. At this time, the running detection system determines whether the object is running through the first foot-print coordinate and the second foot-print coordinate.

Description

METHOD AND SYSTEM FOR DECTECTING RUN

The present invention relates to a running detection method and system.

With increasing interest in social safety and increasing emphasis on environmental safety, there is an increasing need for a method and system for detecting running conditions.

The existing running detection method detects the running by calculating the distance traveled by the object using the transformed image, by converting the real three - dimensional coordinates into two - dimensional coordinates of the image through camera calibration.

Here, the camera calibration method for converting the three-dimensional coordinates of the real world (hereinafter, referred to as "world coordinates") into two-dimensional coordinates of the image is important. For this purpose, a method using a lattice-like planar pattern is generally used. The camera calibration performance depends on the calculation of a transformation matrix for transforming world coordinates into image coordinates, and this transformation matrix is called a homography matrix. The camera calibration method can be expressed by the following equation (1).

는 월드좌표를 영상좌표 변환하기 위한 회전/이동변환 행렬이고, X, Y, Z는 월드좌표이고, x,y는 영상좌표이다. 여기서,

를 호모그래피 매트릭스라고 하며,

는 각각 영상의 x,y에 대한 초점거리, 주점, 비대칭 계수하고 한다. 주점은 이미지 센서(CMOS, CCD 등)의 가로/세로 비율에 의해 발생하며, 비대칭 계수는 카메라 제작과정의 오차에서 발생한다. 카메라가 말들어진 초창기에는 이러한 변수의 영향이 컸지만 기술 발로 인해 영향이 거의 없다. In Equation (1), A is a matrix for correcting intra-camera distortion,

X, Y, and Z are world coordinates, and x and y are image coordinates, respectively. here,

Is called a homography matrix,

Are the focal length, principal point, and asymmetric coefficient for x and y of the image, respectively. The main point is caused by the aspect ratio of the image sensor (CMOS, CCD, etc.), and the asymmetry coefficient occurs in the error of the camera manufacturing process. In the early days when the camera was spoken, the influence of these variables was significant, but it was not affected by the technology foot.

Therefore, in order to obtain the homography matrix, the remaining parameters must be calculated. At this time, the world coordinates and the image coordinates are matched by using the grid pattern as shown in FIG. 1 is a diagram showing a lattice-like flat pattern used for obtaining a homography matrix.

Next, the computed homography matrix may be used to obtain the world coordinates corresponding to the foot-print of the object in the image. The foot-print of the object is defined by the pixel location at the bottom of the y-axis of the label in the image, and the x-axis center point of the label. Using the foot-print of an object acquired at any two time, you can calculate the distance that the object actually moves during that time.

However, this conventional method is not easy to apply because, when a camera is installed in a large quantity, the camera needs to be calibrated using a grid pattern in every installation place. Also, the foot-print described above is greatly influenced by illumination and shadow according to the image processing methodology, which is a major factor in the performance degradation of the system.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a method and system for detecting running by using user input information without using a flat pattern.

A further object of the present invention is to provide a method and system for detecting running using a foot-print that is robust to the effects of illumination and shadow.

According to an embodiment of the present invention, a running detection system is provided. The running detection system includes an object detection unit that detects an object moving in an image captured through a camera, a foreground edge detection unit that detects a foreground edge with respect to the object, a motion vector extraction unit that extracts an average motion vector with respect to the foreground edge,

A foot-print extracting unit for detecting a foot-print in the object, a coordinate transforming unit for obtaining foot-print coordinates, in which the object is coordinates away from the camera, using the foot-print, Estimating previous foot-print coordinates using the current foot-print coordinates and the average motion vector of the print coordinates, and determining whether the object is running through the current foot-print coordinate and the previous foot- And a situation determination unit.

The motion vector extraction unit may calculate the average motion vector by extracting a motion vector only for the foreground edge and calculating an average for the object.

The motion vector extraction unit may calculate the motion vector by differentiating an image for the foreground edge with x-axis, y-axis, and time, respectively.

The coordinate transformation unit may obtain the footprint coordinates based on the inverse transformation method.

The coordinate transformer may calculate a first distance that the footprint is away from the camera in a vertical direction and use the first distance to determine a second distance that is the distance that the footprint is horizontally away from the camera And the first distance and the second distance may be the footprint coordinates.

The running condition determiner may calculate the velocity of the object using the current foot-print coordinate and the previous foot-print coordinate.

The running condition determining unit may determine that the object runs when the speed exceeds the predetermined reference value and may determine that the object does not run when the speed is equal to or smaller than the predetermined reference value.

The object detecting unit may detect the object through a background modeling method.

Wherein the foreground edge detecting unit detects a video edge from a whole image of the object and detects a time edge using an image temporally continuous with a current image of the object and a current image of the object, The common component of the edge can be extracted at the foreground edge.

The motion vector extracting unit may extract the average motion vector using an optical flow method.

According to another embodiment of the present invention, there is provided a method of detecting running of an object using an image photographed through a camera in a running detection system. The running sensing method comprising the steps of: detecting an object moving in the image; detecting a foreground edge for the detected object; extracting an average motion vector for the foreground edge; Obtaining a first footprint co-ordinate, the first foot-print co-ordinate being the coordinates actually offset from the camera, using the first footprint co-ordinate and the average motion vector, Estimating a second footprint coordinate that is a previous footprint coordinate of the coordinates and determining whether the object is running through the first footprint co-ordinate and the second footprint co-ordinate .

The step of extracting the average motion vector may comprise extracting a motion vector only for the foreground edge, and calculating an average for the object with respect to the extracted motion vector.

Wherein obtaining the first footprint co-ordinates comprises: calculating a first coordinate, wherein the footprint is a coordinate that is vertically offset from the camera, and using the first co- And calculating a second coordinate which is a coordinate in a horizontal direction away from the first coordinate.

The determining may include calculating the velocity of the object using the first footprint co-ordinate and the second footprint co-ordinate.

The determining may further include determining whether the velocity exceeds a predetermined reference value, and determining that the object is running if the velocity exceeds the predetermined reference value.

Wherein the step of detecting the foreground edge comprises the steps of: detecting a video edge from a whole image of the object; detecting a time edge using a current image of the object and an image temporally continuous to the current image of the object; And extracting a common component of the image edge and the temporal edge to the foreground edge.

The camera may be a pinhole camera.

The footprint may be a pixel location at the bottom of the y-axis and an x-axis center point in a rectangular area of the detected object.

According to the embodiment of the present invention, the camera can be calibrated simply by detecting the running using the user input information.

According to another embodiment of the present invention, by estimating the footprint using an average motion vector, it is possible to detect running robustly to the influence of illumination and shadow.

1 is a diagram showing a lattice-like flat pattern used for obtaining a homography matrix.
2 is a diagram illustrating a running detection system according to an embodiment of the present invention.
3 is a view showing a pinhole camera model according to an embodiment of the present invention.
4 is a view showing an installation state of a pinhole camera according to an embodiment of the present invention.
Figure 5 shows a top view of the situation of Figure 4;
6 is a flowchart illustrating a running sensing method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

Throughout the specification and claims, when a section is referred to as "including " an element, it is understood that it does not exclude other elements, but may include other elements, unless specifically stated otherwise.

Now, a running detection method and system according to an embodiment of the present invention will be described in detail with reference to the drawings.

2 is a diagram illustrating a running detection system according to an embodiment of the present invention.

2, the running detection system 100 according to the embodiment of the present invention includes an object detection unit 110, a foreground edge detection unit 120, a motion vector extraction unit 130, a footprint extraction unit 140, A coordinate transformation unit 150, and a running condition determination unit 160. [

The object detecting unit 110 detects a moving object by applying a tracking algorithm to an image photographed by a camera. In the field of video surveillance, there are various tracking algorithms, and objects can be detected through a codebook-based background modeling method. The background modeling method learns the variance of the value of each pixel in the image, and detects the object by considering the foreground as a pixel having a defined threshold value or more. The background modeling method can be understood by those skilled in the art to which the present invention pertains, so that a detailed description thereof will be omitted.

The foreground edge detection unit 120 detects the foreground edge with respect to the object detected by the object detection unit 110. The foreground edge detecting unit 120 detects an edge of a plurality of images on the time axis for each pixel of the detected object. More specifically, the foreground edge detecting unit 120 detects an edge (hereinafter, referred to as an 'image edge') from a whole image of an input object and outputs a plurality of edge images (Hereinafter referred to as " time edge "). The foreground edge detection unit 120 extracts the common components of the detected image edge and the temporal edge, and defines an edge satisfying the common component as the foreground edge. A method for obtaining the foreground edge is described in detail in Korean Patent No. 10-1398684, and therefore, a detailed description thereof will be omitted.

Next, the motion vector extracting unit 130 extracts a motion vector with respect to the foreground edge detected by the foreground edge detecting unit 120. There are various methods for extracting the motion vector, but an optical flow method can be used as an embodiment of the present invention. The optical flow method assumes that the brightness value of the pixel does not change for a very short time but only the position change occurs. At this time, when each small position change occurs in the x and y axes, the motion vector becomes as shown in the following equation (2).

In Equation (2), expanding the right term to Taylor series yields Equation (3).

In order to simultaneously satisfy the equations (2) and (3), the sum of the undersigned of the right side of the equation (3) must be 0, and the partial differentiation for each variable (x, y, t)

By rearranging the expression (4) to the term indicating each partial derivative term, the expression (5) is obtained. And the equation (5) is replaced with the determinant.

In other words, the motion vector is calculated by differentiating the image into x-axis, y-axis, and time (t) axes, respectively.

로 할당한다. The calculated result is a value obtained by extracting one motion vector for one image. Generally, a motion vector is calculated for a partial image of a predetermined size around an arbitrary pixel, and motion is estimated for the corresponding pixel by performing this process on all the pixels. However, the motion vector extracting unit 130 according to the embodiment of the present invention extracts the motion vector only for the foreground edge detected by the foreground edge detecting unit 120, calculates an average for the object, Average motion vector of

.

On the other hand, the foot-print extraction unit 140 extracts a foot-print for the object detected by the object detection unit 110. When an object is detected by the object detecting unit 110, a rectangular area for the object is created. The foot-print extraction unit 140 sets a pixel position at the lower end of the y-axis in the rectangular area and a x-axis center point as a foot-print of the object.

The coordinate transforming unit 150 obtains coordinates of a corresponding object from the actual camera using the current object's footprint based on the inverse transformation method. The method of inverse translation is based on a pinhole camera model as shown in FIG. 3 below.

FIG. 3 is a view showing a model of a pinhole camera according to an embodiment of the present invention, and FIG. 4 is a view showing an installation state of a pinhole camera according to an embodiment of the present invention.

In the embodiment of the present invention, it is assumed that a camera is installed at a predetermined height as shown in FIG. 4 and the camera is tilted at an arbitrary angle to shoot an image. The coordinate transformation unit 150 calculates a distance Dy in which the object is vertically separated from the camera, and then calculates a distance Dx in which the object is horizontally separated from the camera.

)는 설치시 측정되며,

는 해당 카메라의 수직 화각(Vertical Angle of View)으로 계산되어 진다.

는 객체의 풋-프린트에 대한 수직 화각(Vertical Angle of Object Foot-Print)을 나타낸다. 따라서, 객체가 카메라에서 수직방향으로 떨어진 거리(Dy)와 카메라라의 설치 높이(H)의 관계는 아래의 수학식 7과 같다. Here, the height (H) of the camera and the tilted camera angle

) Is measured at installation,

Is calculated as a vertical angle of view of the camera.

Represents the Vertical Angle of Object Foot-Print for the object's foot-print. Therefore, the relationship between the distance (Dy) of the object in the vertical direction away from the camera and the installation height (H) of the camera is expressed by Equation (7) below.

Equation (8) is obtained by solving the left side of Equation (7).

와 초점 거리(f)의 관계는 수학식 9와 같다. Meanwhile,

And the focal length f are expressed by Equation (9).

Equation (8) and Equation (9) are summarized as Equation (10) below.

Dy, which is the distance in which the object is vertically away from the camera, is finally calculated using Equation (7) and Equation (10) as shown in Equation (11).

When Dy is calculated using Equation (11), the distance Dx of the object in the horizontal direction away from the camera can be calculated by using Dy.

는 카메라의 수평 화각(Horizontal Angle of View)이며,

는 객체의 풋-프린트에 대한 수평 화각(Horizontal of Angle of Object Foot-Print)을 나타낸다. 그리고 L은 카메라로부터의 기하학적인 거리(Diagonal Distance L from Camera)이며

로 정의된다. Figure 5 shows a top view of the situation of Figure 4; 5,

Is a horizontal angle of view of the camera,

Represents the Horizontal of Angle of Object Foot-Print for the object's foot-print. And L is the geometric distance from the camera (Diagonal Distance L from Camera)

.

의 관계는 수학식 12와 같다. The horizontal distance Dx of the object

Is expressed by Equation (12).

,

와 초점 거리 f와의 관계는 다음의 수학식 13과 같이므로, 최종적으로 객체의 수평 방향의 거리(Dx)는 수학식 14와 같이 계산된다. here,

,

And the focal length f is expressed by the following equation (13), finally the distance Dx in the horizontal direction of the object is calculated as shown in equation (14).

Finally, the coordinates (Dx, Dy) for the current foot-print are calculated as shown in Equations (11) and (14). A common method is to determine the running situation by comparing the coordinates of the current foot-print with the coordinates of the previous foot-print, but this can be affected by lighting and shadows.

를 이용하여, 이전 풋-프린트(foot-print) 좌표를 추정한다. 좌표 변환부(150)에 의해 계산된 현재 풋-프린트 좌표가

이면, n 시간 이전의 풋-프린트 좌표는 다음의 수학식 15와 같이 된다. Accordingly, the running situation determining unit 160 according to the embodiment of the present invention determines the running state of the vehicle by comparing the current footprint coordinates calculated by the coordinate transforming unit 150 with the average motion vector extracted by the motion vector extracting unit 130

To estimate the previous foot-print coordinates. If the current footprint co-ordinates calculated by the coordinate transformation unit 150 are

, The footprint co-ordinates before n hours are as shown in the following equation (15).

와 n 시간 이전의 풋-프린트 좌표

의 차이를 이용하여 해당 객체의 속도를 계산할 수 있다. 이와 같이 계산된 해당 객체의 속도는 아래의 수학식 16과 같이 된다. Accordingly, the running condition determining unit 160 determines whether the current foot-

And footprint coordinates before n hours

The speed of the object can be calculated using the difference of The velocity of the corresponding object thus calculated is expressed by Equation (16) below.

Meanwhile, the running situation determining unit 160 according to the embodiment of the present invention can determine that the corresponding object runs when the velocity of the corresponding object calculated as shown in Equation (16) exceeds a predetermined reference value.

6 is a flowchart illustrating a running sensing method according to an embodiment of the present invention.

6, an image photographed by the camera is input to the object detection unit 110 (S610). The object detecting unit 110 detects an object moving on the input image using a tracking algorithm (S620).

The foreground edge detection unit 120 detects foreground edges of the object detected by the object detection unit 110 (S630). That is, the foreground edge detection unit 120 extracts common components of the image edge and the temporal edge, and detects an edge satisfying the common component as the foreground edge.

Next, the motion vector extracting unit 130 extracts an average motion vector for the foreground edge detected by the foreground edge detecting unit 120 (S640). Here, an optical flow method can be used as a method of extracting a motion vector, and the motion vector extracting unit 130 extracts a motion vector with respect to a foreground edge and calculates an average over an object to extract an average motion vector.

The footprint extracting unit 140 extracts a foot-print for the object detected by the object detecting unit 110 (S650).

In step S660, the coordinate transforming unit 150 obtains coordinates of the corresponding object from the actual camera using the footprint of the current object based on the inverse transformation method. That is, the coordinate transformation unit 150 calculates the coordinates for the current foot-print using Equations (11) and (14).

를 이용하여, 이전 풋-프린트(foot-print)에 대한 좌표 추정한다(S670). The running condition determination unit 160 determines whether the current footprint coordinates calculated by the coordinate transformation unit 150 and the average motion vector extracted by the motion vector extraction unit 130

The coordinates of the previous foot-print are estimated (S670).

The running condition determination unit 160 calculates the velocity of the object using the current foot-print coordinate and the previous foot-print coordinate (S680). The running condition determining unit 160 determines that the object is running when the speed of the object exceeds a predetermined reference value (S690, S691). The running condition determining unit 160 determines that the object is not running if the speed of the object is equal to or smaller than a predetermined reference value (S690, S692).

It is expected that the running sensing method and system according to the embodiment of the present invention can be installed in a workplace handling dangerous goods, thereby greatly contributing to environmental safety.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, It belongs to the scope of right.

Claims (18)

An object detector for detecting a moving object in an image captured through a camera,
A foreground edge detector for detecting a foreground edge with respect to the object,
A motion vector extraction unit for extracting an average motion vector with respect to the foreground edge,
A foot-print extracting unit for detecting a foot-print in the object,
A coordinate transforming unit using the footprint to obtain foot-print coordinates, in which the object is a coordinate away from the camera; and
Estimating previous footprint co-ordinates using the current foot-print co-ordinates and the average motion vector, and determining whether the object is running through the current foot-print co-ordinates and the previous foot- And a running condition determining unit
Wherein the coordinate transformer calculates a first distance that the footprint is away from the camera in a vertical direction and uses the first distance to calculate a second distance that is the distance the foot- In addition,
Wherein the first distance and the second distance are the footprint coordinates
Running detection system. The method according to claim 1,
Wherein the motion vector extracting unit extracts a motion vector only for the foreground edge and calculates an average for the object, and calculates the average motion vector. 3. The method of claim 2,
Wherein the motion vector extracting unit calculates the motion vector by differentiating an image for the foreground edge with x-axis, y-axis, and time, respectively. delete delete The method according to claim 1,
Wherein the running condition determining unit calculates the velocity of the object using the current footprint co-ordinate and the previous footprint co-ordinate. The method according to claim 6,
The running condition determining unit may determine,
Determines that the object is running if the speed exceeds a predetermined reference value,
And determines that the object does not run when the speed is equal to or smaller than the predetermined reference value. The method according to claim 1,
Wherein the object detecting unit detects the object through a background modeling method. The method according to claim 1,
Wherein the foreground edge detecting unit detects a video edge from a whole image of the object and detects a time edge using an image temporally continuous with a current image of the object and a current image of the object, And extracts a common component of the edge to the foreground edge. The method according to claim 1,
Wherein the motion vector extractor extracts the average motion vector using an optical flow method. A method of detecting running of an object using an image captured through a camera in a running detection system,
Detecting a moving object in the image,
Detecting a foreground edge for the detected object,
Extracting an average motion vector for the foreground edge,
Detecting a footprint in the detected object,
Obtaining a first footprint co-ordinate in which the foot-print is an actual distant coordinate from the camera,
Estimating a second footprint co-ordinate, which is the previous foot-print co-ordinate of the first footprint co-ordinate, using the first footprint co-ordinate and the average motion vector, and
Determining whether the object is running through the first footprint co-ordinate and the second footprint co-ordinate,
Wherein obtaining the first footprint co-
Calculating a first coordinate where the footprint is a vertically offset coordinate from the camera, and
Calculating a second coordinate using the first coordinate, wherein the foot-print is a coordinate away from the camera in the horizontal direction
Running detection method. 12. The method of claim 11,
The step of extracting the average motion vector comprises:
Extracting a motion vector only for the foreground edge, and
And calculating an average of the extracted motion vectors for the object. delete 12. The method of claim 11,
Wherein the determining comprises calculating the velocity of the object using the first footprint co-ordinate and the second footprint co-ordinate. 15. The method of claim 14,
Wherein the determining step comprises:
Determining whether the speed exceeds a predetermined reference value, and
And determining that the object is running if the speed exceeds the predetermined reference value. 12. The method of claim 11,
Wherein the step of detecting the foreground edge comprises:
Detecting an image edge from the entire image of the object,
Detecting a time edge using a temporally continuous image of the current image of the object and the current image of the object, and
And extracting a common component of the image edge and the time edge to the foreground edge. 12. The method of claim 11,
Wherein the camera is a pinhole camera. 12. The method of claim 11,
Wherein the footprint is a pixel position at the lower y-axis and an x-axis midpoint in a rectangular area of the detected object.

Images