KR20160014413A - The Apparatus and Method for Tracking Objects Based on Multiple Overhead Cameras and a Site Map - Google Patents

Abstract

The present invention relates to an object tracking apparatus based on multiple overhead cameras and a site map which comprises: an image-based local object tracking unit which tracks an object and creates a local object tracking result when receives camera images from multiple cameras installed in a place at a certain height from a site to be monitored and towards the bottom such that parts to be photographed can be overlapped; a coordinate converting unit between camera images and site map which converts a pixel coordinate of each camera image received from the multiple cameras into a point coordinate of a site map; a map-based global object tracking unit which creates a global object tracking result by using the local object tracking result and a local object point coordinate on the site map; and a real-time site map display unit which displays a location of a global object being tracked on the site map by using the global object tracking result. Therefore, the present invention is advantageous in grasping movements or moving lines of human objects in the site to be monitored, by tracking the human objects from multiple overhead cameras installed in a place at a certain height from the site and towards the bottom so as to partly overlap cameras′ image region; by integrating local human-object tracking results from each camera on the site map; by tracking the objects globally on the site map; and by displaying the global object tracking result on the site map when monitoring in real time or searching.

Description

 BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an object tracking apparatus based on a plurality of overhead cameras and a site map,

The present invention relates to an apparatus and method for tracking an object based on a plurality of overhead cameras and a site map, and more particularly, to a method and apparatus for tracking human objects in a surveillance site through a plurality of installed overhead cameras, To an apparatus and method for integrating human object tracking results on a site map, tracking objects globally on the site map, and utilizing the results.

2. Description of the Related Art In recent years, demand for an intelligent CCTV system that senses and tracks moving objects by analyzing video images input from a CCTV camera in real time and automatically senses meaningful events based on the detected moving objects is increasing day by day. In addition, there is an increasing trend to utilize the intelligent CCTV system not only in the security / surveillance field, but also in the business intelligence (BI) field. Typical examples include counting the number of visitors passing by analyzing the image of a camera installed at the entrance of a store or the top of a corridor, or measuring the amount of time a visitor stays at a specific place.

In most existing intelligent CCTV systems, object tracking is performed independently for each camera, and linkage tracking between cameras is not performed. Therefore, it is difficult to grasp the overall movement (movement path) of a specific object within the monitoring site. In addition, it is difficult to grasp the movement of a plurality of objects tracked by a plurality of cameras in a monitoring site at a glance, and it is difficult to utilize the object tracking results in real time monitoring or searching.

However, since it is a generalized method, it is difficult to actually apply it to a store or the like. SUMMARY OF THE INVENTION Accordingly, the present invention has been made keeping in mind the above problems occurring in the prior art. In addition, it does not provide a method to utilize the object tracking result in real-time monitoring or retrieval such as a method for grasping the movement of objects tracked by a plurality of cameras at a glance.

Korean Patent Registration No. 10-1178687 In a copper wire analysis system and method that integrate RFID technology and video technology, a method of grasping a customer's movement line by using an RFID reader and a camera installed in various places along a store route is proposed, There is a limitation that it is necessary to carry the RFID tag, and there is a disadvantage that the cost is increased because the RFID tag / reader is used together in comparison with the method of using only the camera in terms of system construction cost.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problems, and it is an object of the present invention to provide an overhead camera which is installed so as to face the ground on a certain height of a surveillance site, , Integrating the results of the local human object tracking obtained from each camera on the site map to track the objects globally on the site map and displaying the global object tracking results on the site map in real time monitoring or searching The present invention provides an object tracking apparatus based on a plurality of overhead cameras and a site map that enables a user to grasp the movement or movement of human objects in a monitoring site at a glance, and a method thereof.

Among the embodiments, a plurality of overhead cameras and an object tracking device based on a site map are used to capture a camera image from each of a plurality of cameras installed toward the ground such that a part of the ground surface is photographed at a place of a certain height from the ground surface of the surveillance area An image-based local object tracking unit for tracking the object and generating a local object tracking result upon receipt of the camera image; A site map-based global object tracking unit for generating a global object tracking result using the coordinate transformation unit, the local object tracking result, and the point coordinates of the local object on the site map, and a global object tracking unit Between real-time on the sitemap Map expression includes parts.

Among the embodiments, a plurality of overhead cameras and an object tracking method based on a site map are configured to receive a camera image from each of a plurality of cameras installed toward the ground such that a part of the ground is overlapped at a predetermined height from the ground Generating a local object tracking result by tracking an object in each camera image; converting pixel coordinates of a local object in each camera image into point coordinates on a site map; Generating a global object tracking result using the point coordinates of the local object on the site map, and displaying the position of the global object being tracked on the site map using the global object tracking result.

According to the present invention, a plurality of overhead cameras installed so as to face the ground on a certain height of a surveillance site and provided so as to overlap a photographing region between neighboring cameras are tracked, By integrating the obtained local person object tracking result on the site map, it is possible to track the objects globally on the site map, and to display the global object tracking result on the site map in real time monitoring or searching, Or the copper wire can be grasped at a glance.

1 is a diagram illustrating an object tracking system according to an embodiment of the present invention.
2 is a diagram illustrating an object tracking apparatus 200 according to an embodiment of the present invention.
3 is a flowchart showing an operation flow of the global object tracking unit 203 based on the site map according to an embodiment of the present invention.
4 is a flowchart illustrating a correspondence relationship between a local object and a global object according to an exemplary embodiment of the present invention.
5 is a diagram illustrating a method of setting a mapping relationship between an overhead camera image and a site map according to an embodiment of the present invention.
6 is a diagram illustrating an example of a lens distortion-removed image according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a diagram for explaining a configuration of a human object tracking system based on a plurality of overhead cameras and a site map.

Referring to FIG. 1, an object tracking system includes two or more overhead cameras 101 or 102 and an object tracking device 200. The overhead camera 101 or 102 is installed so as to face the ground at a predetermined height (at least 2.5 m or more) from the ground. In addition, the two cameras 101 and 102 neighboring each other are installed such that a part of the ground surface overlaps with the image so that the same object passing under the neighboring cameras can be continuously tracked. In FIG. 1, images 103 and 104 are examples of images taken at the same time in neighboring overhead cameras 101 and 102, respectively.

In FIG. 1, the object tracking device 200 is connected to two or more overhead cameras 101 and 102 to detect and track people moving under a camera from a camera image input from the camera in real time. In addition, the object tracking system 200 generates object tracking results on a site map (plan view for a surveillance area) in real time from the object tracking results on the camera images of respective cameras. At this time, the same object photographed by the neighboring cameras is processed to be tracked as an object having the same ID on the site map. In FIG. 1, the site map 105 shows an example in which object tracking positions are displayed on the site map using the object tracking results in the images 103 and 104.

FIG. 2 is a diagram illustrating an internal configuration of the object tracking apparatus 200 in detail.

2, the object tracking apparatus 200 includes an image-based area object tracking unit 201, a coordinate conversion unit 202 between the camera image and the site map, and a site map-based global object tracking unit 203 And additionally includes a real-time site map display unit 204, an object metadata storage unit 205, and an object metadata search / playback unit 206.

2, the image-based local object tracking unit 201 receives a real-time camera image from a camera connected to the object tracking apparatus 200, and detects and tracks a person from the camera image to generate a local object tracking result. Here, a local object represents an object being tracked in an image of a specific camera, and has an object unique ID value in the camera and an object position coordinate (pixel coordinate) in the current video frame.

In FIG. 2, a coordinate conversion unit 202 between a camera image and a site map performs a function of converting pixel coordinates of a specific camera image into point coordinates of a corresponding site map.

2, the site-map-based global object tracking unit 203 performs a function of the coordinate transformation unit 202 between the local object tracking result received from the plurality of image-based local object tracking units 201 and the camera image and the site map To generate global object tracking results on the site map in real time. Here, the global object is an object representing the position of the object being tracked in the monitoring area as point coordinates on the site map, and has a unique object ID value on the site map and a current point position coordinate value.

In FIG. 2, the real-time site map display unit 204 uses the global object tracking result generated by the site-map-based global object tracking unit 203 to display the location of the global objects being tracked on the site map in real time.

In FIG. 2, the object metadata storage unit 205 records the global object tracking result (object metadata) generated by the site map-based global object tracking unit 203 in the storage device. The object metadata stored at this time includes the current time, the unique ID of the global object, and the current position of the global object on the site map.

In FIG. 2, the object metadata search / playback unit 206 searches for and reproduces the object metadata stored in the object metadata storage unit 205. The retrieval of the object metadata includes retrieving the retrieved object metadata from the storage device within a specified period. The playback of the object meta data may be performed by displaying the movement of the global objects according to the time on the site map using the searched object meta data, displaying the movement trajectory of the whole or selected global objects on the site map, And counting the positions of the objects on the site map using the counting result values, and displaying the appearance frequency of the objects by position on the site map in a shade or color.

FIG. 3 is a flowchart illustrating a method of tracking global objects from the tracking results of the local objects in the global object tracking unit 203 of FIG.

Referring to FIG. 3, in step S301 of FIG. 3, the tracking information of the specific area object currently being tracked is fetched from the image-based area object tracking unit 201 of FIG. The tracking information includes a unique ID of the local object and a positional coordinate (pixel coordinate) of the local object in the current frame.

In step S302 of FIG. 3, the position coordinates in the current frame of the area object are converted into point coordinates on the site map using the coordinate conversion unit 202 between the camera image and the site map in FIG.

은 지역 객체

을 자식 객체로 가지고 있고, 전역 객체

는 지역 객체

와

을 자식 객체로 가지고 있고, 전역 객체

은 자식 객체

를 자식 객체로 가지고 있다. 이때

와

은 이웃하는 두 카메라 A와 B의 영상 사이의 겹치는 영역에서 동일 대상물에 대한 추적 결과이므로 하나의 전역 객체

로 대응이 된다.In step S303 of FIG. 3, a global object having the local object as a child object is searched among the global objects currently being tracked. Each global object has its corresponding local objects as child objects. For example, in the case of FIG. 4,

A local object

As a child object, and a global object

Is a local object

Wow

As a child object, and a global object

Child object

As a child object. At this time

Wow

Is a result of tracking the same object in the overlapping area between the images of the two neighboring cameras A and B,

.

If there is a global object having the local object as a child object in step S303, the flow advances to step S308. If there is no global object having the local object as a child object in step S303, the flow advances to step S304.

In step S304 of FIG. 3, it is determined whether there is a global object corresponding to the local object. For each global object being tracked, determine the distance between the center of the global object's site map coordinates and the center of the local object's center of the site map coordinates so that the distance is constant (typically half of the width of the object represented on the site map) The global object corresponding to the local object is found.

If there is a global object corresponding to the local object in step S305 of FIG. 3, the local object is added as a child object of the global object in step S306, and the flow advances to step S308.

If there is no global object corresponding to the local object in step S305, a new global object is created in step S307, the local object is added as a child object of the new global object, and then step S308 is performed Go ahead.

In step S308 of FIG. 3, a global object that does not have any child objects among the global objects currently being tracked is removed from the global object tracking list. The image-based area object tracking unit 201 of FIG. 2 notifies the site-map-based global object tracking unit 203 of the completion of the tracking of the specific area object. The site map-based global object tracking unit 203 finds a global object having the local object as a child object, and removes the corresponding local object from the child object list of the corresponding global object. In the case of a global object in which all child objects (ie, corresponding local objects) have been removed in this way, it means that the actual tracking object has disappeared, so that the global object is removed from the global object tracking list.

Steps S301 to S308 of FIG. 3 are performed for all the local objects currently being tracked, and the processing for all the local objects is repeatedly performed for each predetermined period (typically, video analysis frame rate).

As described above, the global object currently being tracked has local objects corresponding thereto as child objects. At this time, the current position of the global object on the site map is determined by the average value of the site map coordinates of the corresponding child objects (local objects).

The coordinate conversion unit 202 between the camera image and the site map in FIG. 2 performs a function of converting pixel coordinates of a specific camera image into point coordinates on a site map. In order to perform the above function, the coordinate conversion unit 202 between the camera image and the site map must know in advance how the camera image of the specific camera is mapped to the site map. The present invention proposes a method of setting a mapping relation between a camera image and a site map by designating a correspondence relationship between a plurality of minutiae points between a site map and a camera image. In this case, the feature point means a point where the image or the site map is easily distinguished, such as an edge point of an object.

FIG. 5 is an example of designating a correspondence relationship between minutiae points between a site map and a camera image. At this time, the site map is a plan view when the observation area is looked down vertically, and the minutiae in the image are limited to the minutiae on the ground.

1, if the overhead camera 101 or 102 is installed so as to be perfectly perpendicular to the ground, the mapping relationship between the ground portion of the camera image and the site map is represented by Equation 1 as shown in Equation (1) Lt; / RTI >

[Equation 1]

는 카메라 영상의 지면부에 속한 픽셀 좌표이고,

는

와 대응하는 사이트 맵의 포인트 좌표이고,

는

와

사이의 매핑 관계를 정의하는 변환 파라미터들이다. 상기 <수학식 1>에서 한 개의 특징점 대응쌍이 주어지면 미지수

에 대한 2개의 방정식을 얻을 수 있다. 따라서 두 개의 특징점 대응쌍이 주어지면 총 4개의 연립 방정식으로부터 변환 파라미터

를 구할 수 있다.In Equation (1)

Is a pixel coordinate belonging to the ground portion of the camera image,

The

Is the point coordinate of the corresponding site map,

The

Wow

Lt; / RTI &gt; are transformation parameters that define the mapping relationship between &lt; RTI ID = 0.0 &gt; If one feature point corresponding pair is given in Equation (1) above,

Two equations can be obtained for. Therefore, given two pairs of feature points, the transformation parameters

Can be obtained.

1, if the overhead camera 101 or 102 is not perpendicular to the ground, the mapping relationship between the ground portion of the camera image and the site map is expressed by Projective Transformation as shown in Equation (2) below .

&Quot; (2) &quot;

는 카메라 영상의 지면부에 속한 픽셀 좌표이고,

는

와 대응하는 사이트 맵의 포인트 좌표이고,

는

와

사이의 매핑 관계를 정의하는 변환 파라미터들이고,

는 0이 아닌 임의의 값을 갖는 스케일 값이다. 상기 <수학식 2>에서 한 개의 특징점 대응쌍이 주어지면 미지수

로 구성된 2개의 방정식을 얻을 수 있다. 따라서 네 개의 특징점 대응쌍이 주어지면 총 8개의 연립 방정식으로부터 변환 파라미터

를 구할 수 있다. 주어진 네 개 이상의 특징점 대응쌍으로부터 상기 변환 파리미터

를 구하는 자세한 방법은 “Multiple View Geometry in Computer Vision (Cambridge University Press)”를 참조하라.In Equation (2)

Is a pixel coordinate belonging to the ground portion of the camera image,

The

Is the point coordinate of the corresponding site map,

The

Wow

Lt; RTI ID = 0.0 &gt; a &lt; / RTI &gt; mapping relationship,

Is a scale value having an arbitrary non-zero value. If one feature point corresponding pair is given in Equation (2) above,

Two equations can be obtained. Therefore, given four pairs of minutiae points, the transformation parameters

Can be obtained. From the given four or more feature point corresponding pairs,

See "Multiple View Geometry in Computer Vision (Cambridge University Press)".

If the overhead camera 101 or 102 of FIG. 1 is equipped with a wide-angle lens, the image obtained by the camera is distorted by a curved portion corresponding to a straight line, thereby causing a lens distortion phenomenon. When there is a lens distortion in the camera image, the conversion relation between the pixel coordinates of the ground portion of the camera image and the corresponding point coordinates on the site map no longer satisfies Equation (1) or Equation (2). Therefore, lens distortion must first be removed from the camera image before applying Equation (1) or Equation (2). The correspondence between the pixel coordinates of the image with lens distortion and the pixel coordinates of the image from which the lens distortion is eliminated is usually given by Equation (3).

&Quot; (3) &quot;

는 렌즈 왜곡이 있는 영상의 픽셀 좌표,

는 렌즈 왜곡이 제거된 영상에서

에 대응하는 픽셀 좌표,

은 영상의 중심과

사이의 거리,

는 방사 왜곡 계수(Radial Distortion Coefficients)이다. 따라서 렌즈의 방사 왜곡 계수

의 값을 알면 렌즈 왜곡이 있는 영상으로부터 렌즈 왜곡이 제거된 영상을 얻을 수 있다. 상기 방사 왜곡 계수

의 값을 측정하는 방법들이 이미 공개되어 있는데, 예를 들면 “Learning OpenCV (O’Reilly Media Inc.)”를 참조하라. 도 6은 렌즈 왜곡이 있는 영상으로부터 렌즈 왜곡을 제거한 영상을 얻은 예이다.
In Equation (3)

The pixel coordinates of the image having the lens distortion,

In the image with the lens distortion removed

The pixel coordinates corresponding to < RTI ID =

The center of the image

The distance between them,

Is the Radial Distortion Coefficients. Therefore, the coefficient of radial distortion

It is possible to obtain an image in which the lens distortion is removed from the image having the lens distortion. The radial distortion coefficient

Are already available, for example, see "Learning OpenCV (O'Reilly Media Inc.)". Fig. 6 is an example of obtaining an image in which lens distortion is removed from an image having lens distortion.

101, 102: Overhead camera
200: Object tracking device
201: image-based local object tracking unit
202: Sitemap-based global object tracking unit
203: correspondence relation setting unit between the camera image and the site map
204: Real-time site map display unit
205: object meta data storage unit
206: Object Metadata Search / Playback Unit

Claims (16)

An image-based local object tracking unit for tracking an object and generating a local object tracking result when a camera image is received from each of a plurality of cameras installed toward the ground such that a part of the ground is overlapped at a predetermined height from the ground surface of the surveillance site; ;
A coordinate conversion unit between a camera image and a site map for converting pixel coordinates of each camera image received from the plurality of cameras into point coordinates of the site map;
A global object tracking unit for generating a global object tracking result using the local object tracking result and the point coordinates of the local object on the site map; And
And a real-time site map display unit for displaying the position of the global object being tracked on the site map in real time using the global object tracking result
An object tracking device based on multiple overhead cameras and sitemaps.
The method according to claim 1,
The site map-based global object tracking unit
The location coordinates of the specific area object currently being tracked are converted into point coordinates on the site map and whether a global object having the specific area object as a child object is present among the global objects being tracked is checked doing
An object tracking device based on multiple overhead cameras and sitemaps.
3. The method of claim 2,
The site map-based global object tracking unit
If there is no global object having the specific region object as a child object, a distance between the site map coordinates of the center of the global object and the site map coordinates of the center of the specific region object is obtained for each global object being tracked And determining whether the global object is found by checking whether the distance is less than a predetermined amount
An object tracking device based on multiple overhead cameras and sitemaps.
The method of claim 3,
The site map-based global object tracking unit
If it is determined that a global object corresponding to the specific region object is found, and adds the specific region object as a child object of the global object
An object tracking device based on multiple overhead cameras and sitemaps.
5. The method of claim 4,
The site map-based global object tracking unit
If it is determined that the global object corresponding to the specific region object is not found, then a new global object is created and the specific region object is added as a child object of the new global object To
An object tracking device based on multiple overhead cameras and sitemaps.
The method according to any one of claims 2 to 5, wherein the site map-based global object tracking unit
And removing a global object that does not have any child objects among the currently tracked global objects from the global object tracking list
An object tracking device based on multiple overhead cameras and sitemaps.
The method according to claim 1,
The coordinate transformation unit between the camera image and the site map
A mapping relationship between each of the camera images and the site map is set by a user so that pixel coordinates of the camera image are converted into point coordinates of the site map Characterized in that
An object tracking device based on multiple overhead cameras and sitemaps.
8. The method of claim 7,
The mapping relationship between the camera image and the site map is
When the plurality of cameras are perpendicular to the paper surface, the following formula (1)
[Equation 1]

Here, (x, y) is the pixel coordinate of the ground, (X, Y) is the point coordinate of the site map corresponding to (x, y)

Is a conversion parameter defining a mapping relationship between (x, y) and (X, Y), and if the plurality of cameras are not perpendicular to the ground,
&Quot; (2) &quot;

Here, (x, y) is the pixel coordinate of the ground, (X, Y) is the point coordinate of the site map corresponding to (x, y) A transformation parameter that defines a mapping relationship,

Is a scale value having an arbitrary non-zero value,
A plurality of equations for the conversion parameters are generated in the equations (1) and (2) using the corresponding pairs of the plurality of characteristic points, and the conversion parameters are calculated from the plurality of equations
An object tracking device based on multiple overhead cameras and sitemaps.
A method for tracking an object based on a plurality of overhead cameras and a site map executed in an object tracking device,
Receiving a camera image from each of a plurality of cameras installed toward the ground so that a part of the ground surface is photographed so as to overlap with the ground surface at a predetermined height from the ground;
Tracking an object in each of the camera images to generate a local object tracking result;
Converting pixel coordinates of each camera image into point coordinates of a site map;
Generating a global object tracking result using the local object tracking result and the point coordinates of the local object on the site map; And
And displaying the location of the global object being tracked on the site map using the global object tracking result
An object tracking method based on multiple overhead cameras and sitemaps.
10. The method of claim 9,
The step of generating the global object tracking result using the local object tracking result and the point coordinates of the local object on the site map
Converting position coordinates in a current frame of a specific local object currently being tracked into point coordinates on a site map; And
Checking whether there is a global object having the specific region object as a child object among the global objects currently being tracked
An object tracking method based on multiple overhead cameras and sitemaps.
11. The method of claim 10,
Checking whether there is a global object having the specific region object as a child object among the currently tracked global objects
If there is no global object having the specific region object as a child object, a distance between the site map coordinates of the center of the global object and the site map coordinates of the center of the specific region object is obtained for each global object being tracked step; And
And determining whether the global object is found by checking whether the distance is less than a predetermined amount
An object tracking method based on multiple overhead cameras and sitemaps.
12. The method of claim 11,
Determining whether the distance is less than a predetermined amount and determining whether the global object is found
Determining that a global object corresponding to the specific region object is found if the global object having the distance is less than a predetermined amount;
And adding the specific region object as a child object of the global object
An object tracking method based on multiple overhead cameras and sitemaps.
12. The method of claim 11,
Determining whether the distance is less than a predetermined amount and determining whether the global object is found
Determining that the global object corresponding to the specific region object is not found if the global object having the distance less than a predetermined amount does not exist;
And creating the new global object and adding the specific local object as a child object of the new global object
An object tracking method based on multiple overhead cameras and sitemaps.
14. The method according to any one of claims 10 to 13,
And removing a global object that does not have any child objects among the currently tracked global objects from the global object tracking list
An object tracking method based on multiple overhead cameras and sitemaps.
9. The method of claim 8,
The step of converting pixel coordinates of each camera image into point coordinates of a site map
A mapping relationship between each of the camera images and the site map is set by a user so that pixel coordinates of the camera image are converted into point coordinates of the site map Characterized in that it comprises the steps of:
An object tracking method based on multiple overhead cameras and sitemaps.
16. The method of claim 15,
The mapping relationship between the camera image and the site map is
When the plurality of cameras are perpendicular to the paper surface, the following formula (1)
[Equation 1]

Here, (x, y) is the pixel coordinate of the ground, (X, Y) is the point coordinate of the site map corresponding to (x, y)

Is a conversion parameter defining a mapping relationship between (x, y) and (X, Y), and if the plurality of cameras are not perpendicular to the ground,
&Quot; (2) &quot;

Here, (x, y) is the pixel coordinate of the ground, (X, Y) is the point coordinate of the site map corresponding to (x, y) A transformation parameter that defines a mapping relationship,

Is a scale value having an arbitrary non-zero value,
A plurality of equations for the conversion parameters are generated in the equations (1) and (2) using the corresponding pairs of the plurality of characteristic points, and the conversion parameters are calculated from the plurality of equations
An object tracking method based on multiple overhead cameras and sitemaps.

Images