WO2012158017A1

WO2012158017A1 - Method and system for multiple objects tracking and display

Info

Publication number: WO2012158017A1
Application number: PCT/MY2012/000096
Authority: WO
Inventors: Ching Hau Chan; Teck Liong CHOONG
Original assignee: Mimos Bhd
Current assignee: Mimos Bhd
Priority date: 2011-05-13
Filing date: 2012-04-30
Publication date: 2012-11-22
Anticipated expiration: 2013-11-13
Also published as: MY162916A

Abstract

The present invention relates generally to a method and system for multiple objects tracking and display wherein said methodology comprising steps of determining location, changing view, tracking object of interest and displaying image. Said system is configured to enable tracking of multiple locations and object of interest at the same time by controlling said image capturing means to cover all user-defined areas by switching the field of view between the areas.

Description

METHOD AND SYSTEM FOR MULTIPLE OBJECTS TRACKING AND

DISPLAY

TECHNICAL FIELD OF THE INVENTION

The present invention relates generally to a method and system for multiple objects tracking and display wherein said methodology comprising steps of determining location, changing view, tracking object of interest and displaying image. Said system is configured to enable tracking of multiple locations and object of interest at the same time by controlling said image capturing means to cover all user-defined areas by switching the field of view between the areas. BACKGROUND OF THE INVENTION

Object detection and tracking is important in many applications. For example, video surveillance system has been widely used in various occasions such as bank surveillance and public surveillance. Said video surveillance system can momentarily track an object such as character or a vehicle and thus play a very important role in the vision-based object tracking system. At present, there are different types of object tracking system. Among those, image capturing device such as pan-tilt-zoom (PTZ) camera is the common device being used to capture images of an area as well as to track activity over a wide area in various surveillance systems. The advantages of using PTZ camera is its capability to change its field of view by panning, tilting and zooming so that its field of view can be adjusted to cover either a large area or a specific object of interest in detail. The typical usage of PTZ cameras is to scan a large area by manually zooming out and then pan (shown in FIG. 1A), tilt (shown in FIG. IB) and zoom (shown in FIG. 1C) into a specific object to see more details such as faces and car license plates.

However, there are some shortcomings of using PTZ cameras such as not able to track multiple objects of interest at the same time if it is not in its current field of view at all times. In addition, adjusting to concentrate on a region of interest necessarily requires a PTZ camera to adjust its view to the region of interest and lose sight of the overall area. Moreover, if there is more than one objects of interest detected at one time, it is difficult for the user to manually control the PTZ camera and follow both the objects of interests fast enough so that not to lose sight of the other objects of interest. This constraint is exacerbated if more objects of interest are involved.

The current solution to overcome the above shortcomings is to automate the PTZ control. A typical approach to automate the control is by using a plurality of cameras as shown in FIG. 2. Said plurality of PTZ cameras are situated to cover for each other's occlusion areas. A viewfinder camera which is usually has a wide field of view for more coverage is used to capture images of the area, which is then processed to detect objects of interest. The results of the detection are then used to drive a slave PTZ camera to zoom in at the object of interest. Nevertheless, the disadvantage of this approach is that it requires redundant cameras to cover the same area.

Another approach without the use of redundant cameras is to process the images produced by the PTZ camera to automatically control the camera in small increments to follow an object and ensure the object remains inside its field of view as shown in FIG. 3. The disadvantage of this approach is that the PTZ camera can only process images in its current field of view. As a result it cannot follow objects that are just outside of its view even if they are in the same area.

It would hence be extremely advantageous if the above shortcoming is alleviated by having a multiple object tracking system which is able to view a wide area within an area or environment as well as to monitor said wide area. It is also be desirable to have the ability to focus and to capture detailed information on objects or events of interest within said area or environment by having the capabilities of simultaneous tracking of multiple objects.

The present invention overcome the above shortcoming by automatically controlling the image capturing device to cover all user- defined areas by rapidly switching the field of view between the areas as well as to detect and to follow objects of interests that appear within the user-defined areas.

Such capabilities are useful for a variety of purposes, including but not limited to surveillance for security applications, robotics, video analytics to assist the monitoring security personnel and so on. SUMMARY OF THE INVENTION

Accordingly, it is the primary aim of the present invention to provide a method and system for multiple objects tracking and display which is able to track multiple locations and objects of interest at the same time.

It is yet another object of the present invention to provide a method and system for multiple objects tracking and display which is able to use a single image capturing means or more while still being able to follow objects that are outside of its current field of view. It is yet another object of the present invention to provide a method and system for multiple objects tracking and display which is able to control the image capturing means automatically to cover all user-defined areas by rapidly switching the field of view between the areas.

Other and further objects of the invention will become apparent with an understanding of the following detailed description of the invention or upon employment of the invention in practice.

In a preferred embodiment a method for multiple object tracking and display, comprising the steps of: i. determining one or more locations as preferred locations; characterized in that said method for multiple object tracking and display further comprising the following steps after Step i: ii. changing the field of view of at least one image capturing means to the preferred locations to capture the sensor data; iii. tracking one or more objects of interest from the captured sensor data; iv. displaying image from the field of view of said image capturing means based on number of preferred locations and objects of interest; wherein said steps are typically being performed in said computing means which is capable of being interfacing with said image capturing means; wherein said step of determining one or more locations as preferred locations can be executed at any stage depending on the user's requisite.

According to another embodiment of the present invention there is provided,

A system for multiple object tracking and display comprising at least one image capturing means; at least one computing means; at least one display means; at least one user interface control means; characterized in that said system is configured to enable tracking of multiple locations and object of interest at the same time by controlling said image capturing means to cover all user-defined areas by switching the field of view between the areas. BRIEF DESCRIPTION OF THE DRAWINGS

Other aspect of the present invention and their advantages will be discerned after studying the Detailed Description in conjunction with the accompanying drawings in which:

FIG. 1A shows a diagram of PTZ camera in pan mode.

FIG. IB shows a diagram of PTZ camera in tilt mode.

FIG. 1C shows a diagram of PTZ camera in zoom mode.

FIG. 2 shows a diagram of a multi camera object tracking system.

FIG. 3 shows a diagram of a single camera automated object tracking system.

FIG. 4 shows a block diagram of multiple objects tracking and display system of the present invention.

FIG. 5 shows a flow chart of multiple objects tracking and display system of the present invention.

FIG. 6 shows a flow chart on the process of determining one or more locations as preferred locations. FIG. 7 shows a block diagram on the process of determining location.

FIG. 8 shows a diagram of memory stacks and buffers.

FIG. 9 shows a flow chart on the process of changing view

FIG. 10 shows a diagram on the process of changing view.

FIG. 11 shows a flow chart on the process of tracking object of interest.

FIG. 12 shows the diagrams on the process of tracking object with different conditions.

FIG. 13 shows a flow chart on the process of displaying image.

FIG. 14 shows an example of the display arrangement (2x2) based on the display image buffer.

FIG. 15 shows an example of the display arrangement (3x2) based on the display image buffer. DETAILED DESCRIPTION OF THE DRAWINGS

In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be understood by those or ordinary skill in the art that the invention may be practised without these specific details. In other instances, well known methods, procedures and/ or components have not been described in detail so as not to obscure the invention. The invention will be more clearly understood from the following description of the embodiments thereof, given by way of example only with reference to the accompanying drawings which are not drawn to scale.

Referring now to FIG. 4, there is shown a block diagram of multiple objects tracking and display system of the present invention which comprises of at least one image capturing means (401), at least one computing means

(402), at least one display means (403) and at least one user interface control means (404). Said image capturing means (401) includes a pan-tilt-zoom camera (PTZ camera) or any suitable motorized-image capturing means which is installed to detect and to track multiple locations (405) such as entry or exit locations and other objects of interest (406). Said image capturing means (401) is connected to said computing means (402) by connecting wires or wireless connections (407) for data exchange by said display means (403) and said user interface control means (404). Said data exchange includes input such as pan-tilt-zoom data, location, image and output such as displaying images of the detected objects of interest. Referring now to FIG. 5, there is shown a flow chart of multiple objects tracking and display system of the present invention. Said multiple objects tracking system typically comprises the steps of determining location (510), changing view (530), tracking object of interest (550) and displaying image (570). Said steps are typically being performed in said computing means (402), wherein any computing means can be used to perform said steps as long as it is capable of being interfacing with said image capturing means (401). While the steps are presented and could be executed in sequence, the present invention is not subject to limitation and said steps could be executed in other alternative sequences. For example, the step of determining location (510) could be executed at any time depending on the user's requisite. While the steps of changing view (530), tracking object of interest (550) and displaying image (570) could be executed simultaneously as a multi-threaded application. The capability of having simultaneous execution would ensure that the multiple objects tracking system of the present invention achieves the highest performance in terms of speed.

Referring now to FIG. 6, there is shown a flow chart on the process of determining one or more locations (510) as preferred locations based on the user's request. The process begins with the initialization of said image capturing means (511) to its default settings. Said image capturing means (401) includes a PTZ camera or any other suitable motorized-image capturing means. Said PTZ camera can be controlled manually by the user (512) to focus to the preferred location by using of at least one user interface control means (404) such as joystick. As shown in FIG. 7, the process of determining location is illustrated in block diagram. For example, Entry/ Exit location No.l, Entry/Exit location No.2 and Entry/Exit location No. 3 are designated as preferred locations (405). The user is firstly manually controls said PTZ camera to focus at Entry/ Exit location No.l. When the pan-tilt-zoom (PTZ) data of said PTZ camera are obtained (513), said PTZ data are retrieved and stored in the Entry/ Exit location stack (514), as shown in FIG. 8 which is the diagram of memory stacks and buffers. Said diagram of memory stacks and buffers, which is a place to store the PTZ motor position or coordinate data as well as image sensor data is located in said computing means (402). Said diagram of memory stacks and buffers comprises of components such as Entry/ Exit location stack (810) with its image buffer (812), object location stack (820) with its image buffer (822) as well as display image buffer (832). Alternatively, the components in said diagram of memory stacks and buffers can be in other configuration, such as Entry / Exit and Object stack being combined as one component.

Said process of determining location (510) is then repeated by having the user manually controls the PTZ camera to focus at Entry/Exit location No.2, followed by Entry/ Exit location No.3 until all said obtained PTZ data of respective locations are retrieved and stored in said location stack. Preferably, prior to the commencing of the process of determining locations, there should not be any person present in the area or the room and is not covered in the PTZ camera's field of view. This is to ensure that all objects in the area or the room is being detected and is being tracked properly. At the end of the process of determining location, all the PTZ data for the preferred locations are stored and hence all the preferred locations (405) are determined.

Referring now to FIG. 9, FIG. 9 is a flow chart showing the process of changing view (530), further substantiated by FIG. 10, showing the diagram of the changing view process. Said process of changing view involves changing said PTZ camera's field of view to the preferred locations to capture sensor data. Since the preferred locations are the non-moving object, this is carried out by continuously changing the field of view according to PTZ data of the non-moving object stored in memory (531), acquiring imaging sensor data after each change (532) and store said sensor data into memory (533). Furthermore, said process of changing view also involves continuously changing the field of view according to PTZ data of moving object data stored in memory (534), acquiring said sensor data after each change (535) and store said sensor data into memory (536). The steps of capturing images of the determined object of interest's location by using of said PTZ camera is typically being carried out in high speed, which is generally in 20 frames per second. All the captured images are then stored in the image buffer (812, 822) as shown in FIG. 8.

Referring now to FIG. 11, there is shown a flow chart on the process of tracking object of interest (550). The process begins by having at least one computing means to retrieve the stored sensor data or images of non-moving object (551) from image buffer (812) as shown in FIG. 8 and to perform object tracking (552) on said stored sensor data. Said stored sensor data or images of non-moving object are then sent (553) to Display Image Buffer (832) or display memory when said object tracking process is completed. At this point, the method takes into account the actual pan-tilt-zoom data of the object by converting the (x, y) coordinates with offset (554) and then store the pan- tilt-zoom data (555) in Object Location stack (820). The zoom factor of the object is determined by the average between two closest points. Said process of tracking object of interest continues by having said computing means to retrieve the stored sensor data or images of moving object (556) from image buffer (822) and then followed by detecting of existing object within a location of interest (557) on said stored sensor data. Said stored sensor data or images of moving object are then sent (558) to Display Image Buffer (832) or display memory when said process of detecting object is completed. At this point, the method takes into account the actual pan-tilt- zoom data of the object by converting the (x, y) coordinates with offset (559) and then store the pan-tilt-zoom data (560) in Object Location stack (820).

Generally, the PTZ location stacks are rearranged for efficient camera movement based on location using conventional nearest neighbour computation method. FIG. 12 shows the diagrams of object tracking process with different conditions. The percentage shown in the diagrams are referring to the portion of cycle time allocated for the PTZ camera to focus on the respective Entry/ Exit location and Object Location. Said process of tracking object of interest enables said PTZ camera to change its field of view to visually follow multiple moving objects within the location of interest while maintaining views of other location of interests and objects.

Referring now to FIG. 13, there is shown a flow chart on the process of displaying image (570) from the PTZ camera's field of view based on number of preferred locations and objects of interest. The process begins by having at least one computing means to acquire the total number of images to be displayed from Display Image buffer (832) prior to reading the actual images data from the Display Image buffer. Based on the number of images, said computing means will then generate a dynamic user interface display arrangement (571) in order to fit said at least one display means (403) such as monitor. By having said display arrangement, the individual images obtained (572) from the Display Image buffer (832) are then displayed (573) onto said display means (403). In addition, said display arrangement can be changed due to any addition, rearrangement or deletion of the display arrangement in memory according to the number of moving and non- moving objects from detection of existing object within location of interest. Therefore, said process of displaying image would ensure that there is minimum waste of space on the display. For example, when there are four objects, which are three user-set locations and one detected object, the display arrangement changes to a 2x2 matrix grid to show all four objects as shown in FIG.14. Further on during the operation of the system, another object might appear in which the arrangement will then change to 3x2 matrix grid or other non-uniform arrangement as shown in FIG. 15. Said display arrangement is in such that the use of the display's space is maximized and the output of the PTZ camera is rearranged to look like a system with multiple static cameras. In addition, when said display arrangement changes, it would alert the user that some objects may have entered or exited an area.

Alternatively, the process of updating the preferred location can be conducted anytime when there is requirement from the user to add, to rearrange or to delete locations of the non-moving objects of interests from said stored sensor data and PTZ data in memory. Said addition or deletion of locations of the objects of interest is also based on the detection of moving object from said detection of existing object within the location of interest. Whilst said rearrangement of locations of the objects of interest is based on the calculation of the PTZ data of the detection of existing object within the location of interest to obtain the efficiency for changing the field of view of the said PTZ camera.

The present invention is targeted to be a solution that could be adapted between image capturing means and software application. The present invention also extends the normal use of image capturing means such as PTZ cameras to cover more areas and to track more objects concurrently. Therefore, said PTZ camera can be any conventional PTZ camera while the application can be any software that intends to use the images and object tracking information captured from this solution.

While the present invention of multiple objects tracking and display system is focusing on surveillance application, the present invention is not restricted to this but may alternatively be applied to other areas such as webcam applications, robotics, business video analytics and the like.

While the preferred embodiment of the present invention and its advantages has been disclosed in the above Detailed Description, the invention is not limited thereto but only by the scope of the appended claim.

Claims

WHAT IS CLAIMED IS:

1. A method for multiple object tracking and display, comprising the steps of: i. determining one or more locations (510) as preferred locations; characterized in that said method for multiple object tracking and display further comprising the following steps after Step i: ii. changing the field of view (530) of at least one image capturing means (401) to the preferred locations to capture the sensor data; iii. tracking one or more objects of interest (550) from the captured sensor data; iv. displaying image (570) from the field of view of said image capturing means (401) based on number of preferred locations (405) and objects of interest (406); wherein said steps are typically being performed in said computing means (402) which is capable of being interfacing with said image capturing means (401); wherein said step of determining one or more locations (510) as preferred locations can be executed at any stage depending on the user's requisite.

2. A method for multiple object tracking and display as claimed in Claim 1 wherein said steps of changing the field of view (530), tracking one or more objects of interest (550) and displaying image (570) could be executed simultaneously as a multi-threaded application.

3. A method for multiple object tracking and display as claimed in Claim 1 wherein said step of determining one or more locations (510) as preferred locations performing the following steps: i. initializing of said at least one image capturing means (511); ii. controlling said image capturing means to preferred locations to obtained pan- tilt-zoom data (512); iii. retrieving said pan-tilt-zoom data from said image capturing means (513); iv. storing of said pan-tilt- zoom data in memory (514).

4. A method for multiple object tracking and display as claimed in Claim 1 wherein said steps of changing the field of view (530) performing the following steps: i. changing said field of view continuously according to pan-tilt- zoom data of non-moving object stored in memory (531); ii. acquiring imaging sensor data after each change (532); iii. storing said sensor data into memory (533).

5. A method for multiple object tracking and display as claimed in Claim 1 wherein said steps of changing the field of view (530) performing the following steps: i. changing said field of view continuously according to pan-tilt- zoom data of moving object stored in memory (534); ii. acquiring said sensor data after each change (535); iii. storing said sensor data into memory (536).

6. A method for multiple object tracking and display as claimed in Claim 1 wherein said steps of tracking one or more objects of interest (550) performing the following steps: i. retrieving said stored sensor data or images of non-moving object from memory (551); ii. performing object tracking on said stored sensor data (552); iii. sending said stored sensor data of non-moving object to display memory (553); iv. calculating (554) and storing (555) said pan-tilt-zoom data into memory.

7. A method for multiple object tracking and display as claimed in Claim 1 wherein said steps of tracking one or more objects of interest (550) performing the following steps; i. retrieving said stored sensor data or images of moving object from memory (556); ii. detecting existing object within a location of interest on said stored sensor data (557); iii. sending said stored sensor data of moving object to display memory (558); iv. calculating (559) and storing (560) said pan-tilt-zoom data into memory.

8. A method for multiple object tracking and display as claimed in Claim 1 wherein said steps of tracking one or more objects of interest (550) enables said image capturing means (401) to change its field of view to visually follow multiple moving objects within the location of interest while maintaining views of other location of interests and objects.

9. A method for multiple object tracking and display as claimed in Claim 1 wherein said steps of displaying image (570) performing the following steps: i. changing the display arrangement based on said stored sensor data or images in memory (571); ii. acquiring said stored sensor data from memory (572); iii. displaying said stored sensor data onto at least one display means (573).

10. A method for multiple object tracking and display as claimed in Claim 9 wherein said display arrangement can be changed due to any addition, rearrangement or deletion of the display arrangement in memory according to the number of moving and non-moving objects from detection of existing object within location of interest.

11. A method for multiple object tracking and display as claimed in Claim 1 further comprising the step of updating the preferred location which can be conducted at any stage as long as there is requirement from the user to add, to rearrange or to delete locations of the non-moving objects of interest from said stored sensor data or pan-tilt-zoom data in memory.

12. A method for multiple object tracking and display as claimed in Claim 11 wherein said addition or deletion of locations is based on the detection of moving object from said detection of existing object within the location of interest.

13. A method for multiple object tracking and display as claimed in Claim 11 wherein said rearrangement of locations is based on the calculation of the pan-tilt-zoom data of the detection of existing object within the location of interest to obtain the efficiency for changing the field of view of the said image capturing means.

14. A system for multiple object tracking and display comprising at least one image capturing means (401); at least one computing means (402); at least one display means (403); at least one user interface control means (404); characterized in that said system is configured to enable tracking of multiple locations (405) and object of interest (406) at the same time by controlling said image capturing means (401) to cover all user- defined areas by switching the field of view between the areas.

15. A system for multiple object tracking and display as claimed in Claim 14 wherein said image capturing means (401) includes a pan- tilt-zoom camera or any suitable motorized-image capturing means.

16. A system for multiple object tracking and display as claimed in Claim 14 wherein said image capturing means (401) is connected to said computing means (402) by connecting wires or wireless connections (407) for data exchange by said display means (403) and said user interface control means (404).

17. A system for multiple object tracking and display as claimed in Claim 14 wherein said user interface control means (404) includes a joystick.

18. A system for multiple object tracking and display as claimed in Claim 14 further comprises of memory stacks and buffers which includes components such as Entry/ Exit location stack (810) with its image buffer (812), object location stack (820) with its image buffer (822) as well as display image buffer (832).

19. A system for multiple object tracking and display as claimed in Claim 18 wherein the components in said memory stacks and buffers can be in other configuration, such as Entry / Exit and Object stack being combined as one component.