WO2022025752A1 - A system and method for tracking objects in a dynamic environment from a sequence of captured images - Google Patents

A system and method for tracking objects in a dynamic environment from a sequence of captured images Download PDF

Info

Publication number
WO2022025752A1
WO2022025752A1 PCT/MY2020/050190 MY2020050190W WO2022025752A1 WO 2022025752 A1 WO2022025752 A1 WO 2022025752A1 MY 2020050190 W MY2020050190 W MY 2020050190W WO 2022025752 A1 WO2022025752 A1 WO 2022025752A1
Authority
WO
WIPO (PCT)
Prior art keywords
static
objects
detected moving
captured images
moving objects
Prior art date
Application number
PCT/MY2020/050190
Other languages
French (fr)
Inventor
Zulaikha BINTI KADIM
Khairunnisa BINTI MOHAMED JOHARI
Hock Woon Hon
Original Assignee
Mimos Berhad
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Mimos Berhad filed Critical Mimos Berhad
Publication of WO2022025752A1 publication Critical patent/WO2022025752A1/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • G06V10/757—Matching configurations of points or features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence

Definitions

  • the present invention relates to a system and method for tracking objects, more particularly, tracking objects in a dynamic environment from a sequence of captured images.
  • Video surveillance enables object monitoring through video displays of one or more areas remote from a human monitor.
  • Various exemplary applications include security surveillance of public and private areas, assembly areas such as train stations and many more.
  • a problem faced with video as a surveillance tool is that it may be manually intensive to monitor, in which it requires a user to be on constant lookout for any issues that might arise within a particular area of interest.
  • human visual attention may be ineffective, particularly for large volumes of video data. Accordingly, it is desirable to implement an automated system suitable for use in video analysis.
  • a United States patent with publication no. US9454819B1 discloses a method for static and moving object detection by employing a motion computation method based on spatio-temporal tensor formulation, a foreground and background modelling method, and a multi-cue appearance comparison method.
  • the invention in use, operates in the presence of shadows, illumination changes, dynamic background, and both stopped and removed objects.
  • it also includes performing motion segmentation and background subtraction as key steps for detecting pixel-level motion from a source of video images, in which by fusing results of the motion segmentation, background and foreground modelling, the moving and static foreground objects are able to be identified and further classified as stopped objects or background models.
  • US9224049B2 discloses a method of extracting foreground object image features from an input video via a background subtraction mask in order to determine whether a foreground object is in motion or static. According to the document, a motion analysis of the foreground object is carried out to determine the likelihood of the object in motion is either static for a threshold time period or moving in a particular direction.
  • a United States patent with publication no. US7801330B2 discloses a technique for video processing which includes receiving a video from a scene, detecting moving pixels in the video, detecting line segments or motion blocks in the video based on the detected motion block, tracking targets in the video based on the identified targets, and managing the tracked targets in the video.
  • the document also recites the method of detecting at least one static or moving identified targets in the scene, counting the detected static targets and estimating a dwell time based on the counted static targets, in which the estimated dwell time will be recorded in a log for future reference.
  • Another technology as disclosed in a United States patent with publication no. US8744123B2 recites a method for detecting objects in the foreground by generating a foreground object blob with a bounding box, further classified as a background, moving foreground, or temporally static object, wherein the temporally static object is distinguished from the background and remains static in the scene for a threshold period. Further, a static value ratio is computed as a number of pixels determined to be static by tracking in a foreground region of the background model corresponding to the tracked object bounding box over a total number of pixels of the foreground region.
  • the present invention discloses a method for tracking objects in a dynamic environment from a sequence of captured images, the method comprising: detecting one or more moving objects from the captured images and updating a background model of said captured images, by an object detection module; filtering the detected moving objects from noise and validates whether any of the detected moving objects are static objects, by a moving object tracking module; determining whether the static objects are in a static condition for a predetermined time period and updates status of the static objects accordingly, by a static object tracking module; and applying a smoothness test if the static objects are static for the predetermined time period to categorize the different static objects, by an event recognition module.
  • the method further comprising the steps of extracting properties of the detected moving objects to be used as inputs for updating the background model of said captured images, by the object detection module, and extracting background feature points based on the updated background model of said captured images, by the object detection module.
  • the background model is updated by the steps of examining whether a tracker exists for the static objects and the detected moving objects in the captured images, and updating a subsequent background model by copying an initial background model to said subsequent background model if the trackers for the static objects and the detected moving objects do not exist.
  • the method further comprises the steps of executing a tracking algorithm for updating statuses of the detected moving objects according to the properties of the detected moving objects obtained from the object detection module, analysing an initial movement status of the detected moving objects and remove trackers for the detected moving objects that are considered noise, and removing the trackers for the detected moving objects that are validated as static objects to be further tracked by the static object tracking module, by the moving object tracking module.
  • the initial movement status of the detected moving objects is analysed by the steps of estimating a movement status of the detected moving objects in a current captured image, accumulating a number of captured images whereby the detected moving object is not moving starting from the initial detection of the moving object in the captured images, and comparing the accumulated number of captured images to a threshold to determine the movement status of the detected moving objects.
  • the movement status of the detected moving objects is estimated by the steps of extracting significant feature points within a detected moving object area in the current captured image, obtaining corresponding matching feature points in the detected moving object area in a previous captured image, computing a magnitude of each feature point in the detected moving object areas for the previous and the current captured images to obtain an average magnitude of the detected moving objects, and comparing the average magnitude of the detected moving objects to a static threshold to determine whether the detected moving objects are moving or not in the current captured image.
  • the detected moving objects are validated as static objects by the steps of obtaining the movement status of the detected moving objects in the current captured image, examining whether the detected moving objects are static or have previously moved based on the initial movement status of the detected moving objects, appending the detected moving objects to a list of validated static objects in the current captured image to be further tracked in the static object tracking module if the detected moving objects are static or have previously moved, and removing the detected moving object trackers of the validated static objects.
  • the method further comprises the steps of comparing a list of existing static object trackers to the validated static objects in the current captured image initializing new trackers to the validated static objects with no existing trackers, registering the current captured image as a reference image to be used in the static object tracking module, and employing a subroutine with at least two computational methods for determining whether the static objects remain static for a predetermined sequence of captured images, by the static object tracking module.
  • the subroutine employed by the static object tracking module comprises the steps of determining an existence of the static objects for each static object tracker using at least two computational methods, calculating the number of captured images in which the static objects are missing from the captured images, comparing the number of captured images to a predetermined missing threshold, removing the trackers for the static objects if the number of captured images exceed the predetermined missing threshold, and updating the status of the static objects accordingly if the static objects remains static in the captured images.
  • a first of the at least two computational method applied by the subroutine comprises the steps of comparing similarities between feature points extracted from the current captured image and the background feature points obtained from the object detection module, and computing a similarity ratio of the feature points extracted from the current captured image and the background feature points.
  • a second of the at least two computational method applied by the subroutine comprises the steps of comparing similarities between the feature points extracted from the current captured image and the registered reference image obtained in the static object tracking module, computing a second similarity ratio of the feature points extracted from the current captured image and registered reference image, computing a final ratio between the first and second similarity ratios, and comparing the computed final ratio to a predetermined threshold to determine whether the static objects remains static in the current captured image or the static objects have already moved.
  • the smoothness test is applied by the steps of applying a low pass filter repetitively on the current captured image to obtain a smoothed image, computing a mean difference between the smoothed image and the current captured image within a static object area, and comparing the mean difference to a non-smooth threshold to determine if the static objects are smooth or not smooth.
  • a system for tracking objects in a dynamic environment from a sequence of captured images comprising of: an object detection module, configured to detect one or more moving objects from the captured images and updating a background model of said captured images; a moving object tracking module, configured to filter the detected moving objects from noise and validates whether any of the detected moving objects are static objects; a static object tracking module, configured to determine whether the static objects are in a static condition for a predetermined time period and updates status of the static object accordingly; and an event recognition module, configured to apply a smoothness test if the static objects are static for the predetermined time period to categorize the different static objects.
  • FIG. 1 is a block diagram of a system for tracking objects in a dynamic environment from a sequence of captured images, according to the present invention.
  • FIG. 2 is a flow chart illustrating a preferred embodiment for a method for tracking objects in a dynamic environment from a sequence of captured images based on the above-mentioned system.
  • FIG. 3 is a flow chart illustrating a preferred embodiment for detecting moving objects in the captured images.
  • FIG. 4 is a flow chart illustrating an exemplary embodiment for updating a background image model based on inputs from the detected moving objects.
  • FIG. 5 is a flow chart illustrating a preferred embodiment for tracking and validating the detected moving objects in the captured images.
  • FIG. 6 is a flow chart illustrating an exemplary embodiment for analysing an initial movement status of the detected moving objects.
  • FIG. 7 is a flow chart illustrating an exemplary embodiment for estimating a movement status of the detected moving objects in a current captured image.
  • FIG. 8 is a flow chart illustrating a preferred embodiment for validating the detected moving objects to be further tracked in a static object tracking module.
  • FIG. 9 is a flow chart illustrating a preferred embodiment for tracking static objects in in the captured images.
  • FIG. 10 is a flow chart illustrating a preferred embodiment of a subroutine in the static object tracking module to examine and update existing static object trackers.
  • FIG. 11 is a flow chart illustrating an exemplary embodiment of a first method employed by the subroutine in the static object tracking module.
  • FIG. 12 is a flow chart illustrating an exemplary embodiment of a second method employed by the subroutine in the static object tracking module.
  • FIG. 13 is a flow chart illustrating a preferred embodiment for determining a static object smoothness based on inputs from the static object tracking module.
  • FIG. 14 is a flow chart illustrating an exemplary embodiment of a smoothness test to categorize different static objects.
  • the computer program instructions may be embodied in one or more computer readable mediums having computer readable program code embodied thereon. These computer program instructions may then direct a computer or a programmable data processing apparatus to function in a specific manner, such that the instructions stored in the computer readable medium may produce an article of manufacture which includes instruction means for implementing the functions in the flowchart or block diagrams to be discussed herein.
  • the computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions that are executed on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or block diagrams.
  • the video acquisition module 1 may include a digital camera, digital video recorder, a network video camera, a closed-circuit television (CCTV) camera, or any other mediums suitable for capturing images in a sequence.
  • CCTV closed-circuit television
  • the captured images 11 may be transferred wirelessly over a communication network to an image processing module 2, wherein the communication network may be a wireless network connection established via a wireless protocol cloud such as a Long-Term Evolution (LTE) cloud, Code Division Multiple Access (CDMA) and its derivatives, Enhanced Data Rates for GSM Evolution (EDGE), 3G protocol, High Speed Packet Access (HSPA), 4G protocol, 5G protocol and the likes, in accordance to the advancement of wireless technology with time.
  • LTE Long-Term Evolution
  • CDMA Code Division Multiple Access
  • EDGE Enhanced Data Rates for GSM Evolution
  • 3G protocol Third Generation
  • High Speed Packet Access (HSPA) High Speed Packet Access
  • 4G protocol High Speed Packet Access
  • 5G protocol 5G protocol and the likes
  • captured images 11 from the video acquisition module 1 may be stored in an internal memory storage device which can be installed within the video acquisition module or externally linked therewith, wherein internal or external memory storage device may include primary storage devices, such as Raw Access Memory (RAM), which is used by computer systems alike to temporarily store and retrieve data, or secondary storage devices such as Hard Disk Drives (HDD) or Solid State Drives (SSD) to store data permanently.
  • RAM Raw Access Memory
  • HDD Hard Disk Drives
  • SSD Solid State Drives
  • the plurality of modules in the image processing module 2 communicate with one another to form an interconnected communication network.
  • the image processing module 2 comprises an object detection module 3, configured to detect any moving objects 30 in the captured images by modelling a background using a combination of appearance and feature point distribution, to which the background is updated using information feedback from the respective tracking modules thereon, in which the features or properties 33 of the detected moving objects 30 may include, by way of example but not limited to, the colour, size, position, orientation and direction of motion of the detected moving objects 30.
  • the image processing module 2 comprises a moving object tracking module 4, configured to track the detected moving objects 30 and differentiate the detected moving objects 30 from noise present in the captured images 11 based on the tracking states 42 of the detected moving objects 30, which may include location 31, appearance model, tracker label and many more.
  • the moving object tracking module 4 is employed as a first level of tracking in order to track all detected moving objects 30 within a region of interest, and a next level of tracking will be subsequently triggered once the moving object tracking module 4 detects any potential static objects 50 within the region of interest by means of validating the detected moving objects 30 based on a set of predetermined conditions, subsequently feeding the information of the potential static objects 50 to the next module for further tracking.
  • the set of predetermined conditions may include the detected moving objects 30 was previously moving or static, or the detected moving objects 30 is not static and is still moving.
  • the image processing module 2 comprises a static object tracking module 5, configured to track valid static objects 50 within the region of interest which is initiated by the output from the moving object tracking module 4.
  • the static object tracking module 5 is a second level of tracking which is initiated when the detected moving objects 30 in the moving object tracking module 4 are detected as potential static objects 50 by means of validation based on the set of predetermined conditions which may include the detected moving objects 30 were previously moving and are currently not moving.
  • the static object tracking module 5 tracks the static objects 50 independently from any other detected moving objects 30.
  • static object tracking module 5 employs certain key steps to overcome this obstruction which will be discussed further herein.
  • the image processing module 2 comprises an event recognition module 6, configured to perform an event analysis and recognition by analysing the output from the static object tracking module 5 based on the condition that the tracked static objects 50 remained static for a predetermined time period.
  • the event recognition module 6 will apply a smoothness test to categorize the different static objects 50 which may include vehicles, plants, roads, trees and the likes, prior to sending out an alert 61 to an alert and display module 7, as signalling the presence of a valid static object 50.
  • the alert and display module 7 may be a user interface integrated into the computer system which notifies the user in the form of a ringing alarm or other notification means.
  • the captured images 11 are input into the object detection module 3, whereby the output of the object detection module 3 are two folds, whereby one of the output is a location 31 of the detected moving objects 30 and a second output is a background model 32 of the captured images 11.
  • the background model 32 is depicted by a fusion of captured images 11 appearance model and feature points distribution collectively.
  • the detected moving object location 31 information will then be passed to the moving object tracking module 4, which is the first level of tracking, whereby the detected moving objects 30 will be labelled with the same tracker label across successive captured images 11 until a change occurs within said captured images 11.
  • the detected moving objects 30 in the moving object tracking module 4 will be further classified, based on the tracker information, as an actual object or noise due to vehicular illumination, particularly in night conditions when noise is predominant in the captured images 11.
  • the output from the moving object tracking module 4 is the updated detected moving objects’ 30 statuses 42, which may include tracker labels, moving object location 31, updated object appearance model and the likes.
  • the output will be used to decide if the second level of tracking is necessary to be initiated.
  • each of the detected moving objects 30 from the moving object tracking module 4 will be examined and validated whether they are valid static objects 50 in the captured images 11 or not.
  • a predetermined set of conditions will be checked in order to validate the detected moving objects 30 in order to initiate the second level of tracking.
  • a list of the detected moving objects 30 found to be valid static objects 50 in the current captured image 12 is subsequently compiled and this information will be passed on to the static object tracking module 5 to be further tracked.
  • the static object tracking module 5 will track the valid static objects 50 by comparing the current captured image 12 to the background models 32 obtained from the object detection module 3.
  • the tracker for said static objects 50 will be maintained. However, when the static objects 50 start to move in a subsequent captured images 11, the tracker for the static objects will be removed accordingly.
  • the output from the static object tracking module 5 will be passed on to the event recognition module 6, in which the output includes the updated static object states 51 such as location, appearance model and the likes. Further, the output will be examined by the event recognition module 6 to detect the event of the static objects 50 remaining in the sequence of captures images 11, by way of example but not limited to, a vehicle stopping by the side of the road or the likes. If the event is confirmed, the alert 61 will be triggered and sent to the alert and display module 7.
  • FIG. 3 is a flow chart illustrating a preferred embodiment for detecting the moving object in the captured images 11 in the object detection module 3.
  • the detected moving objects 30 in the captured images 11 are detected by using any standard moving object detection algorithm, by way of example but not limited to, a mixture of Gaussian (MOG) algorithm, temporal differencing, background subtraction, optical flow, frame differencing, approximation median and the likes, each with their own advantages over each other.
  • the output from the detected moving objects 30 are the detected moving object location 31 and an initial background model 32a from the captured images 11, wherein the detected moving object location 31 will be utilised in extracting the properties 33 of the detected moving objects 30 at Step 302.
  • the properties 33 to be extracted from the detected moving objects 30 may include the object size, object centroid, object orientation, and the likes.
  • the extracted properties 33 of the detected moving objects 30 and the initial background model 32a are then utilised for updating the background model at Step 303.
  • the moving object properties 33 will be used concurrently with the updated static object states 51 obtained from the static object tracking module 5 and the initial background model 32a to be input and analysed for obtaining an updated subsequent background model 32b. Further, background feature points based on the updated subsequent background model 32b are then extracted at Step 304 to be used in the static tracking module 5.
  • FIG. 4 illustrates an exemplary embodiment for updating the background model 32 based on inputs from the properties 33 of the detected moving objects 30 and updated states 51 from the static object tracking module 5.
  • the first background model 32a is for detecting a moving object 30 in the captured images 11.
  • any static objects 50 will be absorbed into the background instead of the foreground, causing the static objects 50 to be detected as the background instead.
  • a second background image model 32b will then be updated to include any static objects 50 in the captured images 11.
  • the object detection module 3 will examine whether there are any trackers for static objects 50 and detected moving objects 30 in the captured images 11.
  • the background model 32 will not be updated. However, if no moving objects 30 and static objects 50 are detected in the current captured image 12, only then will the second background model 32b be updated by copying the first background model 32a, as illustrated by Step 402. This is to ensure that the background model 32 is not tampered with any static objects 50 in the foreground.
  • FIG. 5 illustrates a preferred embodiment for tracking and validating the detected moving objects 30 in the moving object tracking module 4.
  • the moving object tracking module 4 is employed as the first level of tracking objects in the dynamic environment from the sequence of captured images 11.
  • the location 31 and properties 33 of the detected moving objects 30 obtained from the object detection module 3 are input into the moving object tracking module 4.
  • the moving object tracking module 4 will perform the tracking by updating the states 42 of the detected moving objects according to the inputs from the object detection module 3, wherein the states 42 may include tracking labels, appearance models, tracking lifetime, and the likes.
  • the moving object tracking module 4 will analyse the initial movement status of the detected moving objects 30, wherein the initial movement status will be used to further classify whether said detected moving objects 30 are noise or not. The steps for analysing the initial movement status for the detected moving objects 30 will be further discussed in FIG. 6.
  • the moving object tracking module 4 will remove the trackers for any detected moving objects 30 that are considered noise based on the analysis of their initial movement status, in which the detected moving objects 30 that are not considered noise will be further evaluated at Step 504 for their validity as static objects 50 to be further tracked in the subsequent static object tracking module 5.
  • the moving object tracking module 4 will also remove the trackers for the detected moving objects 30 which are validated as static objects 50 to be tracked by the static object tracking module 5 which will be discussed further herein.
  • FIG. 6 illustrates an exemplary embodiment for analysing the initial movement status of the detected moving objects 30 in the moving object tracking module 4.
  • the analysis is conducted by keeping track of the detected moving objects 30 in successive captured images 11, starting from the moment the detected moving objects 30 enter the region of interest in the captured images 11.
  • the moving object tracking module 4 will estimate the movement status of the detected moving objects 30 in the current captured image 12.
  • the movement status of the detected moving objects 30 in the current captured image 12 begins by extracting significant feature points within an area containing the detected moving objects 30 at Step 701.
  • Step 702 corresponding matching feature points in previous captured images 11 will subsequently be obtained, in which the moving object tracking module 4 will compute a magnitude of each optical flow points within the detected moving object 30 area at Step 703, prior to computing an average magnitude of the optical flow points for the detected moving objects 30 at Step 704.
  • the average magnitude will then be compared to a static threshold value in order to subsequently determine the movement status of the detected moving objects 30 in the current captured image 12. If the average magnitude is above the static threshold value, the detected moving objects 30 will be classified as “moving” at Step 707, but if the average magnitude is below the static threshold value, then the detected moving objects 30 will be classified as “static” or “not moving” at Step 706. Referring back to FIG.
  • the estimated movement status of “moving” or “not moving” will be further evaluated at Step 602. If the estimated movement status is “moving”, then the detected moving objects 30 will be labelled as “previously moved” at Step 603, otherwise the estimated movement status of the detected moving objects 30 will be further evaluated in successive captured images 11, whereby a number of captured images 11 in which the detected moving objects 30 are deemed as “not moving” will be accumulated.
  • the accumulated number of captured images 11 whereby the detected moving objects 30 are “not moving”, from the first instance the moving objects were detected, are then compared to a predetermined threshold at Step 605, wherein the threshold, by way of example, may be set as 25 captured images 11, equivalent to a 1 second video duration for a 25 frames- per-second (fps) video.
  • the threshold by way of example, may be set as 25 captured images 11, equivalent to a 1 second video duration for a 25 frames- per-second (fps) video.
  • the detected moving objects 30 will be determined as “never moved”, in which said detected moving objects 30 will be further classified as noise, and the trackers for said detected moving objects 30 will be removed.
  • FIG. 8 illustrates an exemplary embodiment for validating the detected moving objects 30 as static objects 50 to be further tracked in the static object tracking module 5.
  • Step 801 both the initial movement status and the estimated movement status of the detected moving objects 30 in the current captured image 12 are obtained to be evaluated. If the detected moving objects 30 satisfy conditions of “static” or have “previously moved” at Step 802, based on their movement statuses, the detected moving objects 30 will be determined as valid static objects 50 to be further tracked in the static object tracking module 5. However, if the detected moving objects 30 are not “static” or have not “previously moved”, their trackers will be removed and will not be further tracked in the static object tracking module 5 therein, such as illustrated in Step 804.
  • the detected moving objects 30 that have satisfied the conditions of will be appended to a list of validated static objects 50 in the current captured image 12. Subsequently, the validated static objects 50 will have their detected moving objects 30 trackers removed at Step 804, and the information of the validated static objects 50 will be fed into the static object tracking module 5 for further tracking thereof.
  • FIG. 9 illustrates a preferred embodiment for tracking the static objects 50 in the captured images 11.
  • the static object tracking module 5 is responsible for tracking the static objects 50 in the region of interest and subsequently update a duration of the static objects 50 remaining in a static condition.
  • input to the static object tracking module 5 is the list of validated static objects 50 obtained from the moving object tracking module 4.
  • a list of existing static object 50 trackers will be checked against the validated static objects 50 to determine whether there is an existing tracker, as illustrated at Step 901.
  • Step 902 if the existing trackers overlap with the validated static objects 50, then no trackers will be initialized.
  • Step 903 the current captured image 12 will be registered as a reference image 904 for the validated static objects 50.
  • the tracker states 51 of the static objects 50 will be subsequently updated in a subroutine 905 employed by the static object tracking module 5, which will be discussed further in FIG. 10 to FIG. 12.
  • FIG. 10 illustrates an exemplary embodiment of the subroutine 905 from the static object tracking module 5, whereby all existing static objects 50 trackers will be examined to update their tracker states 51.
  • each static objects 50 tracker will be examined to determine the existence of the static objects 50 in the current captured image 12 by applying at least two computational methods which will be shown in FIG. 11 and FIG. 12 respectively.
  • the output of Step 1001 is the statuses of whether the static objects 50 are still in the current captured image 12 or not, which will be further evaluated at Step 1002. If the static objects 50 have disappeared or are missing from the current captured image 12, a counter for the missing static objects 50 will be initiated to determine the number of captured images 11 whereby the static objects 50 have gone missing, such shown at Step 1003.
  • the counter for the missing static objects 50 will be compared to a missing threshold, wherein the missing threshold, by way of example, may be set as 15 or more captured images 11 or equivalent to 1 second in a 15 frames-per-second (fps) video. If the counter is higher than the missing threshold, the trackers for the static objects 50 will be removed at Step 1005. However, if the static objects 50 are still in the current captured image 12, then the trackers for said static objects 50 will be updated accordingly at Step 1006, wherein the trackers may include but not limited to tracker labels, static object 50 location, lifetime and the likes.
  • the missing threshold by way of example, may be set as 15 or more captured images 11 or equivalent to 1 second in a 15 frames-per-second (fps) video. If the counter is higher than the missing threshold, the trackers for the static objects 50 will be removed at Step 1005. However, if the static objects 50 are still in the current captured image 12, then the trackers for said static objects 50 will be updated accordingly at Step 1006, wherein the trackers may include but not limited to
  • FIG. 11 illustrates an exemplary embodiment of a first method applied by the subroutine 905 in the static object tracking module 5.
  • the current captured image 12 and the second background model 32 obtained from the object detection module 3 will be input into the subroutine 905, in which the feature points from both the current capture image 12 and the second background model 32b are extracted as shown by Steps 1101 and 1102.
  • the subroutine will analyse matching feature points between the current captured image 12 and the second background model 32b, in which a first similarity ratio is computed from the outcome of the analysis, as illustrated by Steps 1104 and 1105 respectively. It shall be understood that if a majority of the feature points are not matched according to the ratio, the static objects 50 are potentially still in the current captured image 12.
  • the similarity ratio will then be used in a next method according to FIG. 12.
  • the current captured image 12 and the registered reference image 904 obtained in the static object tracking module 5 will be input into a second method applied by the subroutine 905.
  • the feature points of the current captured image 12 and the background of the registered reference image 904 are extracted at Step 1201 and Step 1202 respectively.
  • the matching feature points of both the current captured image 12 and the background of the registered reference image 904 are analysed, in which a second similarity ratio is computed and assigned at Steps 1204 and Step 1205.
  • the first and the second similarity ratios are further computed to obtain a final ratio to determine the existence of the static objects 50 for each static object 50 trackers.
  • the final ratio is 1 or less than 1, it means the current captured image 12 is more similar to the second background model 32b as obtained from the object detection module 3 as compared to the registered reference image 904 obtained from the static object tracking module 5.
  • the final ratio is less than 1, the static objects 50 are deemed to “still exist” at Step 1208 and will be subsequently utilised in the subroutine 905 accordingly.
  • the final ratio is more than 1, the static objects 50 are deemed to be “missing” at Step 1207 and the counter for the missing static objects 50 will be initiated as shown previously in FIG. 10.
  • FIG. 13 illustrates a preferred embodiment for determining a static object 50 smoothness based on inputs from the static object tracking module 5.
  • the event recognition module 6 will gather the input from the static object tracking module 5 which includes the updated states 51 for the static objects 50, in which the input of a duration for the static objects 50 will be compared against a triggering time threshold as according to Step 1301. If the duration for the static objects 50 is more than the threshold, a test will be applied to examine the static object 50 smoothness to categorize the different static objects 50 present in the captured images 11, as shown at Step 1302. For example, this test will be able to differentiate between a vehicle and a road in the captured images 11.
  • Step 1303 the output from the test is evaluated at Step 1303, which denotes that if the static objects 50 are not smooth, then an alert 61 will be triggered at Step 1304, whereby said alert 61 will be sent to the alert and display module 7 in the system.
  • the steps to test the static object smoothness will be further discussed in FIG. 14.
  • FIG. 14 illustrates an exemplary embodiment of the smoothness test to categorize the different static objects 50.
  • the current capture image 12 is input into the event recognition module 6 and a low pass filter is applied repetitively onto the current captured image 12 in order to smooth out the current captured image 12 by removing high frequency features found in said current captured image 12, which may include edges of the static objects 50 and the likes, resulting in a smoothed image to be further computed.
  • a mean difference value between image pixels of the current captured image 12 and the smoothed image will be computed within the area containing the static objects 50. If the area is smooth, then it is expected that the mean difference value will be small as there are not much differences between the current captured image 12 before and after the application of the low pass filters.
  • the mean difference value will be compared against a non-smooth threshold value, whereby the static objects 50 will be classified as “not smooth” at Step 1405 if the mean difference value is more than the non-smooth threshold, and classified as “smooth” at Step 1404 if the mean difference value is less than the non-smooth threshold.
  • the outcome of the comparison will subsequently categorize the different static objects 50 according to their statuses and any static objects 50 that are “not smooth” will trigger the alert 61 in the event recognition module 6, which will subsequently be sent to the alert and display module 7 accordingly.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Image Analysis (AREA)

Abstract

The present invention discloses a system for tracking objects in a dynamic environment from a sequence of captured images (11), comprising an object detection module (3), a moving object tracking module (4), a static object tracking module (5), and an event recognition module (6). The system executes the following steps: detecting one or more moving objects (30) from the captured images (11) and updating a background model (32) of said captured images (11), filtering the detected moving objects (30) from noise and validates whether any of the detected moving objects (30) are static objects (50), determining whether the static objects (50) are in a static condition for a predetermined time period and updates status of the static objects (50) accordingly, and applying a smoothness test if the static objects (50) are static for the predetermined time period to categorize the different static objects (50).

Description

A SYSTEM AND METHOD FOR TRACKING OBJECTS IN A DYNAMIC ENVIRONMENT FROM A SEQUENCE OF CAPTURED IMAGES
FIELD OF INVENTION
The present invention relates to a system and method for tracking objects, more particularly, tracking objects in a dynamic environment from a sequence of captured images.
BACKGROUND OF THE INVENTION
In recent years, technology has advanced exponentially, bringing forth the integration of artificial intelligence in digital industry, more particularly for use in video surveillance systems. Video surveillance enables object monitoring through video displays of one or more areas remote from a human monitor. Various exemplary applications include security surveillance of public and private areas, assembly areas such as train stations and many more. However, a problem faced with video as a surveillance tool is that it may be manually intensive to monitor, in which it requires a user to be on constant lookout for any issues that might arise within a particular area of interest. In addition, human visual attention may be ineffective, particularly for large volumes of video data. Accordingly, it is desirable to implement an automated system suitable for use in video analysis.
Many technologies related to tracking objects in a dynamic environment have been proposed to further improve the system. For example, a United States patent with publication no. US9454819B1 discloses a method for static and moving object detection by employing a motion computation method based on spatio-temporal tensor formulation, a foreground and background modelling method, and a multi-cue appearance comparison method. The invention in use, operates in the presence of shadows, illumination changes, dynamic background, and both stopped and removed objects. According to the document, it also includes performing motion segmentation and background subtraction as key steps for detecting pixel-level motion from a source of video images, in which by fusing results of the motion segmentation, background and foreground modelling, the moving and static foreground objects are able to be identified and further classified as stopped objects or background models.
Another United States patent with publication no. US9224049B2 discloses a method of extracting foreground object image features from an input video via a background subtraction mask in order to determine whether a foreground object is in motion or static. According to the document, a motion analysis of the foreground object is carried out to determine the likelihood of the object in motion is either static for a threshold time period or moving in a particular direction.
Besides that, a United States patent with publication no. US7801330B2 discloses a technique for video processing which includes receiving a video from a scene, detecting moving pixels in the video, detecting line segments or motion blocks in the video based on the detected motion block, tracking targets in the video based on the identified targets, and managing the tracked targets in the video. The document also recites the method of detecting at least one static or moving identified targets in the scene, counting the detected static targets and estimating a dwell time based on the counted static targets, in which the estimated dwell time will be recorded in a log for future reference.
Another technology as disclosed in a United States patent with publication no. US8744123B2 recites a method for detecting objects in the foreground by generating a foreground object blob with a bounding box, further classified as a background, moving foreground, or temporally static object, wherein the temporally static object is distinguished from the background and remains static in the scene for a threshold period. Further, a static value ratio is computed as a number of pixels determined to be static by tracking in a foreground region of the background model corresponding to the tracked object bounding box over a total number of pixels of the foreground region.
The aforementioned patent documents describe the many methods for tracking objects in a dynamic environment, which is one of the key components in the proposed invention. A major drawback arises therefrom as the systems generally function by detecting and identifying the moving or static object by the pixels occupying a section in the captured images. No system have been able to conduct evaluations or make comparisons to further differentiate the moving or static objects from other factors which may cause inaccuracies in the results, especially in low-light conditions such as night time, when noise is predominantly captured by the video surveillance tool.
There exists a need to provide a system and method of such configuration, particularly being able to accurately distinguish between both the moving and static objects from one another or from other models in the captured images.
SUMMARY OF INVENTION
The present invention discloses a method for tracking objects in a dynamic environment from a sequence of captured images, the method comprising: detecting one or more moving objects from the captured images and updating a background model of said captured images, by an object detection module; filtering the detected moving objects from noise and validates whether any of the detected moving objects are static objects, by a moving object tracking module; determining whether the static objects are in a static condition for a predetermined time period and updates status of the static objects accordingly, by a static object tracking module; and applying a smoothness test if the static objects are static for the predetermined time period to categorize the different static objects, by an event recognition module. Preferably, the method further comprising the steps of extracting properties of the detected moving objects to be used as inputs for updating the background model of said captured images, by the object detection module, and extracting background feature points based on the updated background model of said captured images, by the object detection module.
Preferably, the background model is updated by the steps of examining whether a tracker exists for the static objects and the detected moving objects in the captured images, and updating a subsequent background model by copying an initial background model to said subsequent background model if the trackers for the static objects and the detected moving objects do not exist.
Preferably, the method further comprises the steps of executing a tracking algorithm for updating statuses of the detected moving objects according to the properties of the detected moving objects obtained from the object detection module, analysing an initial movement status of the detected moving objects and remove trackers for the detected moving objects that are considered noise, and removing the trackers for the detected moving objects that are validated as static objects to be further tracked by the static object tracking module, by the moving object tracking module.
Preferably, the initial movement status of the detected moving objects is analysed by the steps of estimating a movement status of the detected moving objects in a current captured image, accumulating a number of captured images whereby the detected moving object is not moving starting from the initial detection of the moving object in the captured images, and comparing the accumulated number of captured images to a threshold to determine the movement status of the detected moving objects.
Preferably, the movement status of the detected moving objects is estimated by the steps of extracting significant feature points within a detected moving object area in the current captured image, obtaining corresponding matching feature points in the detected moving object area in a previous captured image, computing a magnitude of each feature point in the detected moving object areas for the previous and the current captured images to obtain an average magnitude of the detected moving objects, and comparing the average magnitude of the detected moving objects to a static threshold to determine whether the detected moving objects are moving or not in the current captured image.
Preferably, the detected moving objects are validated as static objects by the steps of obtaining the movement status of the detected moving objects in the current captured image, examining whether the detected moving objects are static or have previously moved based on the initial movement status of the detected moving objects, appending the detected moving objects to a list of validated static objects in the current captured image to be further tracked in the static object tracking module if the detected moving objects are static or have previously moved, and removing the detected moving object trackers of the validated static objects.
Preferably, the method further comprises the steps of comparing a list of existing static object trackers to the validated static objects in the current captured image initializing new trackers to the validated static objects with no existing trackers, registering the current captured image as a reference image to be used in the static object tracking module, and employing a subroutine with at least two computational methods for determining whether the static objects remain static for a predetermined sequence of captured images, by the static object tracking module.
Preferably, the subroutine employed by the static object tracking module comprises the steps of determining an existence of the static objects for each static object tracker using at least two computational methods, calculating the number of captured images in which the static objects are missing from the captured images, comparing the number of captured images to a predetermined missing threshold, removing the trackers for the static objects if the number of captured images exceed the predetermined missing threshold, and updating the status of the static objects accordingly if the static objects remains static in the captured images.
Preferably, a first of the at least two computational method applied by the subroutine comprises the steps of comparing similarities between feature points extracted from the current captured image and the background feature points obtained from the object detection module, and computing a similarity ratio of the feature points extracted from the current captured image and the background feature points.
Preferably, a second of the at least two computational method applied by the subroutine comprises the steps of comparing similarities between the feature points extracted from the current captured image and the registered reference image obtained in the static object tracking module, computing a second similarity ratio of the feature points extracted from the current captured image and registered reference image, computing a final ratio between the first and second similarity ratios, and comparing the computed final ratio to a predetermined threshold to determine whether the static objects remains static in the current captured image or the static objects have already moved.
Preferably, the smoothness test is applied by the steps of applying a low pass filter repetitively on the current captured image to obtain a smoothed image, computing a mean difference between the smoothed image and the current captured image within a static object area, and comparing the mean difference to a non-smooth threshold to determine if the static objects are smooth or not smooth.
In another aspect of the invention, there is provided a system for tracking objects in a dynamic environment from a sequence of captured images, the system comprising of: an object detection module, configured to detect one or more moving objects from the captured images and updating a background model of said captured images; a moving object tracking module, configured to filter the detected moving objects from noise and validates whether any of the detected moving objects are static objects; a static object tracking module, configured to determine whether the static objects are in a static condition for a predetermined time period and updates status of the static object accordingly; and an event recognition module, configured to apply a smoothness test if the static objects are static for the predetermined time period to categorize the different static objects.
One skilled in the art will readily appreciate that the present invention is well adapted to carry out the objects and obtain the ends and advantages mentioned, as well as those inherent therein. The embodiment described herein is not intended as limitations on the scope of the invention.
BRIEF DESCRIPTION OF THE DRAWINGS
For the purpose of facilitating an understanding of the invention, there is illustrated in the accompanying drawing the preferred embodiments from an inspection of which when considered in connection with the following description, the invention, its construction and operation and many of its advantages would be readily understood and appreciated.
FIG. 1 is a block diagram of a system for tracking objects in a dynamic environment from a sequence of captured images, according to the present invention.
FIG. 2 is a flow chart illustrating a preferred embodiment for a method for tracking objects in a dynamic environment from a sequence of captured images based on the above-mentioned system. FIG. 3 is a flow chart illustrating a preferred embodiment for detecting moving objects in the captured images.
FIG. 4 is a flow chart illustrating an exemplary embodiment for updating a background image model based on inputs from the detected moving objects.
FIG. 5 is a flow chart illustrating a preferred embodiment for tracking and validating the detected moving objects in the captured images.
FIG. 6 is a flow chart illustrating an exemplary embodiment for analysing an initial movement status of the detected moving objects.
FIG. 7 is a flow chart illustrating an exemplary embodiment for estimating a movement status of the detected moving objects in a current captured image.
FIG. 8 is a flow chart illustrating a preferred embodiment for validating the detected moving objects to be further tracked in a static object tracking module.
FIG. 9 is a flow chart illustrating a preferred embodiment for tracking static objects in in the captured images.
FIG. 10 is a flow chart illustrating a preferred embodiment of a subroutine in the static object tracking module to examine and update existing static object trackers.
FIG. 11 is a flow chart illustrating an exemplary embodiment of a first method employed by the subroutine in the static object tracking module.
FIG. 12 is a flow chart illustrating an exemplary embodiment of a second method employed by the subroutine in the static object tracking module. FIG. 13 is a flow chart illustrating a preferred embodiment for determining a static object smoothness based on inputs from the static object tracking module.
FIG. 14 is a flow chart illustrating an exemplary embodiment of a smoothness test to categorize different static objects.
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, the invention shall be described according to the preferred embodiments of the present invention and by referring to the accompanying description and drawings. However, it is to be understood that limiting the description to the preferred embodiments of the invention is merely to facilitate discussion of the present invention and it is envisioned that those skilled in the art may devise various modifications without departing from the scope of the appended claim.
It shall be understood that each block of the flowchart illustrations and combinations thereof may be implemented by computer program instructions. As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or a computer program product. Accordingly, these computer program instructions of the present invention may take the form of an entirely software embodiment, including firmware, resident software micro -code, or the likes, or an embodiment which combines software and hardware aspects that may all generally be referred to herein as a “system”.
The computer program instructions may be embodied in one or more computer readable mediums having computer readable program code embodied thereon. These computer program instructions may then direct a computer or a programmable data processing apparatus to function in a specific manner, such that the instructions stored in the computer readable medium may produce an article of manufacture which includes instruction means for implementing the functions in the flowchart or block diagrams to be discussed herein.
Furthermore, the computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions that are executed on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or block diagrams.
Referring to FIG. 1, there is provided a system for tracking objects in a dynamic environment from a sequence of captured images 11 from a video acquisition module 1, wherein the video acquisition module 1 may include a digital camera, digital video recorder, a network video camera, a closed-circuit television (CCTV) camera, or any other mediums suitable for capturing images in a sequence. In one preferred embodiment, the captured images 11 may be transferred wirelessly over a communication network to an image processing module 2, wherein the communication network may be a wireless network connection established via a wireless protocol cloud such as a Long-Term Evolution (LTE) cloud, Code Division Multiple Access (CDMA) and its derivatives, Enhanced Data Rates for GSM Evolution (EDGE), 3G protocol, High Speed Packet Access (HSPA), 4G protocol, 5G protocol and the likes, in accordance to the advancement of wireless technology with time. Alternatively, the communication network can also be a wired network via a local network port from one computer to another. In a preferred embodiment, captured images 11 from the video acquisition module 1 may be stored in an internal memory storage device which can be installed within the video acquisition module or externally linked therewith, wherein internal or external memory storage device may include primary storage devices, such as Raw Access Memory (RAM), which is used by computer systems alike to temporarily store and retrieve data, or secondary storage devices such as Hard Disk Drives (HDD) or Solid State Drives (SSD) to store data permanently. Preferably, the plurality of modules in the image processing module 2 communicate with one another to form an interconnected communication network.
In one particular embodiment, the image processing module 2 comprises an object detection module 3, configured to detect any moving objects 30 in the captured images by modelling a background using a combination of appearance and feature point distribution, to which the background is updated using information feedback from the respective tracking modules thereon, in which the features or properties 33 of the detected moving objects 30 may include, by way of example but not limited to, the colour, size, position, orientation and direction of motion of the detected moving objects 30.
In another particular embodiment, the image processing module 2 comprises a moving object tracking module 4, configured to track the detected moving objects 30 and differentiate the detected moving objects 30 from noise present in the captured images 11 based on the tracking states 42 of the detected moving objects 30, which may include location 31, appearance model, tracker label and many more. In a preferred embodiment, the moving object tracking module 4 is employed as a first level of tracking in order to track all detected moving objects 30 within a region of interest, and a next level of tracking will be subsequently triggered once the moving object tracking module 4 detects any potential static objects 50 within the region of interest by means of validating the detected moving objects 30 based on a set of predetermined conditions, subsequently feeding the information of the potential static objects 50 to the next module for further tracking. For example, the set of predetermined conditions may include the detected moving objects 30 was previously moving or static, or the detected moving objects 30 is not static and is still moving. In another particular embodiment, the image processing module 2 comprises a static object tracking module 5, configured to track valid static objects 50 within the region of interest which is initiated by the output from the moving object tracking module 4. The static object tracking module 5 is a second level of tracking which is initiated when the detected moving objects 30 in the moving object tracking module 4 are detected as potential static objects 50 by means of validation based on the set of predetermined conditions which may include the detected moving objects 30 were previously moving and are currently not moving. In a preferred embodiment, the static object tracking module 5 tracks the static objects 50 independently from any other detected moving objects 30. However, a problem faced by many technologies during the tracking or detection of static objects 50 is that the static objects 50 will be absorbed into the background due to the cluster of noise surrounding them or due to many other factors alike. As such, the static objects 50 will fail to be detected by the object detection module 3, in which this is known as a missing observation. Thus, the static object tracking module 5 employs certain key steps to overcome this obstruction which will be discussed further herein.
In another particular embodiment, the image processing module 2 comprises an event recognition module 6, configured to perform an event analysis and recognition by analysing the output from the static object tracking module 5 based on the condition that the tracked static objects 50 remained static for a predetermined time period. In the event that the tracked static objects 50 remained static for the predetermined time period, the event recognition module 6 will apply a smoothness test to categorize the different static objects 50 which may include vehicles, plants, roads, trees and the likes, prior to sending out an alert 61 to an alert and display module 7, as signalling the presence of a valid static object 50. In a preferred embodiment, the alert and display module 7 may be a user interface integrated into the computer system which notifies the user in the form of a ringing alarm or other notification means. FIG. 2 is a flow chart illustrating a preferred embodiment for a method for tracking objects in the dynamic environment from the sequence of captured images 11 based on the above-mentioned system. More particularly, the method for tracking objects in the dynamic environment occur within the image processing module 2. Firstly, the captured images 11 are input into the object detection module 3, whereby the output of the object detection module 3 are two folds, whereby one of the output is a location 31 of the detected moving objects 30 and a second output is a background model 32 of the captured images 11. Preferably, the background model 32 is depicted by a fusion of captured images 11 appearance model and feature points distribution collectively. The detected moving object location 31 information will then be passed to the moving object tracking module 4, which is the first level of tracking, whereby the detected moving objects 30 will be labelled with the same tracker label across successive captured images 11 until a change occurs within said captured images 11. In a preferred embodiment, the detected moving objects 30 in the moving object tracking module 4 will be further classified, based on the tracker information, as an actual object or noise due to vehicular illumination, particularly in night conditions when noise is predominant in the captured images 11. Further, the output from the moving object tracking module 4 is the updated detected moving objects’ 30 statuses 42, which may include tracker labels, moving object location 31, updated object appearance model and the likes. Preferably, the output will be used to decide if the second level of tracking is necessary to be initiated. At Step 41, each of the detected moving objects 30 from the moving object tracking module 4 will be examined and validated whether they are valid static objects 50 in the captured images 11 or not. In a preferred embodiment, a predetermined set of conditions will be checked in order to validate the detected moving objects 30 in order to initiate the second level of tracking. As such, a list of the detected moving objects 30 found to be valid static objects 50 in the current captured image 12 is subsequently compiled and this information will be passed on to the static object tracking module 5 to be further tracked. In turn, the static object tracking module 5 will track the valid static objects 50 by comparing the current captured image 12 to the background models 32 obtained from the object detection module 3. As long as the static objects 50 are still detected by the static object tracking module 5 in the current captured image 12, the tracker for said static objects 50 will be maintained. However, when the static objects 50 start to move in a subsequent captured images 11, the tracker for the static objects will be removed accordingly. The output from the static object tracking module 5 will be passed on to the event recognition module 6, in which the output includes the updated static object states 51 such as location, appearance model and the likes. Further, the output will be examined by the event recognition module 6 to detect the event of the static objects 50 remaining in the sequence of captures images 11, by way of example but not limited to, a vehicle stopping by the side of the road or the likes. If the event is confirmed, the alert 61 will be triggered and sent to the alert and display module 7.
FIG. 3 is a flow chart illustrating a preferred embodiment for detecting the moving object in the captured images 11 in the object detection module 3. At Step 301, the detected moving objects 30 in the captured images 11 are detected by using any standard moving object detection algorithm, by way of example but not limited to, a mixture of Gaussian (MOG) algorithm, temporal differencing, background subtraction, optical flow, frame differencing, approximation median and the likes, each with their own advantages over each other. The output from the detected moving objects 30 are the detected moving object location 31 and an initial background model 32a from the captured images 11, wherein the detected moving object location 31 will be utilised in extracting the properties 33 of the detected moving objects 30 at Step 302. Preferably, the properties 33 to be extracted from the detected moving objects 30 may include the object size, object centroid, object orientation, and the likes. The extracted properties 33 of the detected moving objects 30 and the initial background model 32a are then utilised for updating the background model at Step 303. As such, the moving object properties 33 will be used concurrently with the updated static object states 51 obtained from the static object tracking module 5 and the initial background model 32a to be input and analysed for obtaining an updated subsequent background model 32b. Further, background feature points based on the updated subsequent background model 32b are then extracted at Step 304 to be used in the static tracking module 5.
FIG. 4 illustrates an exemplary embodiment for updating the background model 32 based on inputs from the properties 33 of the detected moving objects 30 and updated states 51 from the static object tracking module 5. In this particular embodiment, two different background models 32 are maintained. The first background model 32a is for detecting a moving object 30 in the captured images 11. For the first background model 32a, any static objects 50 will be absorbed into the background instead of the foreground, causing the static objects 50 to be detected as the background instead. As such, a second background image model 32b will then be updated to include any static objects 50 in the captured images 11. At Step 401, the object detection module 3 will examine whether there are any trackers for static objects 50 and detected moving objects 30 in the captured images 11. If there are trackers for static objects 50 in the captured images 11 or if there are detected moving objects 30 in the captured images 11, then the background model 32 will not be updated. However, if no moving objects 30 and static objects 50 are detected in the current captured image 12, only then will the second background model 32b be updated by copying the first background model 32a, as illustrated by Step 402. This is to ensure that the background model 32 is not tampered with any static objects 50 in the foreground.
FIG. 5 illustrates a preferred embodiment for tracking and validating the detected moving objects 30 in the moving object tracking module 4. In a preferred embodiment, the moving object tracking module 4 is employed as the first level of tracking objects in the dynamic environment from the sequence of captured images 11. According to FIG. 5, the location 31 and properties 33 of the detected moving objects 30 obtained from the object detection module 3 are input into the moving object tracking module 4. At Step 501, the moving object tracking module 4 will perform the tracking by updating the states 42 of the detected moving objects according to the inputs from the object detection module 3, wherein the states 42 may include tracking labels, appearance models, tracking lifetime, and the likes. At Step 502, the moving object tracking module 4 will analyse the initial movement status of the detected moving objects 30, wherein the initial movement status will be used to further classify whether said detected moving objects 30 are noise or not. The steps for analysing the initial movement status for the detected moving objects 30 will be further discussed in FIG. 6. At Step 503, the moving object tracking module 4 will remove the trackers for any detected moving objects 30 that are considered noise based on the analysis of their initial movement status, in which the detected moving objects 30 that are not considered noise will be further evaluated at Step 504 for their validity as static objects 50 to be further tracked in the subsequent static object tracking module 5. Preferably, the moving object tracking module 4 will also remove the trackers for the detected moving objects 30 which are validated as static objects 50 to be tracked by the static object tracking module 5 which will be discussed further herein.
FIG. 6 illustrates an exemplary embodiment for analysing the initial movement status of the detected moving objects 30 in the moving object tracking module 4. The analysis is conducted by keeping track of the detected moving objects 30 in successive captured images 11, starting from the moment the detected moving objects 30 enter the region of interest in the captured images 11. At Step 601, the moving object tracking module 4 will estimate the movement status of the detected moving objects 30 in the current captured image 12. Referring to FIG. 7, the movement status of the detected moving objects 30 in the current captured image 12 begins by extracting significant feature points within an area containing the detected moving objects 30 at Step 701. At Step 702, corresponding matching feature points in previous captured images 11 will subsequently be obtained, in which the moving object tracking module 4 will compute a magnitude of each optical flow points within the detected moving object 30 area at Step 703, prior to computing an average magnitude of the optical flow points for the detected moving objects 30 at Step 704. At Step 705, the average magnitude will then be compared to a static threshold value in order to subsequently determine the movement status of the detected moving objects 30 in the current captured image 12. If the average magnitude is above the static threshold value, the detected moving objects 30 will be classified as “moving” at Step 707, but if the average magnitude is below the static threshold value, then the detected moving objects 30 will be classified as “static” or “not moving” at Step 706. Referring back to FIG. 6, the estimated movement status of “moving” or “not moving” will be further evaluated at Step 602. If the estimated movement status is “moving”, then the detected moving objects 30 will be labelled as “previously moved” at Step 603, otherwise the estimated movement status of the detected moving objects 30 will be further evaluated in successive captured images 11, whereby a number of captured images 11 in which the detected moving objects 30 are deemed as “not moving” will be accumulated. At Step 604, the accumulated number of captured images 11 whereby the detected moving objects 30 are “not moving”, from the first instance the moving objects were detected, are then compared to a predetermined threshold at Step 605, wherein the threshold, by way of example, may be set as 25 captured images 11, equivalent to a 1 second video duration for a 25 frames- per-second (fps) video. At Step 606, if the accumulated number of captured images 11 are more than the threshold, the detected moving objects 30 will be determined as “never moved”, in which said detected moving objects 30 will be further classified as noise, and the trackers for said detected moving objects 30 will be removed.
FIG. 8 illustrates an exemplary embodiment for validating the detected moving objects 30 as static objects 50 to be further tracked in the static object tracking module 5. At Step 801, both the initial movement status and the estimated movement status of the detected moving objects 30 in the current captured image 12 are obtained to be evaluated. If the detected moving objects 30 satisfy conditions of “static” or have “previously moved” at Step 802, based on their movement statuses, the detected moving objects 30 will be determined as valid static objects 50 to be further tracked in the static object tracking module 5. However, if the detected moving objects 30 are not “static” or have not “previously moved”, their trackers will be removed and will not be further tracked in the static object tracking module 5 therein, such as illustrated in Step 804. At Step 803, the detected moving objects 30 that have satisfied the conditions of will be appended to a list of validated static objects 50 in the current captured image 12. Subsequently, the validated static objects 50 will have their detected moving objects 30 trackers removed at Step 804, and the information of the validated static objects 50 will be fed into the static object tracking module 5 for further tracking thereof.
FIG. 9 illustrates a preferred embodiment for tracking the static objects 50 in the captured images 11. According to the present invention, the static object tracking module 5 is responsible for tracking the static objects 50 in the region of interest and subsequently update a duration of the static objects 50 remaining in a static condition. According to FIG. 9, input to the static object tracking module 5 is the list of validated static objects 50 obtained from the moving object tracking module 4. In a preferred embodiment, a list of existing static object 50 trackers will be checked against the validated static objects 50 to determine whether there is an existing tracker, as illustrated at Step 901. At Step 902, if the existing trackers overlap with the validated static objects 50, then no trackers will be initialized. However, if there are no existing trackers assigned to the validated static objects 50, new static trackers will be initialized and assigned to the validated static objects 50, such as in Step 903. During this initialization, at Step 904, the current captured image 12 will be registered as a reference image 904 for the validated static objects 50. Preferably, the tracker states 51 of the static objects 50 will be subsequently updated in a subroutine 905 employed by the static object tracking module 5, which will be discussed further in FIG. 10 to FIG. 12.
FIG. 10 illustrates an exemplary embodiment of the subroutine 905 from the static object tracking module 5, whereby all existing static objects 50 trackers will be examined to update their tracker states 51. At Step 1001, each static objects 50 tracker will be examined to determine the existence of the static objects 50 in the current captured image 12 by applying at least two computational methods which will be shown in FIG. 11 and FIG. 12 respectively. The output of Step 1001 is the statuses of whether the static objects 50 are still in the current captured image 12 or not, which will be further evaluated at Step 1002. If the static objects 50 have disappeared or are missing from the current captured image 12, a counter for the missing static objects 50 will be initiated to determine the number of captured images 11 whereby the static objects 50 have gone missing, such shown at Step 1003. At Step 1004, the counter for the missing static objects 50 will be compared to a missing threshold, wherein the missing threshold, by way of example, may be set as 15 or more captured images 11 or equivalent to 1 second in a 15 frames-per-second (fps) video. If the counter is higher than the missing threshold, the trackers for the static objects 50 will be removed at Step 1005. However, if the static objects 50 are still in the current captured image 12, then the trackers for said static objects 50 will be updated accordingly at Step 1006, wherein the trackers may include but not limited to tracker labels, static object 50 location, lifetime and the likes.
FIG. 11 illustrates an exemplary embodiment of a first method applied by the subroutine 905 in the static object tracking module 5. In this first method, the current captured image 12 and the second background model 32 obtained from the object detection module 3 will be input into the subroutine 905, in which the feature points from both the current capture image 12 and the second background model 32b are extracted as shown by Steps 1101 and 1102. At Step 1103, the subroutine will analyse matching feature points between the current captured image 12 and the second background model 32b, in which a first similarity ratio is computed from the outcome of the analysis, as illustrated by Steps 1104 and 1105 respectively. It shall be understood that if a majority of the feature points are not matched according to the ratio, the static objects 50 are potentially still in the current captured image 12. Preferably, the similarity ratio will then be used in a next method according to FIG. 12. As illustrated in FIG. 12, the current captured image 12 and the registered reference image 904 obtained in the static object tracking module 5 will be input into a second method applied by the subroutine 905. Similar to the first method, the feature points of the current captured image 12 and the background of the registered reference image 904 are extracted at Step 1201 and Step 1202 respectively. At Step 1203, the matching feature points of both the current captured image 12 and the background of the registered reference image 904 are analysed, in which a second similarity ratio is computed and assigned at Steps 1204 and Step 1205. At Step 1206, the first and the second similarity ratios are further computed to obtain a final ratio to determine the existence of the static objects 50 for each static object 50 trackers. In this context, if the final ratio is 1 or less than 1, it means the current captured image 12 is more similar to the second background model 32b as obtained from the object detection module 3 as compared to the registered reference image 904 obtained from the static object tracking module 5. As such, if the final ratio is less than 1, the static objects 50 are deemed to “still exist” at Step 1208 and will be subsequently utilised in the subroutine 905 accordingly. However, if the final ratio is more than 1, the static objects 50 are deemed to be “missing” at Step 1207 and the counter for the missing static objects 50 will be initiated as shown previously in FIG. 10.
FIG. 13 illustrates a preferred embodiment for determining a static object 50 smoothness based on inputs from the static object tracking module 5. According to FIG. 13, the event recognition module 6 will gather the input from the static object tracking module 5 which includes the updated states 51 for the static objects 50, in which the input of a duration for the static objects 50 will be compared against a triggering time threshold as according to Step 1301. If the duration for the static objects 50 is more than the threshold, a test will be applied to examine the static object 50 smoothness to categorize the different static objects 50 present in the captured images 11, as shown at Step 1302. For example, this test will be able to differentiate between a vehicle and a road in the captured images 11. It is assumed that a road image texture in the captured images 11 is smoother as compared to a vehicle image texture. Therefore, the output from the test is evaluated at Step 1303, which denotes that if the static objects 50 are not smooth, then an alert 61 will be triggered at Step 1304, whereby said alert 61 will be sent to the alert and display module 7 in the system. Preferably, the steps to test the static object smoothness will be further discussed in FIG. 14.
FIG. 14 illustrates an exemplary embodiment of the smoothness test to categorize the different static objects 50. At Step 1401, the current capture image 12 is input into the event recognition module 6 and a low pass filter is applied repetitively onto the current captured image 12 in order to smooth out the current captured image 12 by removing high frequency features found in said current captured image 12, which may include edges of the static objects 50 and the likes, resulting in a smoothed image to be further computed. At Step 1402, a mean difference value between image pixels of the current captured image 12 and the smoothed image will be computed within the area containing the static objects 50. If the area is smooth, then it is expected that the mean difference value will be small as there are not much differences between the current captured image 12 before and after the application of the low pass filters. At Step 1403, the mean difference value will be compared against a non-smooth threshold value, whereby the static objects 50 will be classified as “not smooth” at Step 1405 if the mean difference value is more than the non-smooth threshold, and classified as “smooth” at Step 1404 if the mean difference value is less than the non-smooth threshold. As such, the outcome of the comparison will subsequently categorize the different static objects 50 according to their statuses and any static objects 50 that are “not smooth” will trigger the alert 61 in the event recognition module 6, which will subsequently be sent to the alert and display module 7 accordingly.
The present disclosure includes as contained in the appended claims, as well as that of the foregoing description. Although this invention has been described in its preferred form with a degree of particularly, it is understood that the present disclosure of the preferred form has been made only by way of example and that numerous changes in the details of construction and the combination and arrangements of parts may be resorted to without departing from the scope of the invention.

Claims

1. A method for tracking objects in a dynamic environment from a sequence of captured images (11), the method is characterized by having the steps of: detecting, by an object detection module (3), one or more moving objects (30) from the captured images (11) and updating a background model (32) of said captured images (11); filtering, by a moving object tracking module (4), the detected moving objects (30) from noise and validates whether any of the detected moving objects (30) are static objects (50); determining, by a static object tracking module (5), whether the static objects (50) are in a static condition for a predetermined time period and updates status of the static objects (50) accordingly; and applying, by an event recognition module (6), a smoothness test if the static objects (50) are static for the predetermined time period to categorize the different static objects (50).
2. The method according to Claim 1, further comprising the steps of: extracting, by the object detection module (3), properties (33) of the detected moving objects (30) to be used as inputs for updating the background model (32) of said captured images (11); and extracting, by the object detection module (3), background feature points based on the updated background model (32) of said captured images (11).
3. The method according to Claim 2, wherein the background model (32) is updated by the steps of: examining whether a tracker exists for the static objects (50) and the detected moving objects (30) in the captured images (11); and updating a subsequent background model (32b) by copying an initial background model (32a) to said subsequent background model (32b) if the trackers for the static objects (50) and the detected moving objects (30) do not exist.
4. The method according to Claim 1, further comprises the steps of: executing, by the moving object tracking module (4), a tracking algorithm for updating statuses of the detected moving objects (30) according to the properties (33) of the detected moving objects (30) obtained from the object detection module
(3); analysing, by the moving object tracking module (4), an initial movement status of the detected moving objects (30) and remove trackers for the detected moving objects (30) that are considered noise; and removing, by the moving object tracking module (4), the trackers for the detected moving objects (30) that are validated as static objects (50) to be further tracked by the static object tracking module (5).
5. The method according to Claim 4, wherein the initial movement status of the detected moving objects (30) is analysed by the steps of: estimating a movement status of the detected moving objects (30) in a current captured image (12); accumulating a number of captured images (11) whereby the detected moving objects (30) are not moving starting from the initial detection of the moving objects (30) in the captured images (11); and comparing the accumulated number of captured images (11) to a threshold to determine the movement status of the detected moving objects (30).
6. The method according to Claim 5, wherein the movement status of the detected moving objects (30) is estimated by the steps of: extracting significant feature points within the detected moving object (30) area in the current captured image (12); obtaining corresponding matching feature points in the detected moving object (30) area in previous captured images (11); computing a magnitude of each feature point in the detected moving object (30) areas for the previous and the current captured images (11, 12) to obtain an average magnitude of the detected moving objects (30); and comparing the average magnitude of the detected moving objects (30) to a static threshold to determine whether the detected moving objects (30) are moving or not in the current captured image (12).
7. The method according Claim 6, wherein the detected moving objects (30) are validated as static objects (50) by the steps of: obtaining the movement status of the detected moving objects (30) in the current captured image (12); examining whether the detected moving objects (30) are static or have previously moved based on the initial movement status of the detected moving objects (30); appending the detected moving objects (30) to a list of validated static objects (50) in the current captured image (12) to be further tracked in the static object tracking module (5) if the detected moving objects (30) are static or have previously moved; and removing the detected moving object (30) trackers of the validated static objects (50).
8. The method according to Claim 1, further comprises the steps of: comparing, by the static object tracking module (5), a list of existing static objects (50) trackers to the validated static objects (50) in the current captured image (12); initializing, by the static object tracking module (5), new trackers to the validated static objects (50) with no existing trackers; registering, by the static object tracking module, the current captured image (12) as a reference image (904) to be used in the static object tracking module (5); and employing, by the static object tracking module (5), a subroutine (905) with at least two computational methods for determining whether the static objects (50) remain static for a predetermined sequence of captured images (11).
9. The method according to Claim 8, wherein the subroutine (905) employed by the static object tracking module (5) to track the static objects (50), comprises the steps of: determining an existence of the static objects (50) for each static object (50) tracker using at least two computational methods; calculating the number of captured images (11) in which the static objects (50) are missing from the captured images (11); comparing the number of captured images (11) to a predetermined missing threshold; removing the trackers for the static objects (50) if the number of captured images (11) exceed the predetermined missing threshold; and updating the status of the static objects (50) if the static objects (50) remains static in the captured images (11).
10. The method according to Claim 9, wherein a first of the at least two computational methods applied by the subroutine (905) comprises the steps of: comparing similarities between feature points extracted from the current captured image (12) and the background model (32) obtained from the object detection module (3); and computing a similarity ratio of the feature points extracted from the current captured image (12) and the background model (32).
11. The method according to Claim 9, wherein a second of the at least two computational methods applied by the subroutine (905) comprises the steps of: comparing similarities between the feature points extracted from the current captured image (12) and the registered reference image (904) obtained in the static object tracking module (5); computing a second similarity ratio of the feature points extracted from the current captured image (12) and registered reference image (904); computing a final ratio between the first and second similarity ratios; and comparing the computed final ratio to a predetermined threshold to determine whether the static objects (50) remains static in the current captured image (12) or the static objects (50) have already moved.
12. The method according to Claim 1, wherein the smoothness test is applied on to the static objects (50) by the steps of: applying a low pass filter repetitively on the current captured image (12) to obtain a smoothed image; computing a mean difference between the smoothed image and the current captured image (12) within a static object (50) area; and comparing the mean difference to a non-smooth threshold to determine if the static objects (50) are smooth or not smooth.
13. A system for tracking objects in a dynamic environment from a sequence of captured images (11), the system is characterized by having: an object detection module (3), configured to detect one or more moving objects (30) from the captured images (11) and updating a background model (32) of said captured images (11); a moving object tracking module (4), configured to filter the detected moving objects (30) from noise and validates whether any of the detected moving objects (30) are static objects (50); a static object tracking module (5), configured to determine whether the static objects (50) are in a static condition for a predetermined time period and updates status of the static objects (50); and an event recognition module (6), configured to apply a smoothness test if the static objects (50) are static for the predetermined time period to categorize the different static objects (50).
PCT/MY2020/050190 2020-07-28 2020-12-04 A system and method for tracking objects in a dynamic environment from a sequence of captured images WO2022025752A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
MYPI2020003895 2020-07-28
MYPI2020003895 2020-07-28

Publications (1)

Publication Number Publication Date
WO2022025752A1 true WO2022025752A1 (en) 2022-02-03

Family

ID=80035975

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/MY2020/050190 WO2022025752A1 (en) 2020-07-28 2020-12-04 A system and method for tracking objects in a dynamic environment from a sequence of captured images

Country Status (1)

Country Link
WO (1) WO2022025752A1 (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009253449A (en) * 2008-04-02 2009-10-29 Canon Inc Monitoring apparatus and monitoring method
JP2012212238A (en) * 2011-03-30 2012-11-01 Secom Co Ltd Article detection device and stationary-person detection device
KR101496407B1 (en) * 2013-02-27 2015-02-27 충북대학교 산학협력단 Image process apparatus and method for closed circuit television security system
US9454819B1 (en) * 2015-06-03 2016-09-27 The United States Of America As Represented By The Secretary Of The Air Force System and method for static and moving object detection
US20190197313A1 (en) * 2016-09-23 2019-06-27 Hitachi Kokusai Electric Inc. Monitoring device

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009253449A (en) * 2008-04-02 2009-10-29 Canon Inc Monitoring apparatus and monitoring method
JP2012212238A (en) * 2011-03-30 2012-11-01 Secom Co Ltd Article detection device and stationary-person detection device
KR101496407B1 (en) * 2013-02-27 2015-02-27 충북대학교 산학협력단 Image process apparatus and method for closed circuit television security system
US9454819B1 (en) * 2015-06-03 2016-09-27 The United States Of America As Represented By The Secretary Of The Air Force System and method for static and moving object detection
US20190197313A1 (en) * 2016-09-23 2019-06-27 Hitachi Kokusai Electric Inc. Monitoring device

Similar Documents

Publication Publication Date Title
US11170225B2 (en) Stationary target detection by exploiting changes in background model
CN109784162B (en) Pedestrian behavior recognition and trajectory tracking method
CN109887281B (en) Method and system for monitoring traffic incident
US9323991B2 (en) Method and system for video-based vehicle tracking adaptable to traffic conditions
US8798314B2 (en) Detection of vehicles in images of a night time scene
EP1811457A1 (en) Video signal analysis
CN105144705B (en) Object monitoring system, object monitoring method, and program for extracting object to be monitored
WO2009049314A2 (en) Video processing system employing behavior subtraction between reference and observed video image sequences
US20060056702A1 (en) Image processing apparatus and image processing method
JP2013218679A (en) Video-based detection device and notification device for catching short-time parking violation
CN111723773B (en) Method and device for detecting carryover, electronic equipment and readable storage medium
KR101472674B1 (en) Method and apparatus for video surveillance based on detecting abnormal behavior using extraction of trajectories from crowd in images
CN110544271B (en) Parabolic motion detection method and related device
CN111582166B (en) Method for detecting remnants based on Gaussian modeling and YoLo V3 target detection
Tang et al. Hybrid blob and particle filter tracking approach for robust object tracking
KR101492059B1 (en) Real Time Object Tracking Method and System using the Mean-shift Algorithm
Verma et al. Analysis of moving object detection and tracking in video surveillance system
JP7125843B2 (en) Fault detection system
WO2012081969A1 (en) A system and method to detect intrusion event
KR101848367B1 (en) metadata-based video surveillance method using suspective video classification based on motion vector and DCT coefficients
GB2516142A (en) Method of tracking objects using hyperspectral imagery
WO2022025752A1 (en) A system and method for tracking objects in a dynamic environment from a sequence of captured images
Boufares et al. Adaptive threshold for background subtraction in moving object detection using stationary wavelet transforms 2D
Agrawal et al. Performance comparison of moving object detection techniques in video surveillance system
Purwar Crowd density estimation using hough circle transform for video surveillance

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20946667

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20946667

Country of ref document: EP

Kind code of ref document: A1