WO2014007762A1

WO2014007762A1 - A method and system for automated monitoring of traffic

Info

Publication number: WO2014007762A1
Application number: PCT/SG2013/000277
Authority: WO
Inventors: Seow Loong TAN
Original assignee: Tan Seow Loong
Priority date: 2012-07-04
Filing date: 2013-07-04
Publication date: 2014-01-09

Abstract

The objective of a system and method for automated monitoring of traffic is to reduce human intervention and error in detection of traffic rules being violated in certain areas or zones of roads, streets and avenues. The system comprises at least one camera, at least one front end computing system associated with the at least one camera and a back end computing system. Images and videos are captured by the cameras and transmitted to the front end or the back end computing systems. The front end and back end computing systems have image processing based software routines and applications for detecting vehicles in the images and videos and also to determine whether they are violating any traffic rules. Moreover, the front end computing system and the back end computing system have data compilation and reporting software applications to report the violation and the vehicle information.

Description

A METHOD AND SYSTEM FOR AUTOMATED MONITORING OF TRAFFIC

TECHNICAL FIELD OF INVENTION

This invention relates to a method and system of automated monitoring of traffic. This invention has particular, but not exclusive, application in the field of managing vehicular traffic and traffic law enforcement.

BACKGROUND

Law enforcement on roads and traffic management have long been done by various governmental setups and is becoming increasingly important and critical in the current scenario, especially with increasing automobiles on the roads and associated safety of drivers and pedestrians on the roads. Appropriate traffic law enforcement and traffic management is essential to the growth and progress of a society.

In some places, a law enforcer or traffic police personnel police the streets and roads to check if vehicles have been parked in non-parking areas or if any vehicle flouts traffic rules much to the inconvenience of other fellow drivers. This process involves officials on the streets and roads and in a very big city, the cost of manpower involved for this exercise can be quite substantial. Moreover, due to inefficiencies of the officials arising out of fatigue of the officials and monotony of the work, the chances of traffic offenders escaping from their flouting acts are also high. In countries where general law enforcement is not strong and efficient enough, corruption can also creep into this process, resulting in ineffective traffic law enforcement.

In other places, image capturing devices like cameras are installed in most-places to monitor illegal parking of vehicles or flouting of any traffic rules. These devices run continuously and keep capturing images or videos of vehicles or the traffic in general. The data from these devices are transmitted to a data centre or the traffic enforcement centre. The transmitted data, including images and videos is then analysed by officials gathered in the data centre or the traffic enforcement centre to determine if there is any violation of the traffic rules. In case of any violation, action in response to the violation will be taken by officials on the owner or the driver of the vehicle. Here again, inefficiencies or human error of the officials can be caused by fatigue or monotony, leading to traffic offenders escaping their acts. Moreover, there may also be a delay in communicating the fine or punishment to the offender due to any delay in analysing the transmitted data at the data centre. If the offender happens or plans to leave the country after the offence, then there are many legal hurdles involved in getting the offender back to Singapore, if the communication of the offence to offender is delayed and reaches after the offender has left the country.

Moreover, transmission of high resolution and high definition videos from the image capturing devices to data centres or traffic enforcement centres consume a lot of bandwidth, which is also a challenge in some places.

SUMMARY

In accordance with a first aspect of the invention, there is disclosed a method of automated monitoring of traffic. The first step is capturing of visual information associated with the traffic by an image capture device. The image capture device is directed at the traffic. In the next step, the captured visual information is then transmitted by the image capture device to a storage device by a transmission link. In the third step, the visual information transmitted by the image capture device is processed by a violation detection software routine in the storage device. This processing is to determine whether an element of the traffic is in violation of a traffic protocol associated with the violation detection software routine.

In accordance with a second aspect of the invention, there is disclosed a system for automated monitoring of traffic. The system comprises an image capture device for capturing visual information associated with the traffic, the image capture device being directed at the traffic. The system further comprises a transmission link for transmitting the visual information by the image capture device to a storage device. The system also comprises a storage device having a violation detection software routine to process the visual information to determine whether an element of the traffic is in violation of a traffic protocol associated with the violation detection software routine.

BRIEF DESCRIPTION OF DRAWINGS

FIG 1 shows a high level architecture of an embodiment of a system for automated monitoring of traffic with a single image capture device FIG 2 shows a high level architecture of another embodiment of a system for automated monitoring of traffic with two image capture devices

FIG 3 shows a high level architecture of yet another embodiment of a system for automated monitoring of traffic with two image capture devices

FIG 4 shows an exemplary arrangement of two image capture devices monitoring two lanes of a road

FIGs 5(a) - 5(d) shows exemplary enforcement lines along a section of a road or avenue

FIG 6a shows a block diagram of a vehicle recognition module in association with a vehicle image database

FIG 6(b) shows a process for recognition of a vehicle

FIG 7 shows determination of violation of a traffic protocol in the vehicle tracking module

FIG 8 shows a block diagram of a violation detection and analysis module

FIG 9 shows a method of automated monitoring of traffic

FIG 10(a) shows a car about to enter an enforcement zone

FIG 10(b) shows the car just entering the enforcement zone

FIG 10(c) shows the car inside the enforcement zone

FIG 10(d) and (e) show the car well inside the enforcement zone

FIG 10(f) shows the car fully outside the enforcement zone

FIG 1 1 (a) shows a car inside an enforcement zone captured at time point ti sec

FIG 1 1 (b) shows the car inside the enforcement zone captured at time point t₂ sec DETAILED DESCRIPTION

In the disclosure herein, consideration or use of a particular element number in a given FIG. or corresponding descriptive material can encompass the same, an equivalent, or an analogous element number identified in another FIG. or descriptive material corresponding thereto.

In the present disclosure, the term "set" corresponds to or is defined as a non-empty finite organization of elements that mathematically exhibits a cardinality of at least 1 (i.e., a set as defined herein can correspond to a singlet or single element set, or a multiple element set), in accordance with known mathematical definitions. In general, an element of a set can include or be a system, an apparatus, a device, a structure, an object, a process, a physical parameter, or a value depending upon the type of set under consideration.

FIG 1 shows a high level architecture of a system 100 for automated monitoring of traffic with a single image capture device. As illustrated in FIG 1 , along a road or an avenue 10 is mounted an image capture device 15. The road 10 can also be one of a street, a highway or any pathway that allows vehicles to pass through. The system 100 allows for automated monitoring of traffic, especially in a specific zone along the road 10 called an enforcement zone 20. The image capture device 15 can be mounted along the side of the road 10 or somewhere in between the lanes of the road or anywhere along the width of the road 10. The position of the mounting of the image capture device 15 is selected in such a way that the enforcement zone 20 can be viewed and imaged by the image capture device 15. The enforcement zone 20 in FIG 1 is represented by a fuzzy shape, which is purely for the sake of explanation. The different types of enforcement zones 20 will be described hereinafter.

The image capture device 15 is mounted on a pole or a suitable support erected near the road 10 or inside a width of the road 10, which is understood by a person skilled in the art. The image capture device is a camera that is capable of capturing still images and moving images. Still images refer to still images or static images, which are exactly the same as photos. Moving images are dynamic and refer to movies or videos. The cameras are equipped with electronic sensors such as CCD sensors or CMOS sensors. The type of sensor is not limited to the above and can be any other type as well. The output of the image capture device 15 can be monochrome or colour. The camera can be a Pan Tilt Zoom camera (PTZ) or a fixed camera. A PTZ camera is capable of remote directional and zoom control. In other words, the PTZ camera can be controlled from a remote location to make the camera rotate in at least one axis such that the field of view can be changed. Moreover, the PTZ camera can also be controlled from a remote location to adjust the zoom of the camera. Rotating the camera in at least one axis allows the camera to view and capture images and videos from a wide area or a wide field of view. For instance, in the case of multiple enforcement zones, the camera can move from one enforcement zone to the next by rotating along at least one axis. Adjusting the zoom of the PTZ camera from a remote location can be used to capture better quality images of a specific feature of any automobile on the road 10. For instance, the specific feature can be a licence plate or the shape of the hood of any other automobile body part. Fixed cameras always view and capture a fixed field of view only. Fixed cameras can be used where there is only one enforcement zone in the vicinity of the image capture device 15. The camera can also be a day night camera or an Infra red night vision camera. The day night camera and an Infra red night vision camera allow to capture images in low light conditions, which is understood by the person skilled in the art.

As described above, the image capture device 15 is configured to capture visual information associated with the traffic. As described earlier, the image capture device is directed at the traffic to capture the visual information. The visual information refers to still images and/or moving images of the vehicles in the traffic, specifically automobiles. The automobile types can be cars, motorcycles, scooters, light cargo vehicles, heavy cargo vehicles and buses. Cars can be sedans, MPVs, SUVs etc. Light cargo vehicles can be vans, pickups etc. Heavy cargo goods vehicles can be lorries, trucks, trailers etc. Buses can be single deck, double deck, coaches, bendy buses etc. The image capture device is powered by a battery pack which is located in the vicinity of the image capture device. Alternatively, since the image capture device is mounted in the vicinity of roads or avenues, where sunlight is plenty, solar panels can be used to provide power to the image capture device.

The system further comprises a transmission link for transmitting the visual information by the image capture device to a storage device. The storage device is further configured to receive the visual information from the image capture device and also for storing a violation detection software routine to determine whether an element of the traffic is in violation of a traffic protocol associated with the violation detection software routine. The above system features are elaborated further below.

The storage device can be a camera storage device 25 in the image capture device. The storage device 25 can be a Secure Digital Card (SD), Solid State Drive storage (SSD) or a Compact Flash (CF) or any kind of non-volatile memory device. The capacity of the storage device 25 is decided based on the location of the camera or the image capture device 15 and the amount of visual information it is designed to capture. The camera 15 in this case can be a smart camera. A smart camera in other words is an intelligent camera, which has image capturing, processing and communication capabilities built into the same device. Application of the smart camera for this system 100 will be described hereinafter. The storage device 25 is connected to at least the image sensor of the image capture device 15 by a camera transmission link 30, which can be a suitable electronic bus for data transfer, which is readily understood by the person skilled in the art.

Alternatively, the storage device can be a first storage device 35 in a first computing system 40. The first computing system 40 can be a normal computer or an industrial grade computer which can tolerate rugged conditions. The first computing system 40 is located in proximity to the image capture device 15. Proximity here can refer to a distance range of 5 metres to 10 metres. The image capture device 15 is coupled or electrically/electronically linked to the first computing system 40 through a first transmission link 45. The first transmission link 45 can be a wired transmission link. Wired transmission links can be by way of telecommunication cables, which are readily understood by a person skilled in the art. Alternatively, the first transmission link 45 can be a wireless transmission link. Wireless transmission links can be a Wi-Fi link etc, which is readily understood by the person skilled in the art. The first computing system 40 can also be referred to as a front-end computer or front-end computing system or a front-end system. The software applications installed and running in the first computing system 40 will be described hereinafter later.

Further alternatively, the storage device can be a second storage device 50 in a second computing system 55. The second computing system 55 can be a normal computer or an industrial grade computer which can tolerate rugged conditions. The second computing system 55 is located spatially away from the image capture device 15 and the first computing system 40. Spatially away here refers to a distance range of more than 10 metres. The image capture device 15 is coupled or electrically/electronically linked to the second computing system 55 through a second transmission link 60. The second transmission link 60 can be a wired transmission link. Wired transmission links can be by way of telecommunication cables, which are readily understood by a person skilled in the art. Alternatively, the second transmission link 60 can be a wireless transmission link. Wireless transmission links can be a Wi-Fi link etc, which is readily understood by the person skilled in the art. The second computing system 55 can also be referred to as a back-end computer or a back-end computing system or a back-end system. The software applications installed and running in the second computing system 55 will be described hereinafter later. Further, the first computing system 40 and the second computing system 55 is coupled or electrically/electronically linked through a transmission pathway 65. The transmission pathway 65 can be wired. A wired transmission pathway can be by way of telecommunication cables, which are readily understood by a person skilled in the art. Alternatively, the transmission pathway 65 can be wireless. A wireless transmission pathway can be a Wi-Fi link etc, which is readily understood by the person skilled in the art. The purpose of the transmission pathway 65 is to transfer data, including the visual information captured by the image capture device 15 between the first computing system 40 and the second computing system 55.

FIG 2 shows a high level architecture of a system 200 for automated monitoring of traffic with a first image capture device 15 and a second image capture device 15^'. As illustrated in FIG 2, the physical architecture, the storage architecture and the data transmission architecture involving the first image capture device 15, the first computing system 40 and the second computing system 55 are similar to the above description in system 100.

Further, in the system 200, there is a camera storage device 25^' in the second image capture device 15^'. The storage device 25^' is connected to at least the image sensor of the image capture device 15^' by a camera transmission link 30^', which can be a suitable electronic bus for data transfer, which is readily understood by the person skilled in the art.

Additionally, there is a storage device 35^' in a third computing system 40^'. The third computing system 40^' can be a normal computer or an industrial grade computer which can tolerate rugged conditions. The third computing system 40^' is located in proximity to the second image capture device 15^'. Proximity here can refer to a distance range of 5 metres to 10 metres. The second image capture device 15^' is coupled or electrically/electronically linked to the third computing system 40^' through a first transmission link 45^' of the second image capture device 15^'. The first transmission link 45^' can be a wired transmission link. Wired transmission links can be by way of telecommunication cables, which are readily understood by a person skilled in the art. Alternatively, the first transmission link 45^' can be a wireless transmission link. Wireless transmission links can be a Wi-Fi link etc, which is readily understood by the person skilled in the art. The third computing system 40^' can also be referred to as a front-end computer or front-end computing system or a front-end system. The software applications installed and running in the third computing system 40^' will be described hereinafter later. Further, the second image capture device 15^' is connected to the second computing system 55 by a second transmission link 60^' of the second image capture device 15^'. In the system 200, the first image capture device 15 and the first computing system 40 share the same back-end computer or back-end computing system or back-end system with the second image capture device 15^' and the third computing system 40^'. In this case, the back-end computing system is the second computing device 55.

FIG 3 shows a high level architecture of a system 300 for automated monitoring of traffic with a first image capture device 15 and a second image capture device 15^'. As illustrated in FIG 3, the physical architecture, the storage architecture and the data transmission architecture involving the first image capture device 15, the first computing system 40 and the second computing system 55 are similar to the above description in system 100. Further, there is a second image capture device 15^'. In the system 300, both the first image capture device 15 and the second image capture device 15^' share the first computing system 40 and the second computing system 55. There is a camera storage device 25^' in the second image capture device 15^'. The storage device 25^' is connected to at least the image sensor of the image capture device 15^' by a camera transmission link 30^', which can be a suitable electronic bus for data transfer, which is readily understood by the person skilled in the art.

The second image capture device 15^' is coupled or electrically/electronically linked to the first computing system 40 through a first transmission link 45^' of the second image capture device 15^'. The first transmission link 45^' can be a wired transmission link. Wired transmission links can be by way of telecommunication cables, which are readily understood by a person skilled in the art. Alternatively, the first transmission link 45^' can be a wireless transmission link. Wireless transmission links can be a Wi-Fi link etc, which is readily understood by the person skilled in the art.

Similarly, the second image capture device 15^' is coupled or electrically/electronically linked to the second computing system 55 through a second transmission link 60^' of the second image capture device 15^'. The second transmission link 60^' can be a wired transmission link. Wired transmission links can be by way of telecommunication cables, which are readily understood by a person skilled in the art. Alternatively, the second transmission link 60^' can be a wireless transmission link. Wireless transmission links can be a Wi-Fi link etc, which is readily understood by the person skilled in the art.

FIG 4 shows an exemplary arrangement of two image capture devices 15 and 15^' monitoring two lanes of the road 10. It is apparent from FIG 4 that both the image capture devices 15 and 15^' cover two lanes of the road. This is to build redundancy into the systems described above. If the image capture device 15 and 15^' are tracking a particular vehicle on the road and there is another vehicle between the tracked vehicle and one of the image capture devices, then the other image capture device can be used to track the vehicle and retrieve the licence plate information. The two image capture devices 15 and 15^' can also be used if there are two vehicles in the enforcement zone and both need to be tracked, such that each image capture device tracks one vehicle each. In a two image capture device setup, the image capture devices are set up such that each have a coverage of approximately 100m of the enforcement zone and that they are setup approximately 150m apart.

The systems described above for automated monitoring of traffic monitor whether the vehicles in the traffic adhere to a traffic protocol, which includes guidelines for movement and flow of traffic. Not following the guidelines can lead to traffic violations. An example of a traffic violation is parking a vehicle in an enforcement zone.

The enforcement zone is described below. Enforcement zones are sections of roads or avenues that have enforcement lines. The enforcement lines can be on any part of the road, but preferably along the edge of the roads or avenues. FIGs 5(a) - 5(d) shows exemplary enforcement lines along a section of a road or avenue. FIG 5(a) shows double lines in an enforcement zone 105. The colour of the lines can be white or yellow. However, the colour of the lines is not limited to white or yellow, but can be any other colour as well. FIG 5(b) shows a single line in an enforcement zone 1 10. The colour of the line can be yellow. However, the line can be of any other colour as well and is not limited to yellow. FIG 5(c) shows a single zig zag line in an enforcement zone 1 15. The colour of the line can be yellow. However, the line can be of any other colour as well and is not limited to yellow. FIG 5(d) shows a double zig zag line in an enforcement zone 120. The colour of the lines can be yellow. However, the colour of the lines is not limited to yellow, but can be any other colour as well.

To elaborate on the traffic violation as described above, if any vehicle is present in the enforcement zone for more than a specified time period (T), then the vehicle has violated the traffic protocol. For instance, the presence of a vehicle in the enforcement zone can be caused by a vehicle that is moving or moving slowly in the enforcement zone. The presence of the vehicle in the enforcement zone can also be caused by a vehicle that is parked in the enforcement zone. For instance, the time period can be 10 seconds or 60 seconds or any other number. The time period in relation to the systems described above will be described hereinafter when the working of the systems is explained. Software components of the system for automated monitoring of traffic

The software components of the system for automated monitoring of traffic are described hereinafter.

(a) Vehicle Recognition Module

FIG 6a shows a block diagram of a vehicle recognition module in association with a vehicle image database. As illustrated in FIG 6a, a vehicle recognition module 125 facilitates identification or recognition of vehicles in the image captured by the image capture device or both the image capture devices. As illustrated in FIG 6, there is a vehicle image database 130 associated with the vehicle recognition module 25. The vehicle image database 130 comprises a shape database 135, a colour database 140, a logo database 145, a licence plate database 147 and a model database 150.

The shape database 135, as the name suggests comprises a collection of images of different types of automobiles described earlier. For each automobile type, images from different angles and perspectives are compiled and made available in the shape database 135. If there is any shape that is unique to a particular type of automobile, different images of that particular shape, again from different angles and perspectives are compiled and made available in the shape database. For instance, a very unique shape is that of the Volkswagen Beetle which is very familiar as well.

The colour database 140, as the name suggests comprises a full range of different colours that automobiles are painted with. The colour range comprises very subtle colour differences as well.

The logo database 145 comprises images of logos of different automobiles. The logos of the some automobiles are very easily distinguishable. For instance Ferrari has-a logo of a horse on a single leg. AUDI has four interlinked circles. The images of these logos in the logo database 145 are also taken from different angles and perspectives.

The licence plate database 147 comprises images of licence plates with characters in them, representing different characters, including numerals and alphabets. The images of the licence plates in the licence plate database 147 are taken from different angles and perspectives. Licence plates from different types of automobiles are segregated in groups within this database for easy identification of the vehicle type, which will be described below. The model database 150 comprises images of different model names/numbers for different automobiles. For example, model names are usually found on the back of the cars, and for some trucks in the front. The images of these model names/numbers are also captured from different angles and perspectives.

The process of vehicle recognition is performed as follows. FIG 6(b) shows a process for recognition of a vehicle 160. As illustrated in FIG 6(b), the process 160 comprises a step 165 involving detection of objects in consecutive images or frames received from the image capture device or stored in any of the camera storage device 25, 25^', the first storage device 35 or 35^' and the second storage device 50. The images stored in the camera storage device 25, 25^', the first storage device 35 or 35^' or the second storage device 50 have been captured by the image capture device or both the image capture devices. Object detection is readily understood by the person skilled in the art and requires no further explanation. This step determines if there is any object, specifically a vehicle in the latest frame, which is represented by a shape.

Once the presence of an object, specifically a vehicle is determined in step 165, the process moves to a next step 170 which involves identifying unique features in the shape that has been determined in the step 65. This involves the vehicle image database 130 and the various databases in the database 130. The shape of the vehicle determined in step 165 is compared with the various images from the shape database 135, logo database 145, licence plate database 147 and the model database 150 to determine whether it is a heavy vehicle or a light automobile, the model and make of the automobile etc.

The process 160 now moves on to the third step which is step 175, which comprises extracting the licence plate number of the automobile. The types of licence plates covered by this module include the following - standard licence plates of a country including different font type, font size, embossed fonts, different colours and varying reflectivity, off-peak and weekend car licence plates, foreign vehicle licence plate and others including R&D licence and classic car licence plates.

If suppose the shape in a particular image or frame is not clear enough for identification or recognition, then the subsequent frames are subjected to the same process till the above determination is achieved.

The vehicle recognition module can process multiple shapes or vehicles in a single image simultaneously. The vehicle is also referred to as an element of the traffic. The extracted and the determined information are stored in the storage device from which the images or frames were retrieved for the process 160. The vehicle recognition module is installed and runs in any one of the first computing system 40, the third computing system 40^', the second computing system 55 and the image capture device. In other words, this module is installed and runs in any one of the front-end computing system, the back-end computing system and image capture device. If the system comprises more than one image capture device, this module can be installed and run in more than one image capture device.

The image processing techniques used for object detection and recognition of the shape of the vehicle in the vehicle recognition module comprises image subtraction, background subtraction, object segmentation, edge and contour detection etc, which are readily understood by a person skilled in the art.

The image capture device can also be viewing and capturing visual information, wherein the field of view includes areas and zones outside of the enforcement zones 20. In this case, before recognition or identification of the vehicle, the enforcement zones have to be extracted from the visual information. The enforcement zones are identified on the basis of single lines, double lines and zig zag lines.

Enforcement zone identification image processing routines or algorithms try to match objects in the visual information with single lines, double lines, zig zag lines and any other identification criteria that are configured in the system. Once these are matched, the enxt step is to extract the enforcement zone and the objects or vehicles in the enforcement zone from the full image or frame of the visual information and subject it to further processing for violation detection or determination.

As described earlier, the visual information comprises still images and moving images. Moving images which comprise movies and videos are fragmented or split into individual still image frames and subjected to image processing routines and algorithms in the system being described and the method that is described below.

(b) Vehicle Tracking Module

The vehicle tracking module facilitates locking on to the identified or recognized objects or vehicles in the vehicle recognition module and track the vehicle's presence or track the vehicle's movements in the captured images or frames. Additionally, the vehicle tracking module also tracks the length of time the vehicle is in the images or frames captured. Once the vehicle recognition module determines the presence of a vehicle in the images or frames, the vehicle tracking module is invoked. Immediately upon being invoked, the vehicle tracking module starts a time counter to measure the length of time the vehicle is in the frames captured or in the enforcement zone 20. The vehicle tracking module identifies the shape of the vehicle in the successive frames for purposes of tracking. The vehicle tracking module determines the presence or absence of the vehicle by the presence or absence of the shape of the vehicle.

Upon determining the absence of the shape of the vehicle in the captured images or frames, the time counter is stopped. FIG 7 shows the determination of violation of a traffic protocol in the vehicle tracking module. As illustrated in FIG 7, the time elapsed by a vehicle in the enforcement zone is determined as t by the vehicle tracking module in a step 185. In the next step 190, the time elapsed t is compared with the specified time period T. The specified time period T is configurable in the vehicle tracking module and can be varied as and when necessary. If t is less than T, then there is no violation of the traffic protocol as illustrated in step 195. If t is greater than T, then there is a violation of the traffic protocol as illustrated in step 200. The vehicle tracking module is installed and runs in any one of the first computing system 40, the third computing system 40^', the second computing system 55 and the image capture device. In other words, this module is installed and runs in any one of the front-end computing system, the back-end computing system and image capture device. If the system comprises more than one image capture device, this module can be installed and run in more than one image capture device.

The vehicle in the enforcement zone can either be moving or stationery.

The vehicle tracking module can process multiple shapes or vehicles in a single image simultaneously.

(c) Violation Detection and Analysis Module

The violation detection and analysis module is installed and runs in the back-end computing systems. This module facilitates compiling the violation videos, analyses to determine and specify the violation committed, determines the date and time the violation was committed as well as the location and area the violation was committed.

FIG 8 shows a block diagram of a violation detection and analysis module 205. As illustrated in FIG 8, the violation detection and analysis module 205 comprises a step 210 in which the storage device 50 of the second computing system 55 receives the violation videos and images from the front-end computing systems or the camera storage devices. The violation videos and images are received with time stamps and location stamps. The process 205 then proceeds to a next step 215 in which the violation snapshots are extracted from the set of violation images or from processing the violation videos, which are readily understood by the person skilled in the art. The process 205 then proceeds to a next step 220 in which the licence plate number of the violating automobile is extracted from the videos and/or images, which is performed by suitable image processing methods which are understood by the person skilled in the art. The process 205 then proceeds to a next step 225 in which the relevant violation snapshots, images and videos are stored with the captured licence plate number and other suitable information required for reporting the violation and enforcing the traffic law against the violation.

(d) Violation Reporting and Summons Module

The violation reporting and summons module is installed and runs in the back-end computing systems. This module facilitates reporting the violation along with the associated information of the vehicle or the element of traffic like the time, location, licence plate number as evidence to the traffic enforcement arm or the traffic law enforcement centre or the registered owner of the vehicle. This module has provisions for generating daily reports, weekly reports, monthly reports and annual reports. Reports can also be generated between any two dates.

Functional aspects of a method of automated monitoring of traffic

FIG 9 shows a method of automated monitoring of traffic 230. Firstly, the method or process 230 comprises a step 235 of capturing visual information associated with the traffic by the image capture device 15. The image capture device is being directed at the- traffic.

The process 230 now moves to a second step 240 which transmits the visual information captured by the image capture device to the storage device by the transmission link. As explained earlier, the storage device can be in the image capture device, the front-end computing systems or the back-end computing systems. The transmission of visual information by the image capture device to the storage device is through the transmission link, which has been explained earlier. The process 230 now moves to a third step 245 in which the visual information transmitted is processed by a violation detection software routine to determine if there is any violation of the traffic protocol or traffic law.

To elaborate on the processing step 245, the violation detection software routine comprises the vehicle recognition module and the vehicle tracking module. There is data flow between the vehicle recognition module and the vehicle tracking module so as to determine if there is any violation of the traffic protocol or traffic law by any vehicle that enters the enforcement zone. As soon as the vehicle recognition module identifies or recognizes a vehicle in the enforcement zone, instructions are sent to the vehicle tracking module to lock onto the shape of the vehicle and start the time counter for measuring time elapsed t, which is the time for which the vehicle stays in the enforcement zone, which may be by moving slowly or being stationary. The vehicle tracking module locks on to the shape of the vehicle which progresses and recedes in size as the vehicle moves along the enforcement zone. As is apparent, the process 230 keeps running continuously such that the image is captured by the image capture device, transmitted and processed by the violation detection software routine. The process of determining violation has been described earlier with reference to the vehicle tracking module.

When control is passed to the vehicle tracking module for tracking and determining violation, the vehicle recognition module still keeps running to see if there is any new vehicle that is entering the enforcement zone. If any new vehicle has entered the enforcement zone, a new instance of the vehicle tracking module is opened such that more than one instance of the vehicle tracking module is running at the same time. Any instance of the vehicle tracking module terminates when the vehicle that is being tracked moves out of the enforcement zone. Hence, it is apparent that the visual information captured and transmitted is processed by the vehicle recognition module and all the instances of the vehicle tracking module that are running.

As described earlier, the vehicle recognition module extracts the licence plate number of the vehicle in the images or frames. A copy of all the images processed by an instance of the vehicle tracking module, which are images with a vehicle in different locations along the enforcement zone are simultaneously stored in the storage device in which the violation detection software routine is running, resulting in a set of images. If violation is determined by the vehicle tracking module, the set of images is processed by any suitable database software process to package the set of images determined to be violating the traffic protocol along with a header which comprises information on at least the shape, colour, model, logo and the licence plate number. The shape, colour, model, logo and the licence plate number has been extracted by the vehicle recognition module.

The visual information captured by the image capture device or devices have time information embedded in the image, such as a time stamp.

As described earlier, the smart camera is an intelligent camera, which has image capturing, processing and communication capabilities built into the same device. The image capture device 15 can be a smart camera. In that case, the violation detection software routine can be installed in the image capture device 15 and the processing of the visual information can be done in the image capture device 15 itself.

As described earlier, the violation detection software routine can be installed and run in the camera storage device 25 or 25^'. In this case, the visual information is transmitted to the camera storage device 25 or 25^' by the camera transmission link 30 or 30^'. Alternatively, the violation detection software routine can be installed and run in the first storage device 35 or 35^'. In this case, the visual information is transmitted to the first storage device 35 or 35^' from the image capture device 15 or 15^' by the first transmission link 45 or 45^'. Further alternatively, the violation detection software routine can be installed and run in the second storage device 50. In this case, the visual information is transmitted to the second storage device 50 from the image capture device 15 or 15^' by the second transmission link 60 or 60^'.

Alternatively, the violation detection software routine can be installed and run in the second storage device 50 and the visual information is transmitted to the second storage device 50 from the first storage devices 35 or 35^' through the transmission pathway 65 or 65^'. This is to build redundancy and an alternate pathway to the second computing system as the visual information are being processed are in real time by the method and system described above. The transmission pathway 65 or 65^' can be used in routine circumstances or during periods of transmission overload or a glitch on the second transmission link 60 or 60^'.

Once violation has been determined, the control transfers to the violation detection and analysis module 205 and subsequently to the violation reporting and summons module.

As visual information including still images and moving images occupy a lot of memory and storage space and are substantially big to transmit around between the image capture device and the first and second computing systems, described below are a few methods to optimize network bandwidth usage for transmission and storage space usage. (a) Selective Data Streaming Technique

In this method, the enforcement zone has presence sensors using the principles of Infra Red (IR), Ultrasonic and Laser to detect the presence of objects such as vehicles. The working principle of presence sensors such as IR, ultrasound and Laser are readily known to the person skilled in the art. The outputs from these sensors are transmitted to the front-end computing systems or to the image capture device itself, if the image capture device is a smart camera. Images are captured continuously by the image capture device and as the vehicle moves along the enforcement zone 20, the sensors placed at different locations along the path in which the vehicle moves, and specifically along the enforcement zone 20 detect the presence of the vehicle in the sensors vicinity, the time points of which are recorded by the image capture device 15 or 15^' and the front-end computing systems. The visual information or specifically images with embedded time information equivalent to the time points recorded by the image capture device 15 or 15^' and the front-end computing systems are segregated as a set of selected visual information or selected visual information. The set of selected visual information or selected visual information occupies less storage space when compared to the size of all the images captured by the image capture device 15 or 15^' because it is a subset of the visual information captured by the image capture device 15 or 15^'.

If the visual information is required to be transmitted over the first transmission link 45 or 45^', second transmission links 60 or 60^' or the transmission pathway 65 or 65^', the selected visual information is transmitted, the end result being reduction in the utilized network bandwidth and the speed of transmission, as the speed of transmission and processing is an important consideration in the working of the method and system. The selected visual information is then processed to determine violation of the traffic protocol or traffic law.

Alternate to the sensors, software detection can also be used. This involves processing the visual information captured in such a way that visual information showing the vehicle at a predetermined few locations along the enforcement zone are formed into the set of selected visual information or selected visual information. The advantage with using presence sensors is that the amount of computing power required to generate the selected visual information is substantially reduced as there is no image processing involved.

An example implementation of this technique wherein transmitting the visual information by the image capture device to the storage device by the transmission link comprises the first step of transmitting the visual information by the image capture device 15 or 15^' to the first storage device 35 or 35^' by the first transmission link 45 or 45^'. The second step comprises selecting a subset of the visual information by way of processing the transmitted visual information based on predetermined selection criteria. The processing based on predetermined selection criteria is performed by a selection software routine in the first computing system. One example of the predetermined selection criteria can be the same as what has been explained earlier in selecting the selected visual information based on the time points recorded by the image capture device 15 or 15^' and the front-end computing systems. Another example can be selecting images with timestamps equivalent to the time points recorded and a few other time points before and after the recorded time points. The third step is the transmission of the selected visual information to the second storage device 50 for storage or processing or both. This reduces the amount of visual information transmitted and processed thus reducing the network bandwidth utilized for transmission and processing power.

Another example implementation of this technique comprises generating the selected visual information in the image capture device 15 or 15^' and transmitting the selected visual information to the first storage device 35 or 35^'. Yet another example implementation of this technique comprises generating the selected visual information in the image capture device 15 or 15^' and transmitting the selected visual information to the second storage device 50.

(b) Variable Resolution Image Processing Technique

In this technique, two streams of visual information are generated - one stream that has high resolution and another stream that has low resolution. One way of doing it is to capture high resolution visual information by the image capture device and convert them into low resolution, but at the same time maintaining the high resolution visual information. Another way is for the image capture device to generate two streams of visual information - one being high resolution and the other being low resolution.

Further, transmitting the visual information by the image capture device to the storage device by the transmission link comprises - transmitting the high resolution visual information to the camera storage device 25 or 25^' or the first storage device 35 or 35^'. Further, this also involves transmitting the low resolution visual information to the second storage device 50. The advantage of this is limited utilization of network bandwidth as the high resolution visual information is stored in the image capture device or the first computing system. Processing the visual information by the violation detection software routine to determine whether the element of the traffic is in violation of the traffic protocol comprises - processing the low resolution visual information at the second computing system 55 by the violation detection software routine resulting in an information packet to facilitate determine a subset of the high resolution visual information required for further processing. In other words, this involves processing the low resolution visual information at the second computing system 55 to determine which images or frames in high resolution format will be suitable for detection of violation, which can be based on the position of the vehicle on the frame or image and its orientation with respect to the camera. The time stamps of these images or frames or file names or both are compiled and formed into an information packet. The information packet is then transmitted to the second computing system to select the subset of the high resolution visual information from the storage device in one of the image capture device and the first computing system. Based on the contents of the information packet, the high resolution visual information is sieved through to select the subset of the high resolution visual information. In the next step, the subset of the high resolution visual information is then transmitted to the storage device in the second computing system for further processing by the violation detection software routine to determine violation of the traffic protocol or the traffic law.

This technique facilitates reducing network bandwidth utilization as the full set of bulky and voluminous high resolution visual information is not transmitted to the back end computing system or second computing device.

(c) High Resolution Snapshots Technique

In this technique, capturing visual information associated with the traffic by the image capture device comprises capturing a set of high resolution still images of the traffic by the image capture device 15 or 15^'. Further, in this technique, transmitting the visual information by the image capture device 15 or 15^' to the storage device comprises transmitting the set of high resolution still images by the image capture 15 or 15^' to the second computing system 55.

Further, in this technique, the set of high resolution still images of the traffic are captured by the image capture device 15 or 15^' in a predetermined time interval. In other words, the high resolution still images are taken in intervals and the high resolution still images captured during any single capturing phase of the image capture device 15 or 15^' constitutes the set of high resolution still images. Further, in this technique, before the set of high resolution still images is captured by the image capture device 15 or 15^', a predetermined resolution of the image capture device 15 or 15^' is set.

The method described above can also be performed with two image capture devices 15 and 15^'. Two image capture devices are usually used when the view of either a front or a back of the vehicle is blocked by another vehicle in the enforcement zone. Sometimes, the two image capture devices 15 and 15^' are used so as to track the vehicle in the enforcement zone from both the front and the rear of the vehicle. The principles of operation and working of the methods and techniques described above are the same regardless of whether the image capture device 15 or 15^' are used. Depending on the image capture device used, the front end computing system associated with the image capture device will be used. In the event of both image capture device 15 and 15^' being used, the principles of operation and working of the methods and techniques described above will be applied to each of the image capture devices and its associated front-end computing systems and back end computing systems.

FIG 10(a) - 10(f) shows a representative illustration of a vehicle being detected and tracked along the enforcement zone 20.

FIG 10(a) shows a car 300 about to enter the enforcement zone 20. The image capture device 15 captures visual information, being directed at the enforcement zone 20. When the image captured by the image capture device 15 is processed by the vehicle recognition module using image subtraction or other image processing techniques, no object or shape will be detected by the vehicle recognition module and will be available to successively grab and process successively captured visual information.

FIG 10(b) shows the car 300 just entering the enforcement zone 20. The image capture device 15 captures visual information, being directed at the enforcement zone 20. When the image captured by the image capture device 15 is processed by the vehicle recognition module using image subtraction or other image processing techniques, a shape is detected by the vehicle recognition module, which proceeds to match the shape with the images stored in the shape database 135, colour database 140, logo database 145 and model database 150. Since the shape in the image acquired is not complete and full to extract all the information about the car 300 entering the enforcement zone 200, the vehicle recognition module waits for the successive visual information captured by the image capture device 15. FIG 10(c) shows the car 300 inside the enforcement zone 20. The image capture device 15 now captures visual information which comprises the full image of the car as a shape in the enforcement zone. The vehicle recognition module proceeds to match the shape with the images stored in the shape database 135, colour database 140, logo database 145 and model database 150. The shape of the car 300 is possible matched with at least one of a shape, logo, model and colour from the databases 135, 140, 145 and 150. The information about the car is extracted by the vehicle recognition module and an instance of the vehicle tracking module is initiated. The vehicle tracking module first starts a time counter for the purposes of detecting violation later when the car 300 moves out of the enforcement zone 20. The vehicle tracking module also stores at least one image of the shape in the captured visual information for the purposes of tracking.

FIG 10(d) and (e) show the car 300 well inside the enforcement zone 20. The vehicle tracking module processes each successive image in the visual information being captured and tries to match the shape in the captured image to what has been stored in the previous step. By matching the shape, the vehicle tracking module knows that the object or shape it is tracking is still in the enforcement zone 20.

FIG 10(f) shows the car 300 fully outside the enforcement zone 20. The vehicle tracking module no longer has any shape to match with and thus stops the time counter. The elapsed time t is then compared with a specified time period T to detect violation, which has been explained earlier.

FIG 1 1 (a) - 1 1 (b) shows another representative illustration of a stationary vehicle being detected and tracked along the enforcement zone 20.

FIG 1 1 (a) shows the car 300 inside the enforcement zone captured at time point ti sec. The image capture device captures the image at time point d seconds.

FIG 1 1 (b) shows the car 300 inside the enforcement zone captured at time point t₂ sec. The image capture device captures the image at time point t₂ seconds.

In this representative illustration, the car is stationary and the vehicle tracking module tracks the car till the time it is inside the enforcement zone 20. The elapsed time is then compared with a specified time period T to detect violation, which has been explained earlier. It is to be understood that the foregoing description is intended to be purely illustrative of the principles of the disclosed techniques, rather than exhaustive thereof, and that changes and variations will be apparent to those skilled in the art, and that the present invention is not intended to be limited other than as expressly set forth in the following claims.

Claims

1. A method of automated monitoring of traffic, comprising:

capturing visual information associated with the traffic by an image capture device, the image capture device being directed at the traffic;

transmitting the visual information by the image capture device to a storage device by a transmission link; and

processing the visual information by a violation detection software routine in the storage device to determine whether an element of the traffic is in violation of a traffic protocol associated with the violation detection software routine.

2. The method of automated monitoring of traffic as in claim 1 , wherein transmitting the visual information by the image capture device to the storage device by the transmission link comprises transmitting the visual information by the image capture device to a first storage device in a first computing system by a first transmission link, the first computing system located in proximity to the image capture device.

3. The method of automated monitoring of traffic as in claim 1 , wherein transmitting the visual information by the image capture device to the storage device by the transmission link comprises transmitting the visual information by the image capture device to a camera storage device in the image capture device by a camera transmission link.

4. The method of automated monitoring of traffic as in claim 1 , wherein transmitting the visual information by the image capture device to the storage device by the transmission link comprises transmitting the visual information by the image capture device to a second storage device in a second computing system by a second transmission link, the second computing system located spatially away from the image capture device.

5. The method of automated monitoring of traffic as in claim 2, further comprising

transmitting the visual information from the first storage device in the first computing system to a second computing system by one of a wired transmission pathway and a wireless transmission pathway, the second computing system located spatially away from the image capture device and the first computing system.

6. The method of automated monitoring of traffic as in claim 1 , wherein transmitting the visual information by the image capture device to the storage device by the transmission link comprises: transmitting the visual information by the image capture device to a first storage device in a first computing system by a first transmission link, the first computing system located in proximity to the image capture device;

selecting a subset of the visual information based on predetermined selection criteria by a selection software routine in the first computing system, resulting in a selected visual information; and

transmitting the selected visual information to a second storage device in a second computing system by a transmission pathway, the second computing system located spatially away from the image capture device and the first computing system.

7. The method of automated monitoring of traffic as in claim 6, wherein selecting the subset of the visual information based on predetermined selection criteria by the selection software routine in the first computing system comprises selecting images having time information equivalent to a plurality of time points recorded when the presence of the vehicle is detected by at least one sensor located along a path in which the vehicle moves.

8. The method of automated monitoring of traffic as in claim 1 , further comprising generating low resolution visual information and high resolution visual information, wherein transmitting the visual information by the image capture device to the storage device by the transmission link comprises:

transmitting the high resolution visual information to one of a camera storage device in the image capture device by a camera transmission link and a first storage device in a first computing system by a first transmission link; and

transmitting the low resolution visual information to a second storage device located in a second computing system by a second transmission link;

and;

wherein processing the visual information by the violation detection software routine to determine whether the element of the traffic is in violation of the traffic protocol associated with the violation detection software routine comprises:

processing the low resolution visual information at the second computing system by the violation detection software routine resulting in an information packet to facilitate determine a subset of the high resolution visual information required for further processing;

transmitting the information packet to one of the image capture device and the first computing system to select the subset of the high resolution visual information from one of the camera storage device and the first storage device;

selecting the subset of the high resolution visual information based on the information packet; and transmitting the subset of the high resolution visual information to the second storage device in the second computing system through a transmission pathway for further processing by the violation detection software routine.

9. The method of automated monitoring of traffic as in any of the preceding claims, wherein the visual information comprises a set of images, each image in the set the images being one of a still image and a moving image.

10. The method of automated monitoring of traffic as in claim 1 , wherein capturing visual information associated with the traffic by the image capture device comprises:

capturing a set of high resolution still images of the traffic by the image capture device; and;

transmitting the visual information by the image capture device to a storage device by a transmission link comprises:

transmitting the set of high resolution still images by the image capture device to a computing system that is located spatially away from the image capture device.

11 . The method of automated monitoring of traffic as in claim 10, wherein capturing the set of high resolution still images of the traffic by the image capture device is achieved in a predetermined time interval.

12. The method of automated monitoring of traffic as in claim 10, wherein capturing the set of high resolution still images of the traffic by the image capture device is achieved with a predetermined resolution of the image capture device.

13. The method of automated monitoring of traffic as in claim 1 , wherein during processing the visual information by the violation detection software routine, information associated with the element of the traffic is generated.

14. The method of automated monitoring of traffic as in claim 13, wherein the information associated with the element of the traffic comprises at least one of licence plate number, model, logo and paint colour of the automobile in the automobile traffic.

15. The method of automated monitoring of traffic as in claim 13, further comprising:

reporting the violation and the information associated with the element of the traffic to a traffic law enforcement centre.

16. A system for automated monitoring of traffic, comprising: an image capture device configured to capture visual information associated with the traffic, the image capture device being directed at the traffic;

a transmission link for transmitting the visual information by the image capture device to a storage device; and

the storage device configured to receive the visual information from the image capture device and for storing a violation detection software routine to determine whether an element of the traffic is in violation of a traffic protocol associated with the violation detection software routine.

17. The system for automated monitoring of traffic as in claim 16, wherein the storage device is a camera storage device and wherein the transmission link is a camera transmission link, the camera transmission link configured to transmit the visual information by the image capture device to the camera storage device. 8. The system for automated monitoring of traffic as in claim 16, wherein the storage device is a first storage device in a first computing system, the first computing system located in proximity to the image capture device and wherein the transmission link is a first transmission link, the first transmission link configured to transmit the visual information by the image capture device to the first storage device.

19. The system for automated monitoring of traffic as in claim 16, wherein the storage device is a second storage device in a second computing system, the second computing system located spatially away from the image capture device and wherein the transmission link is a second transmission link, the second transmission link configured to transmit the visual information by the image capture device to the second storage device.

20. The system for automated monitoring of traffic as in claim 18, further comprising:

a second storage device in a second computing system, the second computing system located spatially away from the image capture device;

a second transmission link for transmitting the visual information by the image capture device to the second storage device; and

a transmission pathway for transmitting the visual information from the first storage device in the first computing system to the second storage device in the second computing system.