WO2016022020A1

WO2016022020A1 - System and method for detecting and reporting location of unilluminated streetlights

Info

Publication number: WO2016022020A1
Application number: PCT/MY2015/050080
Authority: WO
Inventors: Shang Li YUEN; Zulaikha BINTI KADIM; Norshuhada BINTI SAMUDIN; Hock Woon Hon
Original assignee: Mimos Berhad
Priority date: 2014-08-07
Filing date: 2015-07-29
Publication date: 2016-02-11

Abstract

The present invention provides a system having a video camera, a location sensor and a monitor for monitoring streetlights and reporting location of un-illuminated streetlights comprising; a video acquisition unit (2) to extract video images and convert them into a image frames from the camera; a location acquisition unit (4) to extract location and orientation information from the location sensor; a video storage (5) to record the image frames from the video acquisition unit (2); a location storage (6) to record the location and orientation information from the location acquisition unit (4); a database having a region of interest (ROI) information and streetlight locations information; a streetlight detector (8) to identify any unilluminated streetlight based on the streetlight locations information; a reporting unit (9) to display the location of unilluminated streetlights on the monitor (10); and a map generator unit (7), wherein the map generator (7) generates a map based on the region of interest (ROI) information and streetlight locations information using the image frames from the video storage (5) and location and orientation information from location acquisition unit (4) to display at the monitor (10), in which the streetlight detector (8) detect the unilluminated streetlight based on the map and report using reporting unit (9). Further, the streetlight detector (8) uses Hough transform technique to identify any unilluminated streetlight. Preferably, the map generator unit (7) is a geographic information system (GIS). Preferably, the location acquisition unit (4) is a Global Positioning System (GPS) received. Preferably, the streetlight detector is a photodetector.

Description

SYSTEM AND METHOD FOR DETECTING AND REPORTING LOCATION OF UNILLUMINATED STREETLIGHTS

FIELD OF INVENTION

The present invention relates to a system and method for monitoring streetlights and reporting the location of un-illuminated streetlight.

BACKGROUND ART

Streetlight is an important appliance to provide to user so that user could travel in low illumination environment. It could be dangerous for road user if there are no streetlights. Therefore, performing maintenance on the malfunction streetlight is very crucial for all road users. However, searching and identifying malfunction (un-illuminated) streetlights could be a tedious and time-consuming task because it is required manpower to travel around and verifying the streetlight condition one by one. This task become more difficult when searching area is expanded to city-size range.

In the prior art, some authorities today rely on the information reported by road user to identify the malfunction streetlight. However, in many cases, this information given by user only is not instantly and it is not accurate enough due to only road or street information is reported. These will delay the needed maintenance work to be carried out.

United States Patent No. 5095502 to Finzel discloses a unit for detection and signalizing of functional defects for public lighting. A unit characterized by the fact that it comprises, for the one part, an emitter module with which each street lamp is equipped, preferably located in its post, at the level of the junction box, the emitter module being equipped with a detector for current consumption using the alternative network of the supply line as the carrier of a detection signal, and for the other part, of a receiver module located in the supply cabinet of the same zone, this receiver module managed by a microprocessor which transmits a defect signal to the control room over a telephone line. This invention is of interest for manufacturers and installers of materials, equipment and components for public lighting. United States Patent Application No. 20130234862 to Michael A. Toth discloses a street light monitoring system comprising: a plurality of geographically spaced individual street lights; a plurality of light monitoring sensor nodes, wherein one said sensor node is associated with one said street light, each said sensor node detecting the illumination status of the associated street light; each said sensor node capable of wirelessly communicating with at least one other of said sensor nodes; at least one of said sensor nodes comprising a control node, wherein said control node is capable of wirelessly communicating with a remote lighting control and management system; wherein said sensor nodes and said at least one control node are capable of determining the optimum communication path through multiple said sensor nodes from any one said sensor node to said at least one control node, and wherein said sensor nodes and said at least on control node are capable of determining alternative communication paths from any sensor node to said at least on control node in the event said optimum communication pathway fails. Therefore an invention is proposed to detect the un-illuminated streetlights by analyzing the light spots in sequence of images and comparing them with the corresponding region in database.

SUMMARY OF INVENTION

The present invention provides a system having a video camera, a location sensor and a monitor for monitoring streetlights and reporting location of un-illuminated streetlights comprising; a video acquisition unit (2) to extract video images and convert them into a image frames from the camera; a location acquisition unit (4) to extract location and orientation information from the location sensor; a video storage

(5) to record the image frames from the video acquisition unit (2); a location storage

(6) to record the location and orientation information from the location acquisition unit (4); a database having a region of interest (ROI) information and streetlight locations information; a streetlight detector (8) to identify any unilluminated streetlight based on the streetlight locations information; a reporting unit (9) to display the location of unilluminated streetlights on the monitor (10); and a map generator unit (7), wherein the map generator (7) generates a map based on the region of interest (ROI) information and streetlight locations information using the image frames from the video storage (5) and location and orientation information from location acquisition unit (4) to display at the monitor (10), in which the streetlight detector (8) detect the unilluminated streetlight based on the map and report using reporting unit (9).

Further, a streetlight detector (8) uses Hough transform technique to identify any unilluminated streetlight.

Preferably, a map generator unit (7) is a geographic information system (GIS).

Preferably, a map generator unit (7) is a Web mapping unit.

Preferably, a map is a third-party web mapping service application. Preferably, a map is a map embedded on third-party websites.

Preferably, a location acquisition unit (4) is a Global Positioning System (GPS) received. Preferably, a location acquisition unit (4) is a navigation unit. Preferably, a video camera is a night view camera.

Preferably, a image is an infrared photography image. Preferably, a streetlight detector is a photodetector. Further aspect of the present invention provides a method for monitoring streetlights and reporting location of un-illuminated streetlights as claimed in claim 1, comprising step of: extracting video images and converting them into a image frames from the camera using a video acquisition unit (2); storing the image frames from the video acquisition unit (2) into a video storage (5); extracting location and orientation information from the location sensor using a location acquisition unit (4); storing the location and orientation information from the location acquisition unit (4) into a location storage (6); selecting a region of interest (ROI) information and streetlight locations from the video storage (5) and location storage (6); and storing the region of interest (ROI) information and streetlight locations into a database.

Another aspect of the present invention provides method for monitoring streetlights and reporting location of un-illuminated streetlights as claimed in claim 1, comprising step of: extracting video images and converting them into a image frames from the camera using a video acquisition unit (2); extracting location and orientation information from the location sensor using a location acquisition unit (4); storing the image frames from the video acquisition unit (2) into a video storage (5); storing the location and orientation information from the location acquisition unit (4) into a location storage (6); loading a region of interest (ROI) information and streetlight locations from a database; identifying any unilluminated streetlight based on the streetlight locations information using a streetlight detector (8); displaying the location of unilluminated streetlights on the monitor (10) using a reporting unit (9); generating a map based on the region of interest (ROI) information and streetlight locations information using the image frames from the video storage (5) and location and orientation information from location acquisition unit (4) to display at the monitor (10) using a map generator unit (7); and detecting the unilluminated streetlight based on the map using the streetlight detector (8) and reporting the unilluminated streetlight using reporting unit (9).

The present invention consists of features and a combination of parts hereinafter fully described and illustrated in the accompanying drawings, it being understood that various changes in the details may be made without departing from the scope of the invention or sacrificing any of the advantages of the present invention.

BRIEF DESCRIPTION OF PREFERRED EMBODIMENT

To further clarify various aspects of some embodiments of the present invention, a more particular description of the invention will be rendered by references to specific embodiments thereof, which are illustrated in the appended drawings. It is appreciated that these drawings depict only typical embodiments of the invention and are therefore not to be considered limiting of its scope. The invention will be described and explained with additional specificity and detail through the accompanying drawings in which:

Figure 1 illustrates the block diagram of streetlights monitoring system Figure 2 illustrates the process flow of streetlights monitoring system

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

As illustrated in Figure 1, a system for monitoring streetlights using night view aerial images, the system comprising of a video acquisition unit (2) to acquire video and convert video into frames for video analytics processing, an acquisition unit (4) to acquire location and orientation information from external sensor, a video recorder (5) to record the image frames from the camera sources, a location recorder (6) to record the location and orientation information from the sensor sources, a monitor to display the camera view (10) with and without the illuminated streetlight detection output such as overlay bounding box of the detected uniUuminated streetlight, wherein the monitoring streetlights , further comprising a 3D global map generator (7) to create a 3D global map which includes region of interest (ROI) and streetlight location by utilizing the image position and location sensor information, a streetlight detector (8) to identify the uniUuminated streetlight by analyzing the image contents and retrieving the corresponding location from 3D global map, a reporting unit (9) to report the location of uniUuminated streetlight by displaying on monitor view.

There are two major processes in this invention, an offline process using day view aerial image and an online process using night view aerial image. A database to store 3D location of streetlight is prepared in offline process. When uniUuminated streetlight is detected, the 3D location of this streetlight is displayed on screen and report to user.

As illustrated in Figure 2, a method for detecting and reporting location of uniUuminated streetlights using aerial view images, comprising of an offline process category for creating a 3D Global Map Ml which includes region of interest (ROI) and streetlight location by utilizing the image position and location sensor information, wherein the step of extracting region of interest (ROI) from image based on location sensor information and database includes the steps of receiving sequence of image from camera (11). Then position (longitude, latitude and elevation) and orientation (yaw, pitch and row) information of camera are received using external sensor(s) (12). In addition, intrinsic information (focus length, image center and etc) of camera is computed from camera. After collected these information, each image are tagged with the position, orientation and intrinsic information (13). Then region of interest (ROI) of road and streetlight position is needed to be determined in each image (14). Here, at the first capture image, user need to define the road ROI and streetlight location. Next, features from each captured image are computed and homography matrices between image k and k+1 for all images are computed using matched features between images k an k+1 for all images. After found homography matrices, the road ROI and streetlight location could be projected from first image to another image using computed homography matrices. Then a 3D global map which contains road ROI and streetlight location is generated. Here, using the position, orientation and intrinsic information from each camera, the triangulation for road ROI and streetlight is executed and 3D position of them is labelled on a 3D global map. For road ROI, all the 3D position in global map are connected. (4) After generated all 3D ROI road and streetlight location on a 3D global map Ml. At the end, this 3D global map Ml is stored into database (15).

On the other hand, the online process category for to detect unilluminated streetlights includes the steps of extracting ROI region from captured image based on location sensor and 3D global map (16). First, a sequence of image is captured from camera. For each captured image, position and orientation is retrieved using external location sensor(s) and intrinsic information is retrieved from camera. After that, road ROI from 3D global map Ml in database is projected into each captured image using corresponding position, orientation and intrinsic information. After projected the ROI, filter the region outside ROI by setting them as zero. Then number of non-black region inside the filtered image is checked. If number of non-black equals to zero, this image wouldn't not be processed. Next, converting the filtered image to binary image using binarization method and detecting the illuminated region from converted binary image (17). Before detecting the illuminated region, size of each white blob is measured and the blob which has the size smaller than a threshold (typical: Maximum size of blob - minimum size of blob * 0.1) is identified as noise and is filtered by setting this type of blob with value zero. Then due to the shape of illuminated region from streetlight is round shape, so circle detection is utilized to identify the illuminated region from streetlight. For each blob, identifying the circle properties using Hough transform and for those which is not a circle will be removed. Then this filtered binary image is stored into memory. This step process is repeated until number of stored binary image is larger than a threshold. Next, filtering the moving region (18). This is due to the illuminated region could be caused by the headlight of vehicle, so it is necessary to remove the moving region. To do this, first, all the contour of white blob in each stored binary is retrieved and projected onto a new 3D global map M2 (This 3D global map M2 has the size same as 3D global map, Ml) using position, orientation and intrinsic information. Then ADD operator is used to find the overlapped regions from all projected and filled contour in 3D global map M2. After implemented AND operator, overlapped region is identified as static region and non-overlapping region is identified as moving region. Here, all the non- overlapping regions are removed by setting them as value zero in 3D global map M2. Determining the unilluminated streetlight (19). Here, all streetlight location from 3D global map Ml is copied to 3D global map M2. Next, all centre positions and radius from all overlapped region is computed. From all computed radius from all overlapped region, a maximum radius, max_R is computed. Then a searching sphere which has size max_R ^Λ2 is used at each streetlight location in order to check is there any centre of region is within the searching sphere. If centre of overlapped is found at the searching sphere of streetlight location, then the corresponding streetlight location is identified as illuminated streetlight. Otherwise, it will be identified as unilluminated streetlight. Reporting location of unilluminated streetlight (20). For each found unilluminated streetlight from previous stage, their corresponding 3D geographic location (longitude, latitude and elevation) is retrieved from 3D Global Map M2 and this information is reported to user. The present invention may be embodied in other specific forms without departing from its essential characteristics. The described embodiments are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is, therefore indicated by the appended claims rather than by the foregoing description. All changes, which come within the meaning and range of equivalency of the claims, are to be embraced within their scope.

Claims

1. A system for monitoring streetlights and reporting location of un-illuminated streetlights comprising;

a video acquisition unit (2) to extract video images and convert them into a image frames from the camera;

a location acquisition unit (4) to extract location and orientation information from the location sensor;

a video storage (5) to record the image frames from the video acquisition unit (2);

a location storage (6) to record the location and orientation information from the location acquisition unit (4);

a database having a region of interest (ROI) information and streetlight locations information;

a streetlight detector (8) to identify any unilluminated streetlight based on the streetlight locations information;

a reporting unit (9) to display the location of unilluminated streetlights on the monitor (10); and

a map generator unit (7),

wherein the map generator (7) generates a map based on the region of interest

(ROI) information and streetlight locations information using the image frames from the video storage (5) and location and orientation information from location acquisition unit (4) to display at the monitor (10), in which the streetlight detector (8) detect the unilluminated streetlight based on the map and report using reporting unit (9).

2. A system for monitoring streetlights and reporting location of un-illuminated streetlights according to claims 1, wherein a streetlight detector (8) uses Hough transform technique to identify any unilluminated streetlight.

3. A system for monitoring streetlights and reporting location of un-illuminated streetlights according to claims 1, wherein a map generator unit (7) is a geographic information system (GIS).

4. A system for monitoring streetlights and reporting location of un-illuminated streetlights according to claims 1 , wherein a map generator unit (7) is a Web mapping unit.

5. A system for monitoring streetlights and reporting location of un-illuminated streetlights according to claims 1, wherein a location acquisition unit (4) is a Global Positioning System (GPS) received.

6. A system for monitoring streetlights and reporting location of un-illuminated streetlights according to claims 1 , wherein a image is an infrared photography image.

7. A system for monitoring streetlights and reporting location of un-illuminated streetlights according to claims 1 , wherein a streetlight detector is a photodetector.

8. The method for monitoring streetlights and reporting location of un- illuminated streetlights as claimed in claim 1, comprising step of:

extracting video images and converting them into a image frames from the camera using a video acquisition unit (2);

storing the image frames from the video acquisition unit (2) into a video storage (5);

extracting location and orientation information from the location sensor using a location acquisition unit (4);

storing the location and orientation information from the location acquisition unit (4) into a location storage (6);

selecting a region of interest (ROI) information and streetlight locations from the video storage (5) and location storage (6); and

storing the region of interest (ROI) information and streetlight locations into a database.

9. The method for monitoring streetlights and reporting location of un- illuminated streetlights as claimed in claim 1, comprising step of: extracting video images and converting them into a image frames from the camera using a video acquisition unit (2);

loading a region of interest (ROI) information and streetlight locations from a database;

identifying any unilluminated streetlight based on the streetlight locations information using a streetlight detector (8);

displaying the location of unilluminated streetlights on the monitor (10) using a reporting unit (9);

generating a map based on the region of interest (ROI) information and streetlight locations information using the image frames from the video storage (5) and location and orientation information from location acquisition unit (4) to display at the monitor (10) using a map generator unit (7); and

detecting the unilluminated streetlight based on the map using the streetlight detector (8) and reporting the unilluminated streetlight using reporting unit (9).