WO2015193470A1 - Mobile road sign reflectometer - Google Patents

Mobile road sign reflectometer Download PDF

Info

Publication number
WO2015193470A1
WO2015193470A1 PCT/EP2015/063809 EP2015063809W WO2015193470A1 WO 2015193470 A1 WO2015193470 A1 WO 2015193470A1 EP 2015063809 W EP2015063809 W EP 2015063809W WO 2015193470 A1 WO2015193470 A1 WO 2015193470A1
Authority
WO
WIPO (PCT)
Prior art keywords
road sign
colour
image
camera
photometric
Prior art date
Application number
PCT/EP2015/063809
Other languages
French (fr)
Inventor
Simon Mcloughlin
Catherine Deegan
Sean HAUGHEY
Original Assignee
Institute Of Technology Blanchardstown
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Institute Of Technology Blanchardstown filed Critical Institute Of Technology Blanchardstown
Publication of WO2015193470A1 publication Critical patent/WO2015193470A1/en

Links

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
    • G01N21/55—Specular reflectivity
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
    • G01N21/25—Colour; Spectral properties, i.e. comparison of effect of material on the light at two or more different wavelengths or wavelength bands
    • G01N21/27—Colour; Spectral properties, i.e. comparison of effect of material on the light at two or more different wavelengths or wavelength bands using photo-electric detection ; circuits for computing concentration
    • G01N21/274—Calibration, base line adjustment, drift correction
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
    • G01N21/25—Colour; Spectral properties, i.e. comparison of effect of material on the light at two or more different wavelengths or wavelength bands
    • G01N21/31—Investigating relative effect of material at wavelengths characteristic of specific elements or molecules, e.g. atomic absorption spectrometry
    • G01N21/314—Investigating relative effect of material at wavelengths characteristic of specific elements or molecules, e.g. atomic absorption spectrometry with comparison of measurements at specific and non-specific wavelengths
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
    • G01N21/25—Colour; Spectral properties, i.e. comparison of effect of material on the light at two or more different wavelengths or wavelength bands
    • G01N21/31—Investigating relative effect of material at wavelengths characteristic of specific elements or molecules, e.g. atomic absorption spectrometry
    • G01N21/35—Investigating relative effect of material at wavelengths characteristic of specific elements or molecules, e.g. atomic absorption spectrometry using infrared light
    • G01N21/3563—Investigating relative effect of material at wavelengths characteristic of specific elements or molecules, e.g. atomic absorption spectrometry using infrared light for analysing solids; Preparation of samples therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/40—Analysis of texture
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/586—Depth or shape recovery from multiple images from multiple light sources, e.g. photometric stereo
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/88—Image or video recognition using optical means, e.g. reference filters, holographic masks, frequency domain filters or spatial domain filters
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
    • G06V20/582—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads of traffic signs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/588—Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
    • G01N21/25—Colour; Spectral properties, i.e. comparison of effect of material on the light at two or more different wavelengths or wavelength bands
    • G01N21/31—Investigating relative effect of material at wavelengths characteristic of specific elements or molecules, e.g. atomic absorption spectrometry
    • G01N21/314—Investigating relative effect of material at wavelengths characteristic of specific elements or molecules, e.g. atomic absorption spectrometry with comparison of measurements at specific and non-specific wavelengths
    • G01N2021/3155—Measuring in two spectral ranges, e.g. UV and visible
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
    • G01N21/55—Specular reflectivity
    • G01N2021/551—Retroreflectance
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
    • G01N21/25—Colour; Spectral properties, i.e. comparison of effect of material on the light at two or more different wavelengths or wavelength bands
    • G01N21/27—Colour; Spectral properties, i.e. comparison of effect of material on the light at two or more different wavelengths or wavelength bands using photo-electric detection ; circuits for computing concentration
    • G01N21/274—Calibration, base line adjustment, drift correction
    • G01N21/278—Constitution of standards
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G06T2207/10012—Stereo images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10048—Infrared image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30252—Vehicle exterior; Vicinity of vehicle

Definitions

  • the present invention is concerned with measuring the retro-reflection of road signs, and in particular with an apparatus and a process for performing the retro-reflection measurement by means of a mobile device.
  • Retro- reflection is a measure of how well a road sign reflects light from the headlights of a vehicle back to the driver which makes them appear bright from long distances.
  • instruments which are either handheld or mobile for measuring retro-reflection. The handheld variety require traffic disruption and are slow in comparison with mobile instruments.
  • Mobile instruments offer the advantage of being able to measure more signs per unit time. However, they are traditionally less accurate than handheld units. Current mobile instruments are not portable instruments and are offered only as a service and not a product.
  • a method of measuring the retroreflection of a road sign comprising: illuminating a road sign by an infra-red light source and by a visible light source; capturing an image incorporating the road sign by each of a plurality of cameras;
  • the retroreflection parameters may comprise: the illuminance of each light source on the road sign, the colour of the road sign, the luminance of the road sign.
  • the plurality of cameras may comprise an infra-red photometric camera, a camera associated with the infra-red photometric camera for forming a stereo vision system when combined with the infra-red photometric camera, and a colour photometric camera.
  • the step of isolating the image of the road sign from each of the captured images may comprise the steps of:
  • the step of detecting an image of the road sign in the image captured by the infra-red photometric camera may comprise the steps of:
  • the step of matching a point on the image of road sign detected by the infra-red photometric camera to a corresponding point in the image captured by its associated camera may comprise the steps of:
  • the step of processing the isolated images to obtain a plurality of retroreflection parameters may further comprise the steps of:
  • the step of evaluating the illuminance of the road sign associated with each light source may comprise the steps of:
  • the step of evaluating the luminance of the road sign associated with each photometric camera may comprise the steps of:
  • the step of selecting slope and intercept luminance coefficients based on the detected colour may further comprise the steps of:
  • the step of calculating the retroreflection of the road sign based on the retroreflection parameters may comprise the step of:
  • Mi and Ci are the selected slope, M and intercept, C coefficients
  • I is the calculated luminance value associated with the photometric camera
  • E is the calculated illuminance function of its light source.
  • the method may further comprise the steps of:
  • the retroreflection value of the road sign if the calculated difference exceeds a predetermined value, selecting the retroreflection value of the road sign to be the retroreflection value of the road sign associated with the colour photometric camera.
  • the method may further comprise the step of detecting the colour of the road sign from the isolated image from the colour photometric camera comprises: counting the number of pixels in the isolated image of the road sign that are associated with each colour of a set of colour bands; and
  • the method may further comprise the steps of:
  • the method may further comprise the step of:
  • processing the detected image to classify the road sign based on its shape and colour may comprise the steps of:
  • the method may further comprise measuring heat and/or humidity during the retroreflection measurement.
  • the present invention also provides an apparatus for measuring the retroreflection of a road sign comprising:
  • the present invention also provides a method for calibrating an apparatus for use in the retroreflection measurement of road signs, the apparatus comprising at least one light source and at least one a photometric camera, the method comprising the steps of:
  • calibrating the at least one light source to determine its illumination function; calibrating the at least one photometric camera and the at least one light source to determine a transformation function from the at least one photometric camera XYZ co-ordinate points to the at least one light source XYZ co-ordinate points; and
  • the step of calibrating the at least one light source to determine its illumination function may comprise the step of:
  • the step of calibrating the at least one photometric camera and the at least one light source to determine a transformation function may comprise the steps of: measuring the position of the at least one light source with respect to the at least one photometric camera; and
  • the step of calibrating the slope, M coefficient and intercept, C coefficient associated with each sign colour using a calibration grid may comprise the steps of:
  • the step of determining the normalised luminance value for each colour in the calibration grid for an image of each colour captured by the at least one photometric camera at a standard observation angle may comprise the steps of: calculating the mean intensity for each colour for an image of the colour captured by the at least one photometric camera at a standard observation angle;
  • the at least one light source may comprise an infra-red light source and a visible light source.
  • Figure 1 discloses the main components of the apparatus of the present invention for the retro-reflection measurement of a road sign
  • Figure 2 shows a flow chart of the main steps involved in the detection of a road sign in the IR image
  • Figure 3 shows the defining of a bounding box of the same dimension as that defined during IR sign detection in the colour image of the road sign;
  • Figure 4 shows an image of a light source illuminance emission pattern generated during the calibration process;
  • Figure 5 shows the calibration grid for performing the photometric calibration
  • FIG. 6 shows an example of templates and binary bounding boxes for some road signs.
  • IR light source This illuminates the road signs under examination in the infra-red spectrum.
  • Pulsed Visible light source This illuminates the road signs under examination in the visible spectrum. This source is pulsed at a high rate >50Hz. Therefore, it appears continuous to oncoming traffic. The source remains in phase with image capture at a lower rate.
  • Photometric cameras Two photometric cameras are used to measure the radiance/luminance (apparent brightness) of the road signs. One photometric camera measures the infra-red part of the spectrum. This camera is the right camera in Figure 1 . The other photometric camera is a colour camera that measures the visible spectrum. The colour photometric camera also determines the colour of the road sign. This camera is the middle camera in Figure 1 . It will be appreciated that photometric camera should be in the same side of the car as the driver. For example, in right hand drive countries (such as Europe and the USA), the left camera can be used as the photometric camera.
  • a third camera is associated with the infra-red photometric camera so as to provide a stereo vision system, in order that geometrical measurements can be made on the road signs.
  • This camera is the left camera in Figure 1 .
  • GPS/lnertial Measurement Unit This navigation system along with the stereo vision system may be used for geo-referencing road signs.
  • the retro-reflection measurement, RA of a road sign may be obtained through the performance of a number of steps. Firstly, the road sign is illuminated using the IR and the visible light source. An image incorporating the road sign is then captured by the two photometric cameras along with the third camera for providing a stereo vision system. The image of the road sign is then isolated from each of the captured images, and the isolated images are processed to obtain a plurality of retro-reflection parameters.
  • the retro-reflection of the road sign is calculated based on the retro-reflection parameters.
  • One retro-reflection measurement of the road sign is calculated for the infra-red spectrum and another retro-reflection measurement of the road sign is calculated for the visible spectrum.
  • RA retro-reflection
  • the entrance angle at the standard observation angle is reported.
  • the a angle depends on the separation (baseline) between the light source and observer (i.e. the photometric camera) and the range the measurement is made at. If this baseline was 0.25 metres then a range of 43.4 metres will lead to a 0.33 degree a value. As image capture is discrete, there is no guarantee of an image at 43.4 metres. However, the likelihood is that one will be captured either side of it which allows interpolation (and in some cases extrapolation) towards standard values.
  • the CIE International commission on illumination
  • specify the light source which is to be used in retro-reflection measurement for road signs should follow that of the CIE illuminant A, which is for tungsten lamps.
  • the colour of a road sign can be found from the image from the photometric colour camera. However, it must also be determined how the colour behaves under IR light and how this relates to the CIE standard of illuminant A and V(A). This information can be obtained from a photometric retroreflection calibration process where the reflection of all the different coloured signs (multiple samples of each colour) are measured with a CIE approved instrument ,and also with the IR source/camera combination unit, and then colour weightings are applied to IR coefficients to align them with the luminance based reflectometer.
  • These calibration processes, along with the calibration processes in relation to the photometric cameras and light sources are described in more detail in a later paragraph, and must be established prior to performing the steps to measure the retro-reflection of a road sign.
  • the retro-reflection measurement of a road sign may be performed.
  • a number of parameters are obtained from the images of the road signs captured by the cameras. These include: -the illuminance of the road sign (amount of incident light from each of the light sources)
  • the retro-reflection of the road sign may be calculated by means of a mathematical equation.
  • ⁇ ( ⁇ ) is the illuminance function that models the light source non-uniformity
  • Mi and Ci are the linear colour calibration coefficients that map the normalised intensity to the RA value. The estimation of these coefficients is described in more detail in a later paragraph.
  • the road sign In order to commence the measurement process, the road sign must first be isolated in the images (as done in steps 2, 4 and 9 above). As there are two luminance measurements, one for the infra-red, IR, photometric camera and one for the visible photometric camera, it will be appreciated that the sign needs to be found in both the IR image and in the visible image. The sign also needs be found in the stereo image provided by the third camera which is associated with the infra-red photometric camera, so as to facilitate 3D reconstruction and the computation of the required angles (see steps 4 to 7).
  • the road sign search is initially performed in the IR image, as this is where the signs are most prominent. The main steps in this process are shown in Figure 2.
  • a blob analysis is performed to determine a road sign within the captured image (step 205). This involves the sub-steps of segmenting individual regions from the image. A number of properties are then calculated for each region (step 210). These properties may include major axis, maximum intensity, centroid, area, minor axis, solidity and aspect ratio. A decision tree approach is then taken where a region is deemed to be a road sign if its properties satisfy a number of conditions (step 215). If it is deemed to be a road sign, the sign is recorded (step 220). Otherwise the region is discarded (step 225). Detection rates for this approach are typically > 90% with a small false appositive rate ( ⁇ 1 %).
  • the road sign is found in the IR image, it must be found in the image of the third camera associated with the IR photometric camera comprising the stereo pair (i.e. in the left image assuming the right image is the IR image), as per steps 3 and 4 above.
  • the search for the road sign in the left image is constrained greatly by the known camera geometry (known as the epipolar constraint).
  • the search is reduced from an image search to a line search.
  • the extent of the line can be reduced also by assuming road signs are detected in a fixed range interval.
  • the INFORM system is interested in signs from 70-30 metres in range from the camera; this will give rise to approximately a 100 pixel segment on the epipolar line where the corresponding point can reside.
  • a discriminatory point on the sign is computed and matched in the other image. It should be noted that not all points on the sign are good candidates for matching. Some points will be from a very uniform area on the sign without any discriminatory information, and these should not be chosen. This is because the likelihood of an incorrect match is increased with these points. For this reason, a corner detector is used to detect points with a high degree of 'cornerness' towards the centre of the sign. A neighbourhood around this point is then selected, and this is matched with points in the left image. Cross correlation is used to match the right image neighbourhood along the epipolar line segment in the left image. The maximum of the cross- correlation function is chosen as a match. The local 3D location of the sign can now be estimated using standard mathematical formulae.
  • the road sign also needs to be found in the colour image captured by the colour photometric camera. This process is greatly simplified because the system and all cameras are tightly calibrated. This means that geometry between cameras is known to a high degree of accuracy along with the intrinsic camera parameters.
  • the colour value inside it is examined (step 9). This is done in normalised RGB space by counting the number of pixels in the bounding box that fall within each of the sign colour bands. The colour determination is important as there are different retro- reflection coefficients used for each colour. A homogenous region that is smaller than the whole is then chosen for the retro-reflection measurement, and the mean intensity of this region is calculated as the visible luminance (step 10). The same region is extracted from the photometric IR image and the mean intensity of this region is calculated as the IR luminance (step 1 1 ). The region size is proportional to the sign size. As mentioned above, an initial calibration of the cameras and light sources must be performed in order to enable some of the parameters required for the retro- reflection calculation to be determined.
  • the camera calibration relates known points in the real world to points in an image. It also allows the back projection of image points into world space. If a multiple view system exists, these back projection rays can be intersected to find the coordinates of the image points in world space (triangulation).
  • the calibration of each camera is achieved by using a calibration object with a precise pattern of black and white rectangles of known dimension. Knowledge of the dimensions of the squares on the object and their image locations allows for the recovery of the camera matrix which describes the whole process of how 3D points are projected onto the image plane.
  • Each light source can be considered similar to a camera for the purposes of calibration. It has intrinsic and extrinsic parameters. The intrinsic parameters depend on components such as the lens and the LEDs.
  • an isotropic surface e.g. a white matte wall
  • the extrinsic parameters define where the light is positioned and directed within some coordinate system. As the retroreflection measurements are made in the camera coordinate system, then the extrinsic parameters of the light should be estimated within this.
  • the position of each light source can be measured manually (such as for example with a ruler) with respect to its associated photometric camera assuming the physical centre of the light is known to provide a translation value.
  • the orientation of the light is more difficult. There are only two degrees of freedom here and not three if the light is considered to be symmetric. Therefore, any rotations about the camera z (out) axis may be ignored.
  • the rotations about the camera x (thetax) and y (thetay) axis for the light source need to be estimated.
  • the source centre can be isolated reliably on a plane at different ranges from the camera/source unit. Then the changes in x and y coordinates at the different ranges will give rise to the relative angles. Finding the centre of the light is not easy. As the light has a wide field of illumination, there is no obvious peak in the reflection pattern on the plane (made of diffuse material). The peak is estimated by thresholding the image of the plane such that a circle (fixed angle at different ranges) of pixels is found and the centre of the source is defined as the centre of this circle. Then the dx and dy values can be estimated from the camera parameters and the range. Two line fits are then made to the x-z and y-z values for a more robust estimate of dx and dy.
  • camera XYZ co-ordinate points can be transformed into source XYZ co-ordinate points during step 6 of the procedure set out above from the determined rotation and translation values, so the illuminance function can be evaluated and the scale factor be determined.
  • the photometric retroreflection calibration procedure uses a calibration object which has a number of different road sign materials containing the main colours used in road signs, e.g. green, yellow, red, blue and white.
  • the retro-reflection measurement is recorded using an approved handheld device for each road sign sample in a number of locations and averaged to obtain a single measurement for each sample.
  • the calibration grid for performing the photometric calibration is illustrated in Figure 5. It contains samples of the main road sign colours and grades.
  • the retro-reflection values of each colour are in a wide range so as to best estimate the linear calibration coefficients.
  • images of the calibration target are acquired at a range that gives rise to the standard observation angle (a of 0.33 degrees, EN 1436) across the field of view of each photometric camera. Images of the target are taken at locations where signs are likely to appear. These images are then automatically processed to extract the target and the mean intensities of each sample, along with the XYZ position with respect to the camera(s) which is computed from the stereo. The mean intensities are calculated by finding the corner points on the target and using these to define the sample boundaries. The pixels inside the boundaries are averaged to give the mean values. Each camera XYZ values are then transformed into its light source XYZ values through a rotation and translation. From the source XYZ the incident light angle, ⁇ , can be calculated for each sample.
  • the standard observation angle a of 0.33 degrees, EN 1436
  • the incident angles are passed into the illuminance function ⁇ ( ⁇ ), which returns the illumination scale factor.
  • the image luminance is divided by this to give normalised luminance.
  • the normalised luminance for each separate colour should be linearly related to the Ra values, so a best fit line is determined between the luminance values and Ra values for each colour.
  • the slope (Mi) and intercept (Ci) of these lines are the calibration coefficients which are required in the retroreflection equation for the road sign set out in step 14 above.
  • RA measurements there are two RA measurements made by the system, one using infrared light and one using visible light.
  • the infrared measurements are generally more accurate and repeatable.
  • IR outliers due to pigmentations in the road materials that do not absorb/reflect IR like other pigments of the same colour class. These outliers are rare but do exist.
  • the visible measurement handles these outliers sufficiently.
  • an IR and visible measurement are very much at odds by a preselected amount, the visible measurement is reported. This is because it handles the outliers better (but in general is not as accurate/repeatable as the IR measurement).
  • VIS/I R dual
  • a road sign when a road sign is detected it is classified based on shape and colour.
  • the shape is identified using a binary template matching approach which is very fast.
  • the binary representation of detected road signs include the road sign (with any interior holes filled in) and its bounding box.
  • the bounding box is resized to be the same size as the binary templates, and a simple logical XOR operation is performed on the binary bounding box and the templates. The XOR will return 0 for all pixels that have the same value so the template with the lowest score is chosen as a match for the road sign.
  • Figure 6 shows some of the road sign templates and the bounding boxes for detected some road signs.
  • the present invention provides a number of advantages over the conventional methods of measuring retro-reflection of road signs. Firstly, the apparatus is mobile and requires no precision driving or changes in driving style to measure the signs on both sides of the road being travelled. In addition, the retro- reflection of a variety of sign types can be measured. Furthermore, the process is fully automated after an initial simple calibration procedure. Finally, the accuracy of the instrument is comparable to that of other mobile instruments currently available.
  • the embodiments in the invention described with reference to the drawings comprise a computer apparatus and/or processes performed in a computer apparatus or module adapted to execute computer programs or instructions.
  • the invention also extends to computer programs, particularly computer programs stored on or in a carrier adapted to bring the invention into practice.
  • the program may be in the form of source code, object code, or a code intermediate source and object code, such as in partially compiled form or in any other form suitable for use in the implementation of the method according to the invention.
  • the carrier may comprise a storage medium such as ROM, e.g. CD ROM, or magnetic recording medium, e.g. a floppy disk or hard disk.
  • the carrier may be an electrical or optical signal which may be transmitted via an electrical or an optical cable or by radio or other means.

Abstract

An apparatus and method of measuring the retroreflection of a road sign comprising: illuminating a road sign by an infra-red light source and by a visible light source; capturing an image incorporating the road sign by each of a plurality of cameras; isolating the image of the road sign from each of the captured images; processing the isolated images to obtain a plurality of retroreflection parameters; and calculating the retroreflection of the road sign based on the retroreflection parameters.

Description

Title
Mobile Road Sign Reflectometer Field
The present invention is concerned with measuring the retro-reflection of road signs, and in particular with an apparatus and a process for performing the retro-reflection measurement by means of a mobile device. Background
It will be appreciated that the maintenance of a road sign is extremely important, in order that it can provide its intended level of guidance to road users. One important parameter in this regard is the retro-reflection of the road sign. Retro- reflection is a measure of how well a road sign reflects light from the headlights of a vehicle back to the driver which makes them appear bright from long distances. Minimum standards exist in Ireland, Europe and across the globe for retro-reflection of road signs. Therefore, there is a requirement for instruments to measure this parameter. Currently there are a number of instruments which are either handheld or mobile for measuring retro-reflection. The handheld variety require traffic disruption and are slow in comparison with mobile instruments. Mobile instruments offer the advantage of being able to measure more signs per unit time. However, they are traditionally less accurate than handheld units. Current mobile instruments are not portable instruments and are offered only as a service and not a product.
It is an object of the present invention to provide an apparatus and a method for measuring the retro-reflection of a road sign which overcomes at least some of the above mentioned problems. Summary
According to the present invention there is provided, as set out in the appended claims, a method of measuring the retroreflection of a road sign comprising: illuminating a road sign by an infra-red light source and by a visible light source; capturing an image incorporating the road sign by each of a plurality of cameras;
isolating the image of the road sign from each of the captured images;
processing the isolated images to obtain a plurality of retroreflection parameters; and
calculating the retroreflection of the road sign based on the retroreflection parameters.
The retroreflection parameters may comprise: the illuminance of each light source on the road sign, the colour of the road sign, the luminance of the road sign.
The plurality of cameras may comprise an infra-red photometric camera, a camera associated with the infra-red photometric camera for forming a stereo vision system when combined with the infra-red photometric camera, and a colour photometric camera.
The step of isolating the image of the road sign from each of the captured images may comprise the steps of:
detecting an image of a road sign in the image captured by the infra-red photometric camera;
finding the image of the road sign in the image captured by the camera associated with the infra-red photometric camera by matching a point on the image of road sign detected by the infra-red photometric camera to a corresponding point in the image captured by its associated camera;
calculating the 3D location of the road sign relative to the infra-red photometric camera from the data obtained from detecting the road sign in the photometric camera and its associated camera; and detecting the image of road sign in the image captured by the colour photometric camera by projecting the calculated 3D location of the road sign into the image captured by the colour photometric camera. The step of detecting an image of the road sign in the image captured by the infra-red photometric camera may comprise the steps of:
segmenting the captured image into a plurality of regions;
calculating the properties of each region; and
analysing the properties to determine whether they indicate an image of a road sign.
The step of matching a point on the image of road sign detected by the infra-red photometric camera to a corresponding point in the image captured by its associated camera may comprise the steps of:
detecting a point adjacent to the centre of the image of the road sign detected by the infra-red photometric camera corresponding to a corner;
selecting a neighbourhood of points around the corner;
cross correlating the selected neighbourhood of points with points along the epipolar line segment of the image of the road sign captured by its associated camera; and
selecting that point exhibiting maximum cross correlation as the matched point.
The step of processing the isolated images to obtain a plurality of retroreflection parameters may further comprise the steps of:
evaluating the illuminance associated with each light source based on the isolated images from the plurality of cameras;
evaluating the luminance of the road sign associated with each photometric camera based on the isolated images from the plurality of cameras;
detecting the colour of the road sign from the isolated image from the colour photometric camera; and
selecting slope and intercept luminance coefficients for each photometric camera based on the detected colour. The step of evaluating the illuminance of the road sign associated with each light source may comprise the steps of:
determining the location of the road sign relative to each light source through performing a predetermined transformation on the calculated 3D location of the road sign relative to its photometric camera;
calculating the incident angle of the reflected light from the road sign with respect to the central axis of each light source; and
calculating the illuminance function of each light source using the calculated incident angle.
The step of evaluating the luminance of the road sign associated with each photometric camera may comprise the steps of:
isolating a plurality of images of a road sign from each photometric camera; determining the observation angle associated with each isolated image captured by the infra-red photometric camera and captured by the colour photometric camera;
selecting a homogenous region of each isolated image of the road sign captured by the photometric colour camera;
calculating a mean intensity value for the selected homogenous region of the photometric image;
extracting a corresponding homogenous region in the isolated image of the road sign captured by the infra-red photometric camera;
calculating a mean intensity value for the extracted homogenous region of the colour image;
selecting as the luminance value of the road sign associated with each photometric camera as the mean intensity value of that photometric image having an associated observation angle closest to a standard observation image. The step of selecting slope and intercept luminance coefficients based on the detected colour may further comprise the steps of:
selecting slope, M and intercept, C coefficients based on the selected luminance value. The step of calculating the retroreflection of the road sign based on the retroreflection parameters may comprise the step of:
calculating for each photometric camera the retroreflection of the road sign by the equation:
Figure imgf000006_0001
where Mi and Ci are the selected slope, M and intercept, C coefficients;
I is the calculated luminance value associated with the photometric camera and
E is the calculated illuminance function of its light source.
The method may further comprise the steps of:
calculating the difference between the retroreflection value of the road sign associated with the infra-red photometric camera and the retroreflection value of the road sign associated with the colour photometric camera; and
if the calculated difference exceeds a predetermined value, selecting the retroreflection value of the road sign to be the retroreflection value of the road sign associated with the colour photometric camera.
The method may further comprise the step of detecting the colour of the road sign from the isolated image from the colour photometric camera comprises: counting the number of pixels in the isolated image of the road sign that are associated with each colour of a set of colour bands; and
determining which colour is associated with the greatest number of pixels. The method may further comprise the steps of:
calibrating each light source to determine the illumination function;
calibrating each photometric camera and its light source to determine the predetermined transformation from a calculated location of a road sign relative to each photometric camera to its location relative to its light source; and calibrating the M and C coefficients associated with each sign colour using a calibration grid. The method may further comprise the step of:
processing the detected image to classify the road sign based on its shape and colour. The step of processing the detected image to classify the road sign based on its shape and colour may comprise the steps of:
performing an XOR operation on the isolated image and a plurality of road sign templates. The method may further comprise measuring heat and/or humidity during the retroreflection measurement.
The present invention also provides an apparatus for measuring the retroreflection of a road sign comprising:
means or module for illuminating a road sign by an infra-red light source and by a visible light source;
means or module for capturing an image incorporating the road sign by each of a plurality of cameras;
means or module for isolating the image of the road sign from each of the captured images;
means or module for processing the isolated images to obtain a plurality of retroreflection parameters; and
means or module for calculating the retroreflection of the road sign based on the retroreflection parameters.
The present invention also provides a method for calibrating an apparatus for use in the retroreflection measurement of road signs, the apparatus comprising at least one light source and at least one a photometric camera, the method comprising the steps of:
calibrating the at least one light source to determine its illumination function; calibrating the at least one photometric camera and the at least one light source to determine a transformation function from the at least one photometric camera XYZ co-ordinate points to the at least one light source XYZ co-ordinate points; and
calibrating the slope, M coefficient and intercept, C coefficient associated with each sign colour using a calibration grid.
The step of calibrating the at least one light source to determine its illumination function may comprise the step of:
estimating the relative intensity of the at least one light source at each point in the field of view of the at least one photometric camera.
The step of calibrating the at least one photometric camera and the at least one light source to determine a transformation function may comprise the steps of: measuring the position of the at least one light source with respect to the at least one photometric camera; and
estimating the rotations about the at least one photometric camera x axis and y axis for the at least one light source.
The step of calibrating the slope, M coefficient and intercept, C coefficient associated with each sign colour using a calibration grid may comprise the steps of:
calculating the retroreflection value for each colour in the calibration grid using a standard handheld device;
determining the normalised luminance value for each colour in the calibration grid for an image of the colour captured by the at least one photometric camera at a standard observation angle;
determining the best fit line between the normalised luminance value and the calculated retroreflection value for each colour; wherein the slope and intercept of this line are the slope and intercept calibration coefficients associated with each colour.
The step of determining the normalised luminance value for each colour in the calibration grid for an image of each colour captured by the at least one photometric camera at a standard observation angle may comprise the steps of: calculating the mean intensity for each colour for an image of the colour captured by the at least one photometric camera at a standard observation angle;
transforming the at least one photometric camera XYZ co-ordinate points for each colour into the at least one light source XYZ co-ordinate points;
calculating the incident light angle for each colour from the at least one light source XYZ co-ordinate points;
evaluating the illuminance function for each colour from the calculated incident light angle to determine the illumination scale factor; and
dividing the mean intensity for each colour by the illumination scale factor for each colour to determine the normalized luminance value for each colour.
The at least one light source may comprise an infra-red light source and a visible light source.
Brief Description of the Drawings
The invention will be more clearly understood from the following description of an embodiment thereof, given by way of example only, with reference to the accompanying drawings, in which :-
Figure 1 discloses the main components of the apparatus of the present invention for the retro-reflection measurement of a road sign;
Figure 2 shows a flow chart of the main steps involved in the detection of a road sign in the IR image;
Figure 3 shows the defining of a bounding box of the same dimension as that defined during IR sign detection in the colour image of the road sign; Figure 4 shows an image of a light source illuminance emission pattern generated during the calibration process;
Figure 5 shows the calibration grid for performing the photometric calibration; and
Figure 6 shows an example of templates and binary bounding boxes for some road signs. Detailed Description of the Drawings
The main components of the apparatus of the present invention for measuring the retro-reflection of road signs are shown in Figure 1 . They are as follows:
Pulsed Infra-red (IR) light source: This illuminates the road signs under examination in the infra-red spectrum.
Pulsed Visible light source: This illuminates the road signs under examination in the visible spectrum. This source is pulsed at a high rate >50Hz. Therefore, it appears continuous to oncoming traffic. The source remains in phase with image capture at a lower rate. Photometric cameras: Two photometric cameras are used to measure the radiance/luminance (apparent brightness) of the road signs. One photometric camera measures the infra-red part of the spectrum. This camera is the right camera in Figure 1 . The other photometric camera is a colour camera that measures the visible spectrum. The colour photometric camera also determines the colour of the road sign. This camera is the middle camera in Figure 1 . It will be appreciated that photometric camera should be in the same side of the car as the driver. For example, in right hand drive countries (such as Europe and the USA), the left camera can be used as the photometric camera.
Stereo Vision System: A third camera is associated with the infra-red photometric camera so as to provide a stereo vision system, in order that geometrical measurements can be made on the road signs. This camera is the left camera in Figure 1 .
GPS/lnertial Measurement Unit (IMU): This navigation system along with the stereo vision system may be used for geo-referencing road signs.
Temperature and Relative Humidity Sensor (T+RH): This is used to record the temperature and humidity when doing surveys. This information can be important when interpreting retro-reflection measurements, as changes in these quantities can affect measurements. In accordance with the method of the present invention, the retro-reflection measurement, RA, of a road sign may be obtained through the performance of a number of steps. Firstly, the road sign is illuminated using the IR and the visible light source. An image incorporating the road sign is then captured by the two photometric cameras along with the third camera for providing a stereo vision system. The image of the road sign is then isolated from each of the captured images, and the isolated images are processed to obtain a plurality of retro-reflection parameters. Finally, the retro-reflection of the road sign is calculated based on the retro-reflection parameters. One retro-reflection measurement of the road sign is calculated for the infra-red spectrum and another retro-reflection measurement of the road sign is calculated for the visible spectrum.
In this context it should be understood that the calculation of retro-reflection, RA, is dictated by the European/American standards where an observer, which corresponds to each photometric camera in the context of the measurement process of the present invention, and its associated light source are configured according to a specified geometry. It is really the angles that this geometry produces that are of importance in the calculation. The two main angles, the entrance angle, β, (5 degrees, European/4 degrees, US), and the observation angle, a, (0.33 degrees, European/0.2 degrees, US), are what define the measurement geometry. Therefore, any measurements should be closely related to this. The observation angle is the more influential of the two in terms of RA, so the standard value should be obtained. The standard entrance angle value is difficult (if not impossible in some cases) to obtain from a mobile platform. Accordingly, the entrance angle at the standard observation angle is reported. The a angle depends on the separation (baseline) between the light source and observer (i.e. the photometric camera) and the range the measurement is made at. If this baseline was 0.25 metres then a range of 43.4 metres will lead to a 0.33 degree a value. As image capture is discrete, there is no guarantee of an image at 43.4 metres. However, the likelihood is that one will be captured either side of it which allows interpolation (and in some cases extrapolation) towards standard values. In addition, the CIE (International commission on illumination) specify the light source which is to be used in retro-reflection measurement for road signs should follow that of the CIE illuminant A, which is for tungsten lamps. This has higher power in the red wavelengths than in the blue. In addition, it is required that the observer or photometer should have the response of the 2 degree photopic luminosity function. The manufacturers of handheld instrument state that they conform to these standards. Any measurements made with a source/photometer not having these attributes should have numeric or computational filters that are capable of mapping to these functions, or a calibrated instrument based on these functions. For a narrow bandwidth source, such as an IR source (e.g. 850nm) and an IR sensitive monochromatic sensor, a colour-based calibration that describes how this source/camera combination unit behaves for different coloured signs must therefore be performed. Furthermore, as the light source has a narrow bandwidth, there is little concern about the sensor response inside the light source power spectrum, i.e. only reflected IR light is being measured. The colour of a road sign can be found from the image from the photometric colour camera. However, it must also be determined how the colour behaves under IR light and how this relates to the CIE standard of illuminant A and V(A). This information can be obtained from a photometric retroreflection calibration process where the reflection of all the different coloured signs (multiple samples of each colour) are measured with a CIE approved instrument ,and also with the IR source/camera combination unit, and then colour weightings are applied to IR coefficients to align them with the luminance based reflectometer. These calibration processes, along with the calibration processes in relation to the photometric cameras and light sources are described in more detail in a later paragraph, and must be established prior to performing the steps to measure the retro-reflection of a road sign.
Once the necessary photometric retroreflection calibration has been performed, as well as the necessary calibration in respect of the colour and infra-red photometric cameras and the visible and infra-red light sources, and the resulting calibration data loaded into the apparatus, the retro-reflection measurement of a road sign may be performed. In order to measure the retro- reflection of a road sign, a number of parameters are obtained from the images of the road signs captured by the cameras. These include: -the illuminance of the road sign (amount of incident light from each of the light sources)
-the luminance (reflected/emitted light) of the road sign for both the infra-red and the colour photometric cameras
-the colour of the road sign; and -the location of the road sign with respect to each light source and the observer (the photometric cameras).
Once these parameters are obtained from the image data, the retro-reflection of the road sign may be calculated by means of a mathematical equation.
The main steps in the process of determining the retro-reflection of a road sign are set out below. A more detailed explanation of some of these steps will then be described in the paragraphs that follow.
1 . Load photometric calibration coefficients and the illuminance function obtained from the calibration processes into the data processing application.
2. Extract road sign(s) from the photometric infra-red camera infra-red (IR) images.
3. Extract a corner from the road sign image
4. Match the corner extracted along the epipolar line in the stereo image in the third camera associated with the photometric infra-red camera to determine corresponding points.
5. Calculate sign XYZ co-ordinate points in camera reference (IR and visible) frame, so as to determine 3D location of the road sign relative to the photometric cameras. This is facilitated through the use of the photometric cameras calibration data.
6. Calculate sign XYZ co-ordinate points in light source reference frame (IR and visible) from a camera to light source transformation, the appropriate transformation having being determined using the light sources calibration data. This provides the 3D location of the road sign relative to the light sources. From this, the incident angle with respect to the central axis of the light source may be calculated.
7. Calculate the observation (a) and entrance (β) angles from the vectors in steps 5 and 6 above. If it is assumed that the sign surface normal is parallel to the direction of travel, these angles can be calculated by any standard technique which would be known to a person skilled in the art, provided the scene point coordinate is known with respect to the light sources and photometric cameras. 8. Evaluate the illuminance function of each light source obtained during the calibration process with the incident angle to obtain the equalising scale factor.
9. Determine the sign colour from the colour image of the road sign in the photometric colour camera. The sign can be located in the colour image directly by back-projection and re-projection.
10. Find a homogenous region in the colour bounding box (region size dependent on sign size) and find the mean intensity of this region in the photometric colour image.
1 1 . Extract the same region from the photometric IR image and find the mean intensity of this region in the photometric IR image.
12. Group multiple detections of the same sign together and sort by range or observation angle a. Choose luminance value with a closest to the standard (0.33/0.2) or interpolate to estimate l(a=0.33/0.2).
13. Select the correct M and C coefficients for each photometric camera based on sign colour, as determined during the calibration process. 14. Calculate the retro-reflection of the road sign in both the IR and visible owing equation:
Figure imgf000015_0001
Where la is the sign region light intensity of images captured at the standard observation angle (a of .33 degrees/0.2 degrees US), Ε(θ) is the illuminance function that models the light source non-uniformity and Mi and Ci are the linear colour calibration coefficients that map the normalised intensity to the RA value. The estimation of these coefficients is described in more detail in a later paragraph.
In order to commence the measurement process, the road sign must first be isolated in the images (as done in steps 2, 4 and 9 above). As there are two luminance measurements, one for the infra-red, IR, photometric camera and one for the visible photometric camera, it will be appreciated that the sign needs to be found in both the IR image and in the visible image. The sign also needs be found in the stereo image provided by the third camera which is associated with the infra-red photometric camera, so as to facilitate 3D reconstruction and the computation of the required angles (see steps 4 to 7). The road sign search is initially performed in the IR image, as this is where the signs are most prominent. The main steps in this process are shown in Figure 2. After the image is thresholded (step 200) a blob analysis is performed to determine a road sign within the captured image (step 205). This involves the sub-steps of segmenting individual regions from the image. A number of properties are then calculated for each region (step 210). These properties may include major axis, maximum intensity, centroid, area, minor axis, solidity and aspect ratio. A decision tree approach is then taken where a region is deemed to be a road sign if its properties satisfy a number of conditions (step 215). If it is deemed to be a road sign, the sign is recorded (step 220). Otherwise the region is discarded (step 225). Detection rates for this approach are typically > 90% with a small false appositive rate (< 1 %). Once the road sign is found in the IR image, it must be found in the image of the third camera associated with the IR photometric camera comprising the stereo pair (i.e. in the left image assuming the right image is the IR image), as per steps 3 and 4 above. As the stereo system is tightly calibrated, the search for the road sign in the left image is constrained greatly by the known camera geometry (known as the epipolar constraint). The search is reduced from an image search to a line search. The extent of the line can be reduced also by assuming road signs are detected in a fixed range interval. For example, the INFORM system is interested in signs from 70-30 metres in range from the camera; this will give rise to approximately a 100 pixel segment on the epipolar line where the corresponding point can reside. Rather than search for the whole sign in the other image, a discriminatory point on the sign is computed and matched in the other image. It should be noted that not all points on the sign are good candidates for matching. Some points will be from a very uniform area on the sign without any discriminatory information, and these should not be chosen. This is because the likelihood of an incorrect match is increased with these points. For this reason, a corner detector is used to detect points with a high degree of 'cornerness' towards the centre of the sign. A neighbourhood around this point is then selected, and this is matched with points in the left image. Cross correlation is used to match the right image neighbourhood along the epipolar line segment in the left image. The maximum of the cross- correlation function is chosen as a match. The local 3D location of the sign can now be estimated using standard mathematical formulae.
The road sign also needs to be found in the colour image captured by the colour photometric camera. This process is greatly simplified because the system and all cameras are tightly calibrated. This means that geometry between cameras is known to a high degree of accuracy along with the intrinsic camera parameters. Once the 3D location of the road sign relative to the IR photometric camera is calculated in step 5, which is facilitated through the initial calibration which is performed prior to the process steps, this point can be projected into the colour image to reveal the location of the road sign. A bounding box of the same dimension as that defined during IR sign detection is then defined in the colour image, as shown in Figure 3.
When the colour bounding box is identified, the colour value inside it is examined (step 9). This is done in normalised RGB space by counting the number of pixels in the bounding box that fall within each of the sign colour bands. The colour determination is important as there are different retro- reflection coefficients used for each colour. A homogenous region that is smaller than the whole is then chosen for the retro-reflection measurement, and the mean intensity of this region is calculated as the visible luminance (step 10). The same region is extracted from the photometric IR image and the mean intensity of this region is calculated as the IR luminance (step 1 1 ). The region size is proportional to the sign size. As mentioned above, an initial calibration of the cameras and light sources must be performed in order to enable some of the parameters required for the retro- reflection calculation to be determined. These calibration processes are described below. The camera calibration relates known points in the real world to points in an image. It also allows the back projection of image points into world space. If a multiple view system exists, these back projection rays can be intersected to find the coordinates of the image points in world space (triangulation). The calibration of each camera is achieved by using a calibration object with a precise pattern of black and white rectangles of known dimension. Knowledge of the dimensions of the squares on the object and their image locations allows for the recovery of the camera matrix which describes the whole process of how 3D points are projected onto the image plane.
Each light source can be considered similar to a camera for the purposes of calibration. It has intrinsic and extrinsic parameters. The intrinsic parameters depend on components such as the lens and the LEDs. The intrinsic calibration is defined as estimating the relative intensity of each point in a hemisphere around the light S=\{ θ, φ). For the purposes of the present invention, a full hemisphere calibration is not required, but only 30 degrees or the field of view of the camera. This function can be recovered by taking an image of the emission pattern of the light source on an isotropic surface (e.g. a white matte wall), as shown in Figure 4. The centre of the pattern can then be found as the point of maximal intensity and the illuminance function can evaluated around this. This procedure is performed at a range then is known in advance to facilitate accurate angle computation. If the source is rotationally symmetric, the illuminance function can be simplified to S=l(9). This function returns a scale factor, S, which indicates how much the image intensity of the road sign should be scaled to compensate for the non-uniformity of the source.
The extrinsic parameters define where the light is positioned and directed within some coordinate system. As the retroreflection measurements are made in the camera coordinate system, then the extrinsic parameters of the light should be estimated within this. The position of each light source can be measured manually (such as for example with a ruler) with respect to its associated photometric camera assuming the physical centre of the light is known to provide a translation value. The orientation of the light is more difficult. There are only two degrees of freedom here and not three if the light is considered to be symmetric. Therefore, any rotations about the camera z (out) axis may be ignored. The rotations about the camera x (thetax) and y (thetay) axis for the light source need to be estimated. This can be done if the source centre can be isolated reliably on a plane at different ranges from the camera/source unit. Then the changes in x and y coordinates at the different ranges will give rise to the relative angles. Finding the centre of the light is not easy. As the light has a wide field of illumination, there is no obvious peak in the reflection pattern on the plane (made of diffuse material). The peak is estimated by thresholding the image of the plane such that a circle (fixed angle at different ranges) of pixels is found and the centre of the source is defined as the centre of this circle. Then the dx and dy values can be estimated from the camera parameters and the range. Two line fits are then made to the x-z and y-z values for a more robust estimate of dx and dy. The rotations are then defined by the slopes of the lines. Once these extrinsic parameters are defined, camera XYZ co-ordinate points can be transformed into source XYZ co-ordinate points during step 6 of the procedure set out above from the determined rotation and translation values, so the illuminance function can be evaluated and the scale factor be determined.
The photometric retroreflection calibration procedure uses a calibration object which has a number of different road sign materials containing the main colours used in road signs, e.g. green, yellow, red, blue and white. The retro-reflection measurement is recorded using an approved handheld device for each road sign sample in a number of locations and averaged to obtain a single measurement for each sample. The calibration grid for performing the photometric calibration is illustrated in Figure 5. It contains samples of the main road sign colours and grades. The retro-reflection values of each colour are in a wide range so as to best estimate the linear calibration coefficients.
In order to calibrate the system, images of the calibration target are acquired at a range that gives rise to the standard observation angle (a of 0.33 degrees, EN 1436) across the field of view of each photometric camera. Images of the target are taken at locations where signs are likely to appear. These images are then automatically processed to extract the target and the mean intensities of each sample, along with the XYZ position with respect to the camera(s) which is computed from the stereo. The mean intensities are calculated by finding the corner points on the target and using these to define the sample boundaries. The pixels inside the boundaries are averaged to give the mean values. Each camera XYZ values are then transformed into its light source XYZ values through a rotation and translation. From the source XYZ the incident light angle, Θ, can be calculated for each sample. The incident angles are passed into the illuminance function Ε(θ), which returns the illumination scale factor. The image luminance is divided by this to give normalised luminance. The normalised luminance for each separate colour should be linearly related to the Ra values, so a best fit line is determined between the luminance values and Ra values for each colour. The slope (Mi) and intercept (Ci) of these lines are the calibration coefficients which are required in the retroreflection equation for the road sign set out in step 14 above.
In accordance with the present invention, there are two RA measurements made by the system, one using infrared light and one using visible light. The infrared measurements are generally more accurate and repeatable. However there are IR outliers due to pigmentations in the road materials that do not absorb/reflect IR like other pigments of the same colour class. These outliers are rare but do exist. The visible measurement handles these outliers sufficiently. When an IR and visible measurement are very much at odds by a preselected amount, the visible measurement is reported. This is because it handles the outliers better (but in general is not as accurate/repeatable as the IR measurement). It will be appreciated that using a dual (VIS/I R) light source enables the visible light to be dimmer and less obtrusive for other road users.
In one embodiment of the invention, when a road sign is detected it is classified based on shape and colour. There are three main categories of road signs, namely regulatory sign, warning signs and information signs. The shape is identified using a binary template matching approach which is very fast. The binary representation of detected road signs include the road sign (with any interior holes filled in) and its bounding box. The bounding box is resized to be the same size as the binary templates, and a simple logical XOR operation is performed on the binary bounding box and the templates. The XOR will return 0 for all pixels that have the same value so the template with the lowest score is chosen as a match for the road sign. Figure 6 shows some of the road sign templates and the bounding boxes for detected some road signs. Once the shape and colour are known, a decision tree approach is taken to classify the sign where particular shape/colour combinations give rise to a particular category of sign. The present invention provides a number of advantages over the conventional methods of measuring retro-reflection of road signs. Firstly, the apparatus is mobile and requires no precision driving or changes in driving style to measure the signs on both sides of the road being travelled. In addition, the retro- reflection of a variety of sign types can be measured. Furthermore, the process is fully automated after an initial simple calibration procedure. Finally, the accuracy of the instrument is comparable to that of other mobile instruments currently available. The embodiments in the invention described with reference to the drawings comprise a computer apparatus and/or processes performed in a computer apparatus or module adapted to execute computer programs or instructions. However, the invention also extends to computer programs, particularly computer programs stored on or in a carrier adapted to bring the invention into practice. The program may be in the form of source code, object code, or a code intermediate source and object code, such as in partially compiled form or in any other form suitable for use in the implementation of the method according to the invention. The carrier may comprise a storage medium such as ROM, e.g. CD ROM, or magnetic recording medium, e.g. a floppy disk or hard disk. The carrier may be an electrical or optical signal which may be transmitted via an electrical or an optical cable or by radio or other means.
In the specification the terms "comprise, comprises, comprised and comprising" or any variation thereof and the terms include, includes, included and including" or any variation thereof are considered to be totally interchangeable and they should all be afforded the widest possible interpretation and vice versa.
The invention is not limited to the embodiments hereinbefore described but maybe varied in both construction and detail.

Claims

Claims 1 . A method of measuring the retroreflection of a road sign comprising:
illuminating a road sign by an infra-red light source and by a visible light source;
capturing an image incorporating the road sign by each of a plurality of cameras;
isolating the image of the road sign from each of the captured images;
processing the isolated images to obtain a plurality of retroreflection parameters; and
calculating the retroreflection of the road sign based on the retroreflection parameters.
2. The method of Claim 1 , wherein the retroreflection parameters comprise: the illuminance of each light source on the road sign, the colour of the road sign, the luminance of the road sign.
3. The method of Claim 2, wherein the plurality of cameras comprise an infrared photometric camera, a camera associated with the infra-red photometric camera for forming a stereo vision system when combined with the infra-red photometric camera, and a colour photometric camera.
4. The method of Claim 3, wherein the step of isolating the image of the road sign from each of the captured images comprises the steps of:
detecting an image of a road sign in the image captured by the infra-red photometric camera;
finding the image of the road sign in the image captured by the camera associated with the infra-red photometric camera by matching a point on the image of road sign detected by the infra-red photometric camera to a corresponding point in the image captured by its associated camera; calculating the 3D location of the road sign relative to the infra-red photometric camera from the data obtained from detecting the road sign in the photometric camera and its associated camera; and
detecting the image of road sign in the image captured by the colour photometric camera by projecting the calculated 3D location of the road sign into the image captured by the colour photometric camera.
5. The method of Claim 4, wherein the step of detecting an image of the road sign in the image captured by the infra-red photometric camera comprises the steps of:
segmenting the captured image into a plurality of regions;
calculating the properties of each region; and
analysing the properties to determine whether they indicate an image of a road sign.
6. The method of Claim 4 wherein the step of matching a point on the image of road sign detected by the infra-red photometric camera to a corresponding point in the image captured by its associated camera comprises the steps of:
detecting a point adjacent to the centre of the image of the road sign detected by the infra-red photometric camera corresponding to a corner;
selecting a neighbourhood of points around the corner;
cross correlating the selected neighbourhood of points with points along the epipolar line segment of the image of the road sign captured by its associated camera; and
selecting that point exhibiting maximum cross correlation as the matched point.
7. The method of Claim 6, wherein the step of processing the isolated images to obtain a plurality of retroreflection parameters further comprises the steps of: evaluating the illuminance associated with each light source based on the isolated images from the plurality of cameras;
evaluating the luminance of the road sign associated with each photometric camera based on the isolated images from the plurality of cameras; detecting the colour of the road sign from the isolated image from the colour photometric camera; and
selecting slope and intercept luminance coefficients for each photometric camera based on the detected colour.
8. The method of Claim 7, wherein the step of evaluating the illuminance of the road sign associated with each light source comprises the steps of:
determining the location of the road sign relative to each light source through performing a predetermined transformation on the calculated 3D location of the road sign relative to its photometric camera;
calculating the incident angle of the reflected light from the road sign with respect to the central axis of each light source; and
calculating the illuminance function of each light source using the calculated incident angle.
9. The method of Claim 7, wherein the step of evaluating the luminance of the road sign associated with each photometric camera comprises the steps of: isolating a plurality of images of a road sign from each photometric camera; determining the observation angle associated with each isolated image captured by the infra-red photometric camera and captured by the colour photometric camera;
selecting a homogenous region of each isolated image of the road sign captured by the photometric colour camera;
calculating a mean intensity value for the selected homogenous region of the photometric image;
extracting a corresponding homogenous region in the isolated image of the road sign captured by the infra-red photometric camera;
calculating a mean intensity value for the extracted homogenous region of the colour image;
selecting as the luminance value of the road sign associated with each photometric camera as the mean intensity value of that photometric image having an associated observation angle closest to a standard observation image.
10. The method of Claim 9, wherein the step of selecting slope and intercept luminance coefficients based on the detected colour further comprises the steps of:
selecting slope, M and intercept, C coefficients based on the selected luminance value.
1 1 . The method of Claim 10, wherein the step of calculating the retroreflection of the road sign based on the retroreflection parameters comprises the step of: calculating for each photometric camera the retroreflection of the road sign by the equation:
Figure imgf000025_0001
where Mi and Ci are the selected slope, M and intercept, C coefficients;
I is the calculated luminance value associated with the photometric camera and E is the calculated illuminance function of its light source.
12. The method of Claim 1 1 , further comprising the steps of:
calculating the difference between the retroreflection value of the road sign associated with the infra-red photometric camera and the retroreflection value of the road sign associated with the colour photometric camera; and
if the calculated difference exceeds a predetermined value, selecting the retroreflection value of the road sign to be the retroreflection value of the road sign associated with the colour photometric camera.
13. The method of Claim 7, wherein the step of detecting the colour of the road sign from the isolated image from the colour photometric camera comprises: counting the number of pixels in the isolated image of the road sign that are associated with each colour of a set of colour bands; and
determining which colour is associated with the greatest number of pixels.
14. The method of Claim 1 further comprising the steps of: calibrating each light source to determine the illumination function;
calibrating each photometric camera and its light source to determine the predetermined transformation from a calculated location of a road sign relative to each photometric camera to its location relative to its light source; and calibrating the M and C coefficients associated with each sign colour using a calibration grid.
15. The method of any of the preceding claims further comprising the step of: processing the detected image to classify the road sign based on its shape and colour.
16. The method of Claim 15, wherein the step of processing the detected image to classify the road sign based on its shape and colour comprises the steps of: performing an XOR operation on the isolated image and a plurality of road sign templates.
17. The method of any of the preceding claims, further comprising measuring heat and/or humidity during the retroreflection measurement.
18. An apparatus for measuring the retroreflection of a road sign comprising: means for illuminating a road sign by an infra-red light source and by a visible light source;
means for capturing an image incorporating the road sign by each of a plurality of cameras;
means for isolating the image of the road sign from each of the captured images;
means for processing the isolated images to obtain a plurality of retroreflection parameters; and
means for calculating the retroreflection of the road sign based on the retroreflection parameters.
19. A method for calibrating an apparatus for use in the retroreflection
measurement of road signs, the apparatus comprising at least one light source and at least one a photometric camera, the method comprising the steps of: calibrating the at least one light source to determine its illumination function; calibrating the at least one photometric camera and the at least one light source to determine a transformation function from the at least one photometric camera XYZ co-ordinate points to the at least one light source XYZ co-ordinate points; and
calibrating the slope, M coefficient and intercept, C coefficient associated with each sign colour using a calibration grid.
20. The method of Claim 19, wherein the step of calibrating the at least one light source to determine its illumination function comprises the step of:
estimating the relative intensity of the at least one light source at each point in the field of view of the at least one photometric camera.
21 . The method of Claim 19, wherein the step of calibrating the at least one photometric camera and the at least one light source to determine a
transformation function comprises the steps of:
measuring the position of the at least one light source with respect to the at least one photometric camera; and
estimating the rotations about the at least one photometric camera x axis and y axis for the at least one light source.
22. The method of Claim 19, wherein the step of calibrating the slope, M coefficient and intercept, C coefficient associated with each sign colour using a calibration grid comprises the steps of:
calculating the retroreflection value for each colour in the calibration grid using a standard handheld device;
determining the normalised luminance value for each colour in the calibration grid for an image of the colour captured by the at least one photometric camera at a standard observation angle; determining the best fit line between the normalised luminance value and the calculated retroreflection value for each colour; wherein the slope and intercept of this line are the slope and intercept calibration coefficients associated with each colour.
23. The method of Claim 22, wherein the step of determining the normalised luminance value for each colour in the calibration grid for an image of each colour captured by the at least one photometric camera at a standard observation angle comprises the steps of:
calculating the mean intensity for each colour for an image of the colour captured by the at least one photometric camera at a standard observation angle;
transforming the at least one photometric camera XYZ co-ordinate points for each colour into the at least one light source XYZ co-ordinate points;
calculating the incident light angle for each colour from the at least one light source XYZ co-ordinate points;
evaluating the illuminance function for each colour from the calculated incident light angle to determine the illumination scale factor; and
dividing the mean intensity for each colour by the illumination scale factor for each colour to determine the normalized luminance value for each colour.
24. The method of any of Claims 19 to 23, wherein the at least one light source comprises an infra-red light source and a visible light source.
PCT/EP2015/063809 2014-06-20 2015-06-19 Mobile road sign reflectometer WO2015193470A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
GB1411028.2 2014-06-20
GBGB1411028.2A GB201411028D0 (en) 2014-06-20 2014-06-20 Mobile road sign reflectometer

Publications (1)

Publication Number Publication Date
WO2015193470A1 true WO2015193470A1 (en) 2015-12-23

Family

ID=51409904

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2015/063809 WO2015193470A1 (en) 2014-06-20 2015-06-19 Mobile road sign reflectometer

Country Status (2)

Country Link
GB (1) GB201411028D0 (en)
WO (1) WO2015193470A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111951337A (en) * 2020-08-19 2020-11-17 武汉中海庭数据技术有限公司 Image detection target space positioning method and system

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020106109A1 (en) * 2000-08-12 2002-08-08 Retterath James E. System for road sign sheeting classification
US6594600B1 (en) * 1997-10-24 2003-07-15 Commissariat A L'energie Atomique Method for calibrating the initial position and the orientation of one or several mobile cameras
JP2009201064A (en) * 2008-02-25 2009-09-03 Pioneer Electronic Corp Method and apparatus for specifying related region, and method and apparatus for recognizing image
US20090232358A1 (en) * 2008-03-11 2009-09-17 Geoffrey Mark Timothy Cross Method and apparatus for processing an image
DE102009014437A1 (en) * 2008-03-26 2009-11-12 Continental Teves Ag & Co. Ohg Object e.g. pedestrian, recognition system i.e. stereoscopic system, for motor vehicle, has validation device checking whether object and/or preset character e.g. contour, of object is detected in image region in image

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6594600B1 (en) * 1997-10-24 2003-07-15 Commissariat A L'energie Atomique Method for calibrating the initial position and the orientation of one or several mobile cameras
US20020106109A1 (en) * 2000-08-12 2002-08-08 Retterath James E. System for road sign sheeting classification
JP2009201064A (en) * 2008-02-25 2009-09-03 Pioneer Electronic Corp Method and apparatus for specifying related region, and method and apparatus for recognizing image
US20090232358A1 (en) * 2008-03-11 2009-09-17 Geoffrey Mark Timothy Cross Method and apparatus for processing an image
DE102009014437A1 (en) * 2008-03-26 2009-11-12 Continental Teves Ag & Co. Ohg Object e.g. pedestrian, recognition system i.e. stereoscopic system, for motor vehicle, has validation device checking whether object and/or preset character e.g. contour, of object is detected in image region in image

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
ÁLVARO GONZALEZ ET AL: "Automatic Traffic Signs and Panels Inspection System Using Computer Vision", IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, IEEE, PISCATAWAY, NJ, USA, vol. 12, no. 2, 1 June 2011 (2011-06-01), pages 485 - 499, XP011325854, ISSN: 1524-9050, DOI: 10.1109/TITS.2010.2098029 *
PHILIP SIEGMANN ET AL: "Fundaments in Luminance and Retroreflectivity Measurements of Vertical Traffic Signs Using a Color Digital Camera", IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, IEEE SERVICE CENTER, PISCATAWAY, NJ, US, vol. 57, no. 3, 1 March 2008 (2008-03-01), pages 607 - 615, XP011199146, ISSN: 0018-9456, DOI: 10.1109/TIM.2007.911643 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111951337A (en) * 2020-08-19 2020-11-17 武汉中海庭数据技术有限公司 Image detection target space positioning method and system
CN111951337B (en) * 2020-08-19 2022-05-31 武汉中海庭数据技术有限公司 Image detection target space positioning method and system

Also Published As

Publication number Publication date
GB201411028D0 (en) 2014-08-06

Similar Documents

Publication Publication Date Title
EP2140251B1 (en) Apparatus and methods for assessment, evaluation and grading of gemstones
US9207069B2 (en) Device for generating a three-dimensional model based on point cloud data
US20180340962A1 (en) Apparatus and method for diagnosing electric power equipment using thermal imaging camera
GB2503328B (en) Tire detection for accurate vehicle speed estimation
JP4997252B2 (en) How to identify the illumination area in an image
CN108027328B (en) Colour measurement of gemstones
US10032088B2 (en) Method and system for classifying painted road markings in an automotive driver-vehicle-assistance device
US20120008021A1 (en) Shadow Removal in an Image Captured by a Vehicle-Based Camera for Clear Path Detection
Kwon Atmospheric visibility measurements using video cameras: Relative visibility
US10429298B2 (en) Method for determining the reflectance of an object and associated device
CN102194103B (en) For active visibility support apparatus and the method for vehicle
US9875415B2 (en) Method and system for classifying painted road markings in an automotive driver-vehicle-asistance device
US20130342694A1 (en) Method and system for use of intrinsic images in an automotive driver-vehicle-assistance device
JP2003242505A (en) Automatic method for determining illuminant of digital color image
Yang et al. An RGB channel operation for removal of the difference of atmospheric scattering and its application on total sky cloud detection
CN112833812B (en) Measuring instrument for testing a sample and method for determining a height map of a sample
KR101687712B1 (en) Image processing apparatus using spectrum equipment and control method thereof
US20220326145A1 (en) Inspection method, program, and inspection system
WO2016081240A1 (en) Method and system for classifying painted road markings in an automotive driver-vehicle-assistance device
WO2007004864A1 (en) Method and apparatus for visual object recognition
WO2015193470A1 (en) Mobile road sign reflectometer
Babari et al. Computer vision for the remote sensing of atmospheric visibility
WO2014096398A1 (en) System and method for multiline retroreflection measurement of road markings
CN115656053A (en) Rock mineral content testing method and system
US10724920B2 (en) Method of analyzing a light distribution of a vehicle headlight

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15736413

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 15736413

Country of ref document: EP

Kind code of ref document: A1