GB2598082A

GB2598082A - Road mirror detection system and method

Info

Publication number: GB2598082A
Application number: GB2010474.1A
Authority: GB
Inventors: Itagaki Noriaki
Original assignee: Continental Automotive GmbH
Current assignee: Continental Automotive GmbH
Priority date: 2020-07-08
Filing date: 2020-07-08
Publication date: 2022-02-23
Also published as: JP7128941B2; JP2022016359A; GB202010474D0

Abstract

The invention relates to a road mirror detection method, comprising: obtaining an image of a road scene; determining and analysing optical flow lines in the image of the road scene; and determining whether the image comprises an image of a road mirror based on said analysis. The method could also identify different regions (e.g. 178 and 186) within the image and analyse optical flow lines within these regions. A road mirror could be identified as corresponding to a region where optical flow lines 188 do not converge to a point at which lines in other regions do converge. The invention is aimed at autonomous vehicles, giving them a means for detecting road mirrors which present a hazard that is difficult to detect. Some embodiment methods first find a horizon 168 and a vanishing point 164 for optical flow lines and use the fact that mirrors are likely to be in the upper portion of a captured image, i.e. above the horizon. The density of OFLs can be compared to a threshold so as to eliminate insignificant regions.

Description

ROAD MIRROR DETECTION SYSTEM AND METHOD FIELD OF THE INVENTION

The invention relates to a road mirror detection system for detectingroadmirrors, andacorrespondingroadmirrordetection method.

BACKGROUND

Traffic accidents frequently occur at traffic junctions, such as box junctions or T-junctions, because it is usually difficult to detect oncoming traffic approaching the traffic junctions from adjoining roads. Hence, road mirrors have been placed at traffic junctions to help improve the detection of oncoming traffic approaching the traffic junctions from adjoining roads. However, it maybe difficult for autonomous vehicular driving systems to detect road mirrors because these may be of various shapes or tilted at awkward angles.

SUMMARY

An objective is to provide a road mirror detection system for reliably detecting road mirrors, or a corresponding method.

According to a first aspect of the invention, there is provided a road mirror detection method comprising the acts of: obtaining an image of a road scene; determining optical flow lines in the image of the road scene; analysing the optical flow lines in the image of the road scene; and determining whether the image of the road scene comprises an image of a road mirror based on the analysis of the optical flow lines in the image of the road scene.

An image of a road scene may comprise a scene of a road captured by an image capturing device, such as a front camera of a vehicle. The image of the road scene may comprise an image of a ground, an image of a horizon and an image of a space above the ground. The image of the ground may comprise an image of the road on which the vehicle is travelling or images of pavements that sandwich the road. The image of the horizon may comprise an image of a line where the ground seems to meet the sky or an image of a vanishing point on which lane markings on the road converge. The image of the space above the ground may comprise an image of buildings, an image of a road mirror, an image of traffic lights or an image of the sky.

One advantage of the road mirror detection method comprising the acts of determining and analysing optical flow lines is that road mirrors maybe reliably detected regardless of the shape of the road mirrors. Moreover, the road mirror detection method maybe able to reliably detect road mirrors that are tilted at awkward angles such that the road mirrors no longer appear to be regularly shaped.

Optionally, the act of determining the optical flow lines comprises the acts of: identifying a first region in the image of the road scene comprising a first set of optical flow lines; and identifying a second region in the image of the road scene comprising a second set of optical flow lines. By detecting different regions with different sets of optical flow lines, road mirrors may, advantageously, be reliably detected regardless of the shape of the road mirrors. Furthermore, road mirrors that are tilted at awkward angles such that the road mirrors no longer appear to be regularly shaped may also be reliably detected.

Optionally, the act of identifying the second region in the image of the road scene comprises the act of identifying the second region in the image of the road scene that is at least partially within the first region in the image of the road scene. In other words, the second region would at least be partially surrounded by the first region. Thus, at least a portion of a perimeter of the second region would at least be adjacent to or touch the first region. Hence, an image of a road mirror would at least be partially within an image of a background of the road mirror. In other words, totally separated regions would be deemed ir-relevant, and thus ignored because an image of a road mirror and an image of a background of the road mirror would be adjacent to each other. Hence, advantageously, the accuracy of the road mirror detection method may be improved because the totally separated regions with different sets of optical flow lines may be ignored, and thus reducing the number of erroneous road mirror detections.

Optionally, the act of determining the optical flow lines further comprises the act of determining whether the first region in the image of the road scene occupies a larger proportion of the image of the road scene than the second region. An image of a road mirror usually only occupies a small proportion of the whole image of a road scene captured by a vehicle's image capturing device. Hence, an image of a background of the road mirror would usually occupy a larger proportion of the image of the road scene than the image of the road mirror. In other words, the image of the background of the road mirror would occupy a larger area than the image of the road mirror. Therefore, the accuracy of the road mirror detection method may, advantageously, be improved by determining whether the first region in the image of the road scene occupies a larger proportion of the image of the road scene than the second region.

Optionally, the act of analysing the optical flow lines comprises the acts of: determining whether the first set of optical flow lines substantially converge on a point; and determining whether the second set of optical flow lines do not substantially converge on the point. The determination of whether the optical flow lines converge may be accurately and efficiently performed, thus, advantageously, improving the accuracy and efficiency of the road mirror detection method.

As used in this summary, in the description below, in the claims below, and in the accompanying drawings, the term "point" means a position, for instance, on an image or a scene, where two lines intersect or appear to intersect when extrapolated. Hence, a point on an image may, for instance, comprise one pixel of the image or a plurality of pixels, as long as it is a position where two lines intersect or appear to intersect when extrapolated.

The optical flow lines are deemed to substantially converge on 5 a point if the optical flow lines converge on an area centred around the point that is 10% of the area of the entire image of the road scene, which may, for example, be calculated in terms of length or the number of pixels. For instance, for an image of a road that is 500 pixels long and 400 pixels wide, the area that 10 optical flow lines would be deemed to substantially converge would be 50 pixels long and 40 pixels wide. Alternatively, for an image of a road that is 30 centimetres long and 20 centimetres wide, the area that optical flow lines would be deemed to substantially converge would be 3 centimetres long and 2 centimetres wide.

Optionally, the act of analysing the optical flow lines further comprises the act of resizing the second region in the image of the road scene to a predetermined number of pixels. Advanta-geously, the analysis of the optical flow lines may be consistently performed, if each region in each image of each road scene is resized to the predetermined number of pixels.

Optionally, the act of determining whether the image of the road scene comprises the image of the road mirror comprises the act of determining whether the second region in the image of the road scene comprises the image of the road mirror, and thus, the road mirror detection method may accurately ascertain whether the second region in the image of the road scene comprises the image of the road mirror.

Optionally, the act of determining whether the second region in the image of the road scene comprises the image of the road mirror comprises the act of determining whether a density of optical flow lines in the second region exceeds a threshold value. This, advantageously, improves the accuracy of the road mirror detection because irrelevant regions with very few or sparse optical flow lines may be ignored, and thus reducing the number of erroneous road mirror detections.

Each optical flow line may, for instance, be one or more pixel thick. In addition, a density of optical flow lines in a region maybe ascertained by determining the number of optical flow lines in the region, which may, for example, be measured in terms of number of pixels.

The road mirror detection method may further comprise the act of determining a vanishing point comprised in the image of the road scene. Hence, the image of the ground In the image of the road scene and the image of the space above the ground in the image of the road scene may be accurately determined after determining the vanishing point.

Optionally, the act of determining the vanishing point from the image of the road scene is performed before the act of determining the optical flow lines in the image of the road scene. Hence, the image of the ground in the image of the road scene and the image of the space above the ground in the image of the road scene may be accurately determined after determining the vanishing point. Since road mirrors are usually found in the space above the ground, the road mirror detection method may focus the road mirror detection on the region above the horizon or the vanishing point. Thus, advantageously, the efficiency and accuracy of the road mirror detection method may be improved.

Optionally, the act of determining the vanishing point comprises the act of geometrically calculating a position of the vanishing point. Since, advantageously, geometric calculation techniques used to determine the position of the vanishing point are established and robust, the accuracy of the road mirror detection method may be improved.

Optionally, the act of geometrically calculating the position of the vanishing point comprises the act of determining a centre of the image of the road scene. Since, advantageously, the centre of the image of the road scene may be efficiently and quickly performed, the efficiency of the road mirror detection method may be improved.

Optionally, the act of determining the vanishing point comprises the act of detecting lane markings on the road in the image of the road scene. The lane markings on the road would seem to converge if the road is long and straight. Since the vanishing point may, advantageously, be efficiently and quickly determined by detecting the lane markings on the road in the image of the road scene, the efficiency of the road mirror detection method may be improved.

Optionally, the act of detecting the lane markings on the road comprises the act of determining whether the lane markings on the road substantially converge on the vanishing point. The lane markings on the road would seem to converge if the road is long and straight. Since the vanishing point may, advantageously, be efficiently and quickly determined by determining whether the lane markings on the road substantially converge, the efficiency of the road mirror detection method may be improved.

Optionally, the act of determining the vanishing point comprises the act of identifying a third region in the image of the road scene comprising a third set of optical flow lines, wherein the third region of the image of the road scene comprises a lower portion of the image of the road scene below the vanishing point. The road mirror detection method would be quicker if it were focussed on the lower portion of the image of the road scene than if it were focussed on the whole image of the road scene. Thus, the vanishing point may, advantageously, be efficiently and quickly determined from the optical flow lines in the lower portion of the image of the road scene.

Optionally, the act of determining the vanishing point further comprises the act of determining whether the third set of optical flow lines substantially converge on the vanishing point. The determination of whether the optical flow lines converge maybe accurately and efficiently performed, thus, advantageously, improving the accuracy and efficiency of the road mirror detection method. The optical flow lines are deemed to substantially converge on the vanishing point if the optical flow lines converge on an area centred around the vanishing point that is 10% of the area of the entire image of the road scene, which may, for example, be calculated in terms of length or the number of pixels.

Optionally, the act of determining the optical flow lines comprises the act of identifying a fourth region in the image of the road scene comprising a fourth set of optical flow lines, wherein the fourth region of the image of the read scene comprises an upper portion of the image of the road scene above the vanishing point. Since road mirrors are usually found in the space above the ground, the road mirror detection method would be quicker if it were focussed on the upper portion of the image of the road scene than if it were focussed on the whole image of the road scene.

Optionally, the act of determining the optical flow lines further comprises the acts of: identifying a first region in the image of the road scene comprised in the fourth region in the image of the road scene, wherein the first region in the image of the road scene comprises a first set of optical flow lines; and identifying a second region in the image of the road scene comprised in the fourth region in the image of the road scene, wherein the second region in the image of the road scene comprises a second set of optical flow lines. Since road mirrors are usually found in the space above the ground, the road mirror detection method would be quicker if it were focussed on the upper portion of the image of the road scene than if it were focussed on the whole image of the road scene. Moreover, by detecting different regions with different sets of optical flow lines, road mirrors may, ad-vantageously, be reliably detected regardless of the shape of the road mirrors. Furthermore, road mirrors that are tilted at awkward angles such that the read mirrors no longer appear to be regularly shaped may also be reliably detected.

Optionally, the act of identifying the second region in the image of the road scene comprises the act of identifying the second region in the image of the road scene that is at least partially within the first region in the image of the road scene. In other words, the second region would at least be partially surrounded by the first region. Thus, at Least a portion of a perimeter of the second region would at least be adjacent to or touch the first region. Hence, an image of a road mirror would at least be partially within an image of a background of the road mirror. In other words, totally separated regions would be deemed irrelevant, and thus ignored because an image of a road mirror and an image of a background of the road mirror would be adjacent to each other. Hence, advantageously, the accuracy of the road mirror detection method may be improved because the totally separated regions with different sets of optical flow lines may be ignored, and thus reducing the number of erroneous road mirror detections.

Optionally, the act of analysing the optical flow lines comprises the act of: determining whether the first set of optical flow lines substantially converge on the vanishing point; and de-termining whether the second set of optical flow lines do not substantially converge on the vanishing point. The determination of whether the optical flow lines converge may be accurately and efficiently performed, thus, advantageously, improving the 5 accuracy and efficiency of the road mirror detection method. The optical flow lines are deemed to substantially converge on a vanishing point if the optical flow lines converge on an area centred around the vanishing point that is 10% of the area of the entire image of the road scene, which may, for example, be 10 calculated in terms of length or the number of pixels.

Any feature or step disclosed in the context of the first aspect of the invention may also be used, to the extent possible, in combination with and/or in the context of other aspects of the invention, and in the inventions generally. In addition, any feature or step disclosed in the context of any other aspect of the invention may also be used, to the extent possible, in combination with and/or in the context of the first aspect of the Invention, and in the inventions generally.

According to a second aspect of the invention, there is provided a road mirror detection system comprising: an image retriever configured to obtain an image of a road scene; an optical flow line determiner configured to determine optical flow lines in the image of the road scene; an optical flow line analyser configured to analyse the optical flow lines in the image of the road scene: and a road mirror image determiner configured to determine whether the image of the road scene comprises an image of a road mirror based on the analysis of the optical flow lines in the image of the road scene.

One advantage of the road mirror detection system comprising the optical flow line determiner and optical flow line analyser is that road mirrors maybe reliably detected regardless of the shape of the road mirrors. Moreover, the road mirror detection system may be able to reliably detect road mirrors that are tilted at awkward angles such that the road mirrors no longer appear to be regularly shaped.

Optionally, the optical flow line determiner comprises an optical flow line region identifier configured to identify a region in the image of the road scene comprising a set of optical flow lines. The optical flow line region identifier is able to, advanta-geously, identify different regions with different sets of optical flow lines, and thus, the efficiency and accuracy of the road mirror detection system may be improved.

Optionally, the optical flow line analyser comprises an optical flow lines convergence determiner configured to determine whether the set of optical flow lines substantially converge on a point. The optical flow lines convergence determiner is able to accurately and efficiently determine whether the set of optical flow lines converge, and thus improve the accuracy and efficiency of the road mirror detection system. The optical flow lines are deemed to substantially converge on a point if the optical flow lines converge on an area centred around the point that is 10% of the area of the entire image of the road scene, which may, for example, be calculated in terms of length or the number of pixels.

Optionally, the optical flow line analyser further comprises an image resizer configured to resize the region in the image of the road scene to a predetermined number of pixels. The image resizer is able to allow the road mirror detection system to consistently analyse optical flow lines by resizing each region in each image of each road scene to the predetermined number of pixels.

Optionally, the road mirror image determiner comprises an optical flow line density determiner configured to determine a density of optical flow lines in the region. The optical flow line density determiner is able to improve the accuracy of the road mirror detection system by ignoring irrelevant regions with very few or sparse optical flow lines, and thus reducing the number of erroneous road mirror detections.

The road mirror detection system may further comprise a vanishing 25 point determiner configured to determine a vanishing point comprised in the image of the road scene. Hence, the road mirror detection system may accurately determine the image of the ground in the image of the road scene and the image of the space above the ground in the image of the road scene after it has determined the vanishing point. Since road mirrors are usually found in the space above the ground, the road mirror detection system is able to focus the road mirror detection on the region above the horizon or the vanishing point. Thus, advantageously, the efficiency and accuracy of the road mirror detection system may be improved.

Optionally, the vanishing point determiner comprises a vanishing point geometric calculator configured to geometrically calculate a position of the vanishing point. Since, advantageously, the vanishing point geometric calculator is able to use established and robust geometric calculation techniques to geometrically calculate the position of the vanishing point, the accuracy of the road mirror detection system may be improved.

Optionally, the vanishing point determiner further comprises a vanishing point lane marking determiner configured to detect lane markings on the road in the image of the road scene. The lane markings on the road would seem to converge if the road is long and straight. Since, the vanishing point lane marking determiner is able to efficiently and quickly determine the vanishing point by detecting the lane markings on the road in the image of the road scene, the efficiency of the road mirror detection system may be improved.

Optionally, the vanishing point determiner further comprises an optical flow line region identifier configured to identify another region in the image of the road scene comprising another set of optical flow lines. The optical flow line region identifier is able to, advantageously, identify different regions with different sets of optical flow lines, and thus, the efficiency and accuracy of the road mirror detection system maybe improved.

Optionally, the vanishing point determiner further comprises an optical flow lines convergence determiner configured to determine whether the another set of optical flow lines substantially converge on the vanishing point. The optical flow lines convergence determiner is able to accurately and efficiently determine whether a set of optical flow lines converge, and thus improve the accuracy and efficiency of the road mirror detection system. The optical flow lines are deemed to substantially converge on a point if the optical flow lines converge on an area centred around the point that is 10% of the area of the entire image of the road scene, which may, for example, be calculated in terms of length or the number of pixels.

Optionally, a road vehicle may comprise the road mirror detection system.

Any feature or step disclosed in the context of the second aspect of the invention may also be used, to the extent possible, in combination with and/or in the context of other aspects of the invention, and in the inventions generally. In addition, any feature or step disclosed in the context of any other aspect of the invention may also he used, to the extent possible, in combination with and/or in the context of the second aspect of the invention, and in the inventions generally.

According to a third aspect of the Invention, there is provided a road mirror detection method comprising the acts of: obtaining an image of a road scene; determining a vanishing point comprised in the image of the road scene, wherein the act of determining the vanishing point comprises the acts of: geometrically calculating a position of the vanishing point, wherein the act of geometrically calculating the position of the vanishing point comprises the act of determining a centre of the image of the road scene; detecting lane markings on the road in the image of the road scene; identifying a third region in the image of the road scene comprising a third set of optical flow lines, wherein the third region of the image of the road scene comprises a lower portion of the image of the road scene below the vanishing point; and determining whether the third set of optical flow lines substantially converge on the vanishing point; determining optical flow lines in the image of the road scene, wherein the act of determining the optical flow lines further comprises the acts of: identifying a fourth region in the image of the road scene comprising a fourth set of optical flow lines, wherein the fourth region of the image of the road scene comprises an upper portion of the image of the road scene above the vanishing point; identifying a first region in the image of the road scene comprised in the fourth region in the image of the road scene, wherein the first region in the image of the road scene comprises a first set of optical flow lines; identifying a second region in the image of the road scene comprised in the fourth region in the image of the road scene, wherein the second region in the image of the road scene comprises a second set of optical flow lines; wherein the act of identifying the second region in the image of the road scene comprises the act of identifying the second region in the image of the road scene that is at least partially within the first region in the image of the road scene; and determining whether the first region in the image of the road scene occupies a larger proportion of the image of the road scene than the second region; and analysing the optical flow lines in the image of the road scene, wherein the act of analysing the optical flow lines comprises the acts of: determining whether the first set of optical flow lines substantially converge on the vanishing point; and determining whether the second set of optical flow lines do not substantially converge on the vanishing point; and resizing the second region in the image of the road scene to a predetermined number of pixels; and determining whether the image of the road scene comprises an image of a road mirror based on the analysis of the optical flow lines in the image of the road scene; wherein the act of determining whether the image of the road scene comprises the image of the road mirror comprises the act of determining whether the second region in the image of the road scene comprises the image of the road mirror; and wherein the act of determining whether the second region in the image of the road scene comprises the image of the road mirror comprises the act of determining whether a density of optical flow lines in the second region exceeds a threshold value.

Any feature or step disclosed in the context of the third aspect of the invention may also be used, to the extent possible, in combination with and/or in the context of other aspects of the invention, and in the inventions generally. In addition, any feature or step disclosed in the context of any other aspect of the invention may also be used, to the extent possible, in combination with and/or in the context of the third aspect of the invention, and in the inventions generally.

In this summary, in the description below, in the claims below, and in the accompanying drawings, reference is made to particular features (including method steps) of the invention. It is to be understood that the disclosure of the invention in this specification includes all possible combinations of such particular features. For example, where a particular feature is disclosed in the context of a particular aspect or embodiment of the invention, or a particular claim, that feature can also be used, to the extent possible, in com-bination with and/or in the context of other particular aspects and embodiments of the invention, and in the inventions generally.

In this summary, in the description below, in the claims below, and in the accompanying drawings, where reference is made herein to a method comprising two or more defined steps, the defined steps can be carried out in any order or simultaneously (except where the context excludes that possibility), and the method can include one or more other steps which are carried out before any of the defined steps, between two of the defined steps, or after all the defined steps (except where the context excludes that possibility).

As used in this summary, in the description below, in the claims below, and in the accompanying drawings, the term "comprises" and grammatical equivalents thereof are used herein to mean that other components, ingredients, steps, et cetera are optionally present. For example, an article "comprising" (or "which comprises") components A, B, and C can consist of (that is, contain only) components A, B, and C, or can contain not only components A B, and C but also one or more other components.

As used in this summary, in the description below, in the claims below, and in the accompanying drawings, the term "at least" followed by a number is used in to denote the start of a range beginning with that number (which may be a range having an upper limit or no upper limit, depending on the variable being defined) . For example, "at least 1" means 1 or more than 1. The term "at most" followed by a number is used herein to denote the end of a range ending with that number (which may be a range having 1 or 0 as its lower limit, or a range having no lower limit, depending on the variable being defined). For example, -at most 4" means 4 or less than 4, and "at most 40%" means 40% or less than 40%. When, in this specification, a range is given as "(a first number) to (a second number)" or "(a first number) -(a second number)", this means a range whose lower limit is the first number and whose upper limit is the second number For example, 25 to 100 mm means a range whose lower limit is 25 mm, and whose upper limit is 100 mm.

As used in this summary, in the description below, in the claims below, and in the accompanying drawings, the term "volatile memory" means any type of computer memory where the contents of the memory are lost if there is no power to the computer.

Random-access memory (RAM) is an example of a type of volatile memory. As used in the summary above, in this description, in the claims below, and in the accompanying drawings, the term "nonvolatile memory" or the term 'non-transitory comput- er-readable medium" means any type of computer memory where the 25 contents of the memory are retained even if there is no power to the computer. Hard disk and solid-state drive (SSD) are examples of types of nonvolatile memory or non-transitory comput-er-readable medium.

As used in this summary, in the description below, in the claims below, and in the accompanying drawings, the term "image" means a two-dimensional or three-dimensional picture of an actual location in the real world. An image maybe captured by one single image capturing device, such as a camera or a LiDAR sensor, or created by fusing data from several devices, such as ultrasonic sensor, LiDAR sensor, radar sensor or camera.

As used in this summary, in the description below, in the claims below, and in the accompanying drawings, the term "within" means "inside a perimeter of" or "touching at least a portion of a perimeter of".

As used in this summary, in the description below, in the claims below, and in the accompanying drawings, the term -road mirror" means a mirror configured to be placed at traffic junctions to help improve the detection of oncoming traffic approaching the traffic junctions from adjoining roads, but does not include reflective glass on buildings or a puddle of water.

As used in this summary, in the description below, in the claims below, and in the accompanying drawings, the term "converge" means to move towards a location, such as a point, an area or a region, from different directions and meet at the location, or to move towards the location from different directions and do not actually meet but meet at the location when extrapolated. However, two lane markings along a long straight road may appear to converge or substantially converge on a point. Hence, an image of the road would comprise an image of the lane markings converging or substantially converging on a point or a vanishing point.

As used in this summary, in the description below, in the claims below, and in the accompanying drawings, the term "vanishing point" means a point in the distance at which two parallel lines appear to meet.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features, aspects, and advantages will become better understood with regard to the following description, appended claims, and accompanying drawings where: Figure 1 shows a road mirror detection system; Figure 2 shows a road vehicle comprising the road mirror detection system of Figure 1; Figure 3 shows a diagram for a road mirror detection method using the road mirror detection system of Figure 1; Figure 4 shows an image of a road scene comprising lane markings along a long straight road that converge on a primary vanishing 5 point; Figure 5 shows an image of a road scene comprising a first region comprising a first set of optical flow lines; Figure 6 shows a centre of an image of a road scene; and Figure 7 shows optical flow lines of an entire image of a road scene.

In the drawings, like parts are denoted by like reference numerals.

DESCRIPTION

In the summary above, in this description, in the claims below, and in the accompanying drawings, reference is made to particular features (including method steps) of the invention. It is to be 20 understood that the disclosure of the invention in this specification includes all possible combinations of such particular features. For example, where a particular feature is disclosed in the context of a particular aspect or embodiment of the invention, or a particular claim, that feature can also be used, to the extent possible, in combination with and/or in the context of other particular aspects and embodiments of the invention, and in the inventions generally.

Figure 1 shows a road mirror detection system 100 comprising an image retriever 110, a vanishing point determiner 120, an optical flow line determiner 130, an optical flow line analyser 140 and a road mirror image determiner 150. The road mirror detection system 100 may comprise software, hardware or a combination of software and hardware.

The image retriever 110 is configured to obtain an image from an image capturing device. The image retriever 110 may obtain an image of a road scene captured by a camera of a vehicle. The image retriever 110 may comprise software, hardware or a combination of software and hardware.

The vanishing point determiner 120 may comprise a vanishing point geometric calculator 122, a vanishing point lane marking determiner 124, an optical flow line region identifier 126 or an optical flow lines convergence determiner 128. The vanishing point determiner 120 determines a vanishing point from an image of a road scene. The image of the road scene may comprise a primary vanishing point of the entire image of the road scene that represents a vanishing point of the entire road scene captured by an image capturing device of a vehicle. The vanishing point determiner 120 is configured to determine the primary vanishing point of the entire image of the road scene. The image of the road scene may also have several regions, each region comprising a respective secondary vanishing point. The vanishing point determiner 120 is configured to determine the secondary vanishing points of the regions. The primary vanishing point may be distinguished from the secondary vanishing points in that the primary vanishing point is the vanishing point of the entire image of the road scene whilst the secondary vanishing points are each a vanishing point of a region in the image of the road scene. The vanishing point determiner 120 may comprise software, hardware or a combination of software and hardware.

The road mirror detection system 100 may be able to accurately determine the image of the ground in the image of the road scene and the image of the space above the ground in the image of the road scene after the vanishing point determiner 120 determines 5 the primary vanishing point of the entire image of the road scene. Since road mirrors are usually found in the space above the ground, the road mirror detection system 100 may focus the road mirror detection on the region above the horizon or the primary vanishing point. Thus, advantageously, the efficiency and 10 accuracy of the road mirror detection system 100 maybe improved.

The vanishing point geometric calculator 122 is configured to geometrically calculate the position of the vanishing point of the image of the road scene. Since, advantageously, the vanishing point geometric calculator 122 is able to use established and robust geometric calculation techniques to geometrically calculate the position of the vanishing point of the image of the road scene, the accuracy of the road mirror detection system 100 maybe improved. The vanishing point geometric calculator 122 may be configured to calculate the position of the vanishing point of the image of the road scene by determining a centre of the image of the road scene. Since, advantageously, the vanishing point geometric calculator 122 is able to use efficient and straightforward techniques to determine the centre of the image of the road scene, the efficiency of The road mirror detection system 100 may be improved. The vanishing point geometric calculator 122 may comprise software, hardware or a combination of software and hardware.

The vanishing point lane marking determiner 124 is configured to detect lane markings on the road in the image of the road scene. The lane markings on the road would seem to converge if the road is long and straight. Since, the vanishing point lane marking determiner 124 is able to efficiently and quickly determine a vanishing point by detecting the lane markings on the road in the image of the road scene, the efficiency of the road mirror detection system 100 may be improved. The vanishing point lane marking determiner 124 may comprise software, hardware or a combination of software and hardware.

The optical flow line region identifier 126 is configured to identify a region in the image of the road scene comprising a set of optical flow lines. The optical flow line region identifier 126 is also configured to identify different regions in the image of the road scene, each region comprising a respective set of optical flow lines. A region in an image of a road scene comprising a set of optical flow lines may, for example, be an image of a road mirror, an image of a background of a road mirror or an image of the ground. Since the optical flow line region identifier 126 is able to identify different regions with different sets of optical flow lines, the optical flow line region identifier 126 is able to reliably detect road mirrors regardless of the shape of the road mirrors. The optical flow line region identifier 126 is also able to reliably detect road mirrors that are tilted at awkward angles such that the road mirrors no longer appear to be regularly shaped. The optical flow line region identifier 126 may comprise software, hardware or a combination of software and hardware.

The optical flow lines convergence determiner 128 is configured to determine whether a set of optical flow lines substantially converge on a vanishing point. The optical flow lines convergence determiner is able to accurately and efficiently determine whether the set of optical flow lines converge, and thus improve the accuracy and efficiency of the road mirror detection system 100. The optical flow lines are deemed to substantially converge on a point if the optical flow lines converge on an area centred around the point that is 10% of the area of the entire image of the road scene, which may, for example, be calculated in terms of length or the number of pixels. The optical flow lines convergence determiner 128 may comprise software, hardware or a combination of software and hardware.

The vanishing point determiner 120 may comprise at least one of the vanishing point geometric calculator 122, the vanishing point lane marking determiner 124, the optical flow line region identifier 126 or the optical flow lines convergence determiner 128. The vanishing point determiner 120 may comprise the vanishing point geometric calculator 122, the vanishing point lane marking determiner 124, the optical flow line region identifier 126 and the optical flow lines convergence determiner 128. The vanishing point determiner 120 may also comprise a priority system, wherein the vanishing point geometric calculator 122, the vanishing point lane marking determiner 124, the optical flow line region identifier 126 or the optical flow lines convergence determiner 128 are placed in order of priority. For instance, the vanishing point lane marking determiner 124 may be given priority over the vanishing point geometric calculator 122, the optical flow line region identifier 126 and the optical flow lines convergence determiner 128, such that the vanishing point lane marking determiner 124 would be used to determine a vanishing point whenever possible, such as when lane markings on a long straight road are detected.

The optical flow line determiner 130 comprises the optical flow line identifier 126. The optical flow line determiner 130 is configured to determine optical flow lines in an image of a road scene. The optical flow line determiner 130 may comprise software, hardware or a combination of software and hardware. 25 The optical flow line analyser 140 comprises the optical flow line convergence determiner 128 and an image resizer 142. The optical flow line analyser 140 may comprise software, hardware or a combination of software and hardware.

The image resizer 142 is configured to resize the region in the image of the road scene to a predetermined number of pixels. For instance, if the predetermined number of pixels is 30 pixels by 30 pixels, the image resizer 142 may resize each region in the image of the road scene to 30 pixels by 30 pixels. Hence, if every region that may potentially comprise an image of a road mirror is resized to the predetermined number of pixels, for example, 30 pixels by 30 pixels, then the road mirror detection system 100 is able to consistently analyse optical flow lines in every resized region. The image resizer 142 may comprise software, hardware or a combination of software and hardware.

The road mirror image determiner 150 comprises an optical flow line density determiner 152. The road mirror image determiner 150 is able to determine whether an image of a road scene comprises an image of a road mirror based on the analysis of optical flow lines in the image of the road scene. The road mirror image determiner 150 is able to determine whether a region in the image of the road scene comprises the image of the road mirror. The road mirror image determiner 150 may comprise software, hardware or a combination of software and hardware.

The optical flow line density determiner 152 is configured to determine a density of optical flow lines in a region in an image of a road. The optical flow line density determiner 152 is able to improve the accuracy of the road mirror detection system 100 by ignoring irrelevant regions with very few or sparse optical flow lines, and thus reducing the number of erroneous road mirror detections. The optical flow line density determiner 152 maybe configured to determine whether the density of optical flow lines in the region in the image of the road exceed a threshold value. The optical flow line density determiner 152 may also be paired with the image resizer 142 such that the consistency of the road mirror detection system 100 maybe improved. The optical flow line density determiner 152 may comprise software, hardware or a combination of software and hardware.

Figure 2 shows a road vehicle 190 comprising the road mirror detection system 100.

Figure 3 shows a diagram for a road detection method 200 using the road detection system 100. A non-transitory comput-er-readable medium may comprise instructions stored thereon, that when executed by a processor, perform the road detection method 200.

At step 202, the road detection method 200 initialises. At step 204, the image retriever 110 obtains an image of a road scene from an image capturing device of the road vehicle 190. At step 206, the vanishing point determiner 120 determines a primary vanishing point of the image of the road scene.

At step 208, the process of step 206 starts. Figure 4 shows an image of a road scene comprising lane markings 166 along a long straight road that converge on a primary vanishing point 164. At step 210, the vanishing point lane marking determiner 124 detects the lane markings 166 on the road in the image of the road scene. A horizon 168 may be considered to comprise a horizontal line comprising the primary vanishing point 164 drawn across the image of the road. An upper portion 170 of the image of the road may be considered to be the region above the primary vanishing point 164 or the horizon 168, and a lower portion 172 of the image of the road may be considered to be the region below the primary vanishing point 164 or the horizon 168. Hence, the upper portion 170 of the image of the road may, for example, comprise an image of the sky, an image of a road mirror or an image of buildings, and the lower portion 172 of the image of the road may comprise an image of a ground.

Figure 5 shows an image of a road scene comprising a first region 174 comprising a first set of optical flow lines 176. At step 212, the optical flow line region identifier 126 identifies a first region 174 in the image of the road scene comprising a first set of optical flow lines 176, wherein the first region 174 of the image of the road scene comprises a lower portion 172 of the image of the road scene. Since, the optical flow line region identifier 126 focuses on the lower portion 172 of the image of the road scene instead of the whole image of the road scene, the efficiency of the road detection method 200 may be improved.

At step 214, the optical flow lines convergence determiner 128 determines whether the first set of optical flow lines 176 substantially converge on a primary vanishing point 164. The first set of optical flow lines 176 are deemed to substantially converge on the primary vanishing point 164 if the first set of optical flow lines 176 converge on an area centred around the primary vanishing point 164 that is 10% of the area of the entire image of the road scene, which may, for example, be calculated 5 in terms of length or the number of pixels. A horizon 168 maybe considered to comprise a horizontal line comprising the primary vanishing point 164 drawn across the image of the road. Hence, an upper portion 170 of the image of the road may, for example, comprise an image of the sky, an image of a road mirror or an image 10 of buildings, and the lower portion 172 of the image of the road may comprise an image of a ground.

Figure 6 shows a centre 162 of an image of a road scene. At step 216, the vanishing point geometric calculator 122 geometrically calculates a position of the primary vanishing point 164. The vanishing point geometric calculator 122 may calculate the position of the primary vanishing point 164 by determining a centre 162 of the image of the road scene. The centre 162 of the image of the road scene 160 is considered to be the primary vanishing point 164 of the entire image of the road scene.

Thereafter, the process of step 206 ends at step 218.

Figure 7 shows optical flow lines of the entire image of the road scene.

Then, at step 220, the optical flow line determiner 130 determines the optical flow lines in the image of the road scene. At step 222, the process of step 220 starts. At step 224, the optical flow line region identifier 126 identifies a second region 178 in the image of the road scene comprising a second set of optical flow lines 180, wherein the second region 178 of the image of the road scene comprises an upper portion 170 of the image of the road scene above a primary vanishing point 164. Since road mirrors are usually found in the space above the ground, the road mirror detection method 200 would be quicker if the road mirror detection method 200 were focussed on the upper portion 170 of the image of the road scene than if it were focussed on the whole image of the road scene.

At step 226, the optical flow line region identifier 126 identifies a third region 182 in the image of the road scene comprised in the second region 178 in the image of the road scene, wherein the third region 182 in the image of the road scene comprises a third set of optical flow lines 184. The third region 182 in the image of the road scene may be an image of a background 10 of a road mirror.

At step 228, the optical flow line region identifier 126 identifies a fourth region 186 in the image of the road scene comprised in the second region 178 in the image of the road scene, wherein the fourth region 186 in the image of the road scene comprises a fourth set of optical flow lines 188. The fourth region 186 in the image of the road scene may be an image of a road mirror. By detecting different regions with different sets of optical flow lines, road mirrors may, advantageously, be reliably detected regardless of the shape of the road mirrors. Furthermore, road mirrors that are tilted at awkward angles such that the road mirrors no longer appear to be regularly shaped may also be reliably detected.

At step 230, the optical flow line region identifier 126 identifies the fourth region 186 in the image of the road scene that is at least partially within the third region 182 in the image of the road scene. In other words, the fourth region 186 would at least be partially surrounded by the third region 182. Thus, at least a portion of a perimeter of the fourth region 186 would at least be adjacent to or touch the third region 182. Hence, an image of a road mirror would at least be partially within an image of a background of the road mirror. In other words, totally separated regions would be deemed irrelevant, and thus ignored because an image of a road mirror and an image of a background of the road mirror would be adjacent to each other. Hence, advantageously, the accuracy of the road mirror detection method 200 may be improved because the totally separated regions with different sets of optical flow lines may be ignored, and thus reducing the number of erroneous road mirror detections.

At step 232, the optical flow line determiner 130 determines whether the third region 182 in the image of the road scene occupies a larger proportion of the image of the road scene than the fourth region 186. An image of a road mirror usually only occupies a small proportion of the whole image of a road scene captured. Hence, an image of a background of the road mirror would usually occupy a larger proportion of the image of the road scene than the image of the road mirror. In other words, the image of the background of the road mirror would occupy a larger area than the image of the road mirror. Therefore, the accuracy of the road mirror detection method 200 may, advantageously, be improved by determining whether third region 182 in the image of the road scene occupies a larger proportion of the image of the road scene than the fourth region 186.

Thereafter, at step 234, the process of step 220 ends.

At step 236, the optical flow line analyser 140 analyses the optical flow lines in the image of the road scene. At step 238, the process of step 236 starts. At step 240, the optical flow lines convergence determiner 128 determines whether the third set of optical flow lines 184 substantially converge on the primary vanishing point 164. If the third set of optical flow lines 184 substantially converge on the primary vanishing point 164, the third region 182 in the image of the road scene may be an image of a background of a road mirror.

At step 242, the optical flow lines convergence determiner 128 determines whether the fourth set of optical flow lines 188 do not substantially converge on the primary vanishing point 164. If the fourth set of optical flow lines 188 do not substantially converge on the primary vanishing point 164, the fourth region 186 in the image of the road scene may be an image of a road mirror.

At step 244, the image resizer 142 resizes the fourth region 186 in the image of the road scene to a predetermined number of pixels. For instance, if the predetermined number of pixels is 30 pixels by 30 pixels, the image resizer 142 may resize the fourth region 186 in the image of the road scene to 30 pixels by 30 pixels.

Thereafter, at step 246, the process of step 236 ends.

At step 248, the road mirror image determiner 150 determines whether the image of the road scene comprises an image of a road mirror based on the analysis of the optical flow lines in the image of the road scene. At step 250, the process of step 248 starts. At step 252, the road mirror image determiner 150 determines whether the fourth region 186 in the image of the road scene comprises an image of a road mirror. The road mirror image determiner 150 determines that the fourth region 186 in the image of the road scene comprises an image of a road mirror if the fourth set of optical flow lines 188 do not substantially converge on the primary vanishing point 164 whilst the third set of optical flow lines 184 substantially converge on the primary vanishing point 164.

At step 254, the optical flow line density determiner 152 determines whether a density of optical flow lines in the second region exceeds a threshold value. This, advantageously, improves the accuracy of the road mirror detection method 200 because irrelevant regions with very few or sparse optical flow lines may be ignored, and thus reducing the number of erroneous road mirror detections.

Thereafter, at step 256, the process of step 248 ends.

Finally, at step 258, the road mirror detection method 200 ends.

One advantage of the road mirror detection method 200 comprising the acts of determining and analysing optical flow lines is that road mirrors may be reliably detected regardless of the shape of the road mirrors. Moreover, the road mirror detection method 200 may be able to reliably detect road mirrors that are tilted at awkward angles such that the road mirrors no longer appear to be regularly shaped.

Although the invention has been described in considerable detail with reference to certain embodiments or aspects, other embodiments or aspects are possible.

For example, the various vanishing point determination steps 210, 10 212, 214, 216 of step 206 may be performed in any order or sequence, or Indeed simultaneously.

Therefore, the spirit and scope of the appended claims should not he limited to the description of the embodiments contained 15 herein.

All features disclosed in this specification (including the appended claims, abstract, and accompanying drawings) may be replaced by alternative features serving the same, equivalent, or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic series of equivalent or similar features.

Claims

PATENT CLAIMS1. A road mirror detection method (200) comprising the acts of: obtaining an image of a road scene; determining optical flow lines in the image of the road scene; analysing the optical flow lines in the image of the road scene; and determining whether the image of the road scene comprises 10 an image of a road mirror based on the analysis of the optical flow lines in the image of the road scene.
2. The road mirror detection method (200) of claim 1, wherein the act of determining the optical flow lines comprises the acts 15 of: identifying a first region in the image of the road scene comprising a first set of optical flow lines; and identifying a second region in the image of the road scene comprising a second set of optical flow lines.
3. The road mirror detection method (200) of claim 2, wherein the act of identifying the second region in the image of the road scene comprises the act of identifying the second region in the image of the road scene that is at least partially within the first region in the image of the road scene.
4. The road mirror detection method (200) of anyone of claims 2-3, wherein the act of determining the optical flow lines further comprises the act of determining whether the first region in the image of the road scene occupies a larger proportion of the image of the road scene than the second region.
5. The road mirror detection method (200) of anyone of claims 2-4, wherein the act of analysing the optical flow lines comprises 35 the acts of: determining whether the first set of optical flow lines substantially converge on a point; and determining whether the second set of optical flow lines do not substantially converge on the point.
6. The road mirror detection method (200) of anyone of claims 2-5, wherein the act of analysing the optical flow lines further comprises the act of resizing the second region in the image of the road scene to a predetermined number of pixels.
7. The road mirror detection method (200) of anyone of claims 2-6, wherein the act of determining whether the image of the road scene comprises the image of the road mirror comprises the act of determining whether the second region in the image of the road scene comprises the image of the road mirror.
8. The road mirror detection method (200) of claim 7, wherein the act of determining whether the second region in the image of the road scene comprises the image of the road mirror comprises the act of determining whether a density of optical flow lines in the second region exceeds a threshold value.
9. The road mirror detection method (200) of claim 1, further comprising the act of determining a vanishing point comprised in the image of the road scene.
10. The road mirror detection method (200) of claim 9, wherein the act of determining the vanishing point comprises the act of geometrically calculating a position of the vanishing point.
11. The road mirror detection method (200) of claim 10, wherein 30 the act of geometrically calculating the position of the vanishing point comprises the act of determining a centre (162) of the image of the road scene.
12. The road mirror detection method (200) of anyone of claims 35 9-11, wherein the act of determining the vanishing point comprises the act of detecting lane markings (166) on the road in the image of the road scene.
13 The road mirror detection method (200) of anyone of claims 9-12, wherein the act of determining the vanishing point comprises the act of identifying a third region in the image of the road scene comprising a third set of optical flow lines, wherein the third region of the image of the road scene comprises a lower portion (172) of the image of the road scene below the vanishing point.
14. The road mirror detection method (200) of claim 13, wherein 10 the act of determining the vanishing point further comprises the act of determining whether the third set of optical flow lines substantially converge on the vanishing point.
15. The road mirror detection method (200) of anyone of claims 9-14, wherein the act of determining the optical flow lines comprises the act of identifying a fourth region in the image of the road scene comprising a fourth set of optical flow lines, wherein the fourth region of the image of the road scene comprises an upper portion (170) of the image of the road scene above the vanishing point.
16. The road mirror detection method (200) of claim 15, wherein the act of determining the optical flow lines further comprises the acts of: identifying a first region in the image of the road scene comprised in the fourth region in the image of the road scene, wherein the first region in the image of the road scene comprises a first set of optical flow lines; and identifying a second region in the image of the road scene 30 comprised in the fourth region in the image of the road scene, wherein the second region in the image of the road scene comprises a second set of optical flow lines.
17. The road mirror detection method (200) of claim 16, wherein the act of identifying the second region in the image of the road scene comprises the act of identifying the second region in the image of the road scene that is at least partially within the first region in the image of the road scene.
18. The road mirror detection method (200) of anyone of claims 16-17, wherein the act of determining the optical flow lines further comprises the act of determining whether the first region in the image of the road scene occupies a larger proportion of the image of the road scene than the second region.
19. The road mirror detection method (200) of anyone of claims 16-18, wherein the act of analysing the optical flow lines 10 comprises the acts of: determining whether the first set of optical flow lines substantially converge on the vanishing point; and determining whether the second set of optical flow lines do not substantially converge on the vanishing point.
20. The road mirror detection method (200) of anyone of claims 16-19, wherein the act of analysing the optical flow lines further comprises the act of resizing the second region in the image of the road scene to a predetermined number of pixels.
21 The road mirror detection method (200) of anyone of claims 16-20, wherein the act of determining whether the image of the road scene comprises the image of the road mirror comprises the act of determining whether the second region in the image of the road scene comprises the image of the road mirror.
22. The road mirror detection method (200) of claim 21, wherein the act of determining whether the second region in the image of the road scene comprises the image of the road mirror comprises the act of determining whether a density of optical flow lines in the second region exceeds a threshold value.
23. A road mirror detection method (200) comprising the acts of: obtaining an image of a road scene; determining a vanishing point comprised in the image of the road scene, wherein the act of determining the vanishing point comprises the acts of: geometrically calculating a position of the vanishing point, wherein the act of geometrically calculating the position of the vanishing point comprises the act of determining a centre (162) of the image of the road scene; detecting lane markings (166) on the road in the image of the road scene; identifying a third region in the image of the road scene comprising a third set of optical flow lines, wherein the third region of the image of the road scene comprises a lower portion (172) of the image of the road scene below the vanishing point; and determining whether the third set of optical flow lines substantially converge on the vanishing point; determining optical flow lines in the image of the road 15 scene, wherein the act of determining the optical flow lines further comprises the acts of: identifying a fourth region in the image of the road scene comprising a fourth set of optical flow lines, wherein the fourth region of the image of the road scene comprises an upper portion (170) of the image of the road scene above the vanishing point; identifying a first region in the image of the road scene comprised in the fourth region in the image of the road scene, wherein the first region in the image of the road scene 25 comprises a first set of optical flow lines; identifying a second region in the image of the road scene comprised in the fourth region in the image of the road scene, wherein the second region in the image of the road scene comprises a second set of optical flow lines; wherein the act of identifying the second region in the image of the road scene comprises the act of identifying the second region in the image of the road scene that is at least partially within the first region in the image of the road scene; and determining whether the first region in the image of the road scene occupies a larger proportion of the image of the road scene than the second region; and analysing the optical flow lines in the image of the road scene, wherein the act of analysing the optical flow lines comprises the acts of: determining whether the first set of optical flow lines 5 substantially converge on the vanishing point; and determining whether the second set of optical flow lines do not substantially converge on the vanishing point; and resizing the second region in the image of the road scene to a predetermined number of pixels; and determining whether the image of the road scene comprises an image of a road mirror based on the analysis of the optical flow lines in the image of the road scene; wherein the act of determining whether the image of the road scene comprises the image of the road mirror comprises the act 15 of determining whether the second region in the image of the road scene comprises the image of the road mirror; and wherein the act of determining whether the second region in the image of the road scene comprises the image of the road mirror comprises the act of determining whether a density of optical flow 20 lines in the second region exceeds a threshold value.
24. A road mirror detection system (100) comprising: an image retriever (110) configured to obtain an image of a road scene; an optical flow line determiner (130) configured to de-termine optical flow lines in the image of the road scene; an optical flow line analyser (140) configured to analyse the optical flow lines in the image of the road scene; and a road mirror image determiner (150) configured to determine whether the image of the road scene comprises an image of a road mirror based on the analysis of the optical flow lines in the image of the road scene.
25. A road vehicle (190) comprising the road mirror detection 35 system (100) of claim 24.