DE102019102561A1 - Process for recognizing a plaster marking - Google Patents

Abstract

A method for recognizing a pavement marking in an image of the surroundings of a vehicle (1) comprises the acquisition (50) of at least two images of essentially the same surroundings by one or more cameras (3); Identifying (52) within each image at least one point on a surface of a 3-dimensional object within the field of view of the one or more cameras; Excluding (54) pixels within an image of the at least two images, located behind the surface of the 3-dimensional object relative to the camera position, for the image from analysis for the presence of patch marks; Analyzing (56) at least a portion of non-excluded pixels from the image to identify patch marks; and supplying details for at least some of the identified plaster markings to a driver assistance system in order to support the control of the vehicle (1).

Description

area

The present invention relates to a method for recognizing a plaster marking in an image of an environment of a vehicle.

General status

The detection of pavement markings is of great significance for several different driver assistance systems including autonomous driving systems, lane keeping systems or systems for detecting or recognizing parking spaces / spaces. For example, it is desirable for a parking space detection system to recognize the dimensions, the position and the orientation of any parking spaces in a parking area. Such parking space detection systems can be designed to support a parking operation of the vehicle in a detected parking space either actively (for example by intervening in a driving function of the vehicle) or passively (for example by providing driving instructions for a driver of the vehicle).

As is known, for example, plaster markings are recognized on the basis of one or more images captured from the surroundings of the vehicle by forming image gradients over different areas of the images and then recognizing a plaster marking if an image gradient corresponds to a predetermined pattern.

It goes without saying that in order for any driver assistance system to function correctly, it is necessary for patch markings with sufficient degrees of sensitivity and selectivity to be reliably and reliably recognized.

US2008 / 136673A detects parking spaces within a bird's eye view (or to be viewed).

KR2008 / 0024776A detects parking space markings and then the parking space using a Hough transformation.

US2013 / 265429A recognizes a parking space identified by line markings. It uses a corner detection process to detect an angled corner in an image containing the space line marker and a cross detection process to detect a cross point by combining adjacent corners under the detected corners. A space detection process detects a space that is an end periphery of the parking space line marker by combining the detected cross points. As a result, a selection process is used to select the desired type of storage space.

US2014 / 112542A describes a method for recognizing parking spaces defined by line markings. It detects several parking space candidates in an input image and filters them based on the degree of overlap between the detected parking space and the degree of brightness of the lines in order to select the best storage space candidate.

US2016 / 307052A recognizes parking spaces using an input image and vehicle movement based on an in-vehicle motion sensor. It identifies features within the input image after applying an inverse transformation perspective, then constructs a map of these features relative to the vehicle and uses a geometric model to identify the orientation of the parking spaces relative to the vehicle.

US2017 / 096167A discloses a parking guidance device including: an image acquisition device for taking an image around a vehicle; an image processor for generating an around view monitoring (AVM) image by using images obtained by the image acquisition device; and a controller. Upon entering a parking guidance mode, the controller displays a parking position mark near a vehicle within the AVM image, determines a target parking space using the parking position mark, displays a dot mark for setting an initial moving position from which the vehicle can move into the target parking space movement of the vehicle to the initial movement position using the dot mark and causes the vehicle to move to the target parking space when the vehicle reaches the initial movement position.

German patent application no. DE102018113559.9 (Ref: SIE688), filed June 7, 2018, discloses recognizing a lane marking for a vehicle by detecting a potential lane marking in an image of the environment, determining a development of the potential lane marking across a main direction of extension (the potential lane marking as the line width of the potential) Lane marking and recognition of the lane marking by validating the potential lane marking as plausible based on its line width.

Short presentation

According to the present invention, a method for recognizing a plaster marking in an image of an environment of a vehicle according to claim 1 is provided.

Further aspects of the invention provide a control device, a camera system, a driver assistance system, a computer program product and a motor vehicle, designed to carry out the method according to the claims.

Embodiments hide pixels corresponding to 3-dimensional objects from images of an environment because they can be a common source of false positives for any system that tries to detect patch marks. Examples of these false positives include any features, reflections or bright areas on the surfaces of such 3-dimensional objects.

This in turn can allow analysis of the remainder of such images to apply more appropriate criteria to detect patch marks without generating false positives from the surfaces of 3-dimensional objects.

Embodiments of the invention can treat a surface of a 3-dimensional object as the amount of space on a pavement within which a vehicle can navigate. In addition, a driver assistance system advantageously only needs to search within this free space for pavement markings that can be used by the driver assistance system.

An advantage of the present approach is that it targets false positives directly and at the same time has a minimal effect on the rate of true positives.

Figure list

• 1 eine Draufsicht auf ein Fahrzeug mit einem Kamerasystem gemäß einer Ausführungsform der vorliegenden Erfindung;
• 2 ein durch die Kamera von 1 aufgenommenes beispielhaftes Bild;
• 3 aus dem Bild von 2 erfasste Informationen, auf eine horizontale Ebene transformiert;
• 4 aus einer Analyse zur Anwesenheit von Pflastermarkierungen ausgeschlossene Bildpixel gemäß einer Ausführungsform der vorliegenden Erfindung; und
• 5 ein Verfahren gemäß einer Ausführungsform der vorliegenden Erfindung.

An embodiment of the invention will now be described by way of example with reference to the accompanying drawings. Show it:

• 1 a plan view of a vehicle with a camera system according to an embodiment of the present invention;
• 2nd one by the camera of 1 example image taken;
• 3rd from the image by 2nd captured information, transformed to a horizontal level;
• 4th image pixels excluded from analysis for the presence of patch marks according to an embodiment of the present invention; and
• 5 a method according to an embodiment of the present invention.

Description of the embodiment

Now referring to 1 a top view of a vehicle 1 with a camera system 2nd that's a camera 3rd and an evaluation unit 4th contains, shown according to an embodiment of the invention. In the embodiment, the camera is 3rd at a stern 5 of the vehicle 1 arranged so that pictures of at least part of one at the stern 5 of the vehicle 1 surrounding area 6 through the camera 3rd recorded and with a view to recognizing any plaster markings in the surrounding area 6 of the vehicle 1 to the evaluation unit 4th can be delivered.

Note that in the present application, the term pavement refers to any surface over which the vehicle is located 1 may be located while pavement marking includes any traffic control device located on a pavement surface, including line markings, whether continuous or broken, or raised pavement markings, RPMs, the location of which can be usefully used by a driver assistance system.

It is understood, however, that the camera system 2nd may include multiple cameras with a known spatial and orientation relationship and projection model so that, for example, images provided by separate cameras may be stitched together to provide a composite image with a field of view that extends a greater portion of the area around the vehicle than one any single camera could be captured. In this case, for example, such cameras could be on the sides and / or the front of the vehicle 1 arranged and could otherwise be used, for example, to create a bird's eye view of the surroundings of the vehicle. It is understood that the evaluation unit 4th can be anywhere in the vehicle and that the evaluation unit 4th can be integrated into or into each camera or can be designed as a separate unit and / or several component units.

In any case, embodiments of the invention are concerned with identifying 3-dimensional objects such as the object 7 in the surrounding area 6 of the vehicle 1 and taking these objects into account when recognizing any pavement markings in images of the surrounding area 6 . Such 3-dimensional objects can be defined as objects with a surface that extend more than a threshold amount away from a surface of the patch on which the motor vehicle is located 1 located.

The coordinates of such objects 7 can be expressed in a three-dimensional world coordinate system that includes, for example, x, y, and z coordinates relative to an origin. However, it goes without saying that any alternative coordinate system, for example based on polar coordinates, could also be used.

It is believed that embodiments include the location and orientation of the camera 3rd know within the coordinate system. For example, in some embodiments, the location of the camera 3rd defined relative to the origin of the coordinate system with an orientation defined relative to an axis of the system, conventionally this is the x-axis corresponding to a longitudinal axis of the vehicle (note that this does not always have to be parallel to a paved surface). The location and orientation of any other cameras in the system can then be relative to the location of a reference camera such as the camera 3rd To be defined.

Changes in the location and orientation of the camera 3rd from frame to frame, captured by the camera, can be determined, for example, using vehicle odometry, tapped by a CAN bus of the motor vehicle 1 , and / or by visual odometry, based on the or each camera of the vehicle 1 captured images. This odometry enables changes in the angle of the yaw, pitch and / or the roll of the vehicle and translation changes in the position of the vehicle to be determined and applied to the location and orientation of the camera 3rd . The position for placing the masking area on the top view image can be compensated for in-vehicle movement using mechanical odometry from the CAN bus. This should take into account any differences in the time stamps associated with the image arrival time and output time stamp of the object detection algorithms, and thus take into account the vehicle's own movement within this time period.

It is understood that since the vehicle can move over greater distances, it may be preferred to have the location / orientation values of all objects relative to the camera 3rd maintain, so essentially the camera 3rd (or the vehicle 1 ) is treated as if it were at the origin x, y = 0.0 of the coordinate system. It is also understood that the patch in the immediate vicinity of the vehicle can be viewed as substantially flat, ie with a constant z coordinate value, and the height of the camera, ie its z coordinate, can be defined relative to this surface.

WO2016 / 146559 (Ref: SIE0239) discloses an optical flow method for detecting 3-dimensional objects within the surroundings of a vehicle 1 based on capturing a pair of images with a given object 7 , from the camera 3rd delivered.

In a first position and orientation of a vehicle 1 becomes a first picture through the camera 3rd delivered, and in a second different position and / or orientation of the vehicle 1 takes a second picture through the camera 3rd delivered. A respective set of characteristic pixels is determined in each of the first and second images. The characteristic pixels in each image can be determined, for example, using a Harris operator, for example. Any number of techniques can be used to correlate characteristic pixels in a pair of images, for example using texture matching and / or odometry to predict the movement of pixels from one image frame to another. When the respective x, y positions of a given characteristic pixel in each image are known and the respective x, y, z location, the angular orientation and the projection model of the camera 3rd upon acquiring each of the first and second images, it is possible to transform characteristic pixel x, y locations into the three-dimensional world coordinate system. Incidentally, it goes without saying that characteristic pixels are less common on relatively homogeneous pavement surfaces and are more likely to appear on more detailed surfaces of 3-dimensional objects such as other vehicles, buildings or even road signs.

So with reference to 2nd that is a typical view of one from the camera 3rd captured environment 6 illustrated.

For example, applying a Harris operator to this scene will result in a number of characteristic pixels on the surfaces of objects such as the vehicle 7 deliver. Using a different image from essentially the same environment, captured when the camera is on 3rd located in a different position and / or in a different orientation enables the x, y, z locations of those pixels to be determined. The different, in 2nd The vertical lines shown indicate points P that are detected on surfaces with a Z coordinate that is greater than a threshold value height above the surface of the pavement, and are thus considered to be points on one or more surfaces of one or more 3 -dimensional objects.

In some implementations, a recently captured image from a camera can be considered a first reference image are treated, wherein a previously captured (and probably previously processed to determine characteristic pixels) image is treated as a second image that is used to determine the x, y, z locations of characteristic pixels within the reference image.

Now referring to 3rd showing a top view of the surrounding area 6 from 2nd shows, it can be seen how the points P can now be mapped onto the plane of the plaster surface and how a polyline C can be defined on this plane and the points P can be defined separately from the camera position on the plane of the plaster. In one embodiment, the polyline C can be defined with multiple control points CP1 ... CPn, providing the required degree of freedom to closely conform to the outline of the points P from the surface of the object 7 to the level of the pavement surface. Note that any technique for defining the polyline can be used, and can be any form of polyline, including B-spline, Bezier or polynomial curve, and non-linear curves including, for example, multiple discrete line segments.

If the x, y characteristics of any point of the polyline C in the plane of the plaster surface are known, then with knowledge of the location, the angular orientation and the projection model of any camera imaging this environment, it is possible to place those points of the polyline C in the image plane the camera.

Thus, as in 2nd shown the polyline C on any, through the camera 3rd captured image are projected.

If, for example, it is now assumed that the optical axis (OA) of the camera 3rd Aligned to the center of any such image, all image pixels separated from the center of the image by a dot on polyline C can be excluded from analysis for the presence of patch marks.

With reference to 4th therefore pixels with the area EXCL, bounded by the polyline C, the outer edge of the image and lines which extend from the OA through the final control points CP1, CPn of the polyline C when projected into the image plane, are excluded from a pixel map on the presence of pavement markings is analyzed, which in turn can be useful for driver assistance systems.

Note that, especially for cameras with a large field of view, the lines extending from the OA through the ends CP1, CPn of the polyline C to the outer edge of the image need not be straight and can take into account distortion caused by the camera 3rd and / or the camera projection model is caused.

It can be seen that in some cases the points P detected within a given field of view can be divided into more than one unique cluster instead of the spread of points, for the example of FIG 2nd to 4th shown, as an example, the vehicle 1 can cross between two separate rows of parked cars in a parking lot. In this case, more than one polyline C can be defined for any given image. This can lead to several distinct areas, including pixels, that can be excluded from the pixel map that is analyzed for the presence of patch marks.

Alternatively, if the analysis of points P indicates that multiple polylines are required, other implementations may join them to always create a single continuous area within the pixel map that is excluded from analysis for the presence of patch marks.

It is understood that in addition or as an alternative to that in WO2016 / 146559 (Ref: SIE0239) disclosed methods, alternative methods can be used to identify points within an image that lie on the surface of a 3-dimensional object and that are excluded from analysis for the presence of patch marks that may be useful in driver assistance systems can be.

Also revealed WO2017 / 042224 (Ref: SDE1766) a number of methods for identifying pixels on a street surface as different from pixels on the surfaces of 3-dimensional objects. In one example, blocks of pixels within an image whose intensity falls below a predetermined threshold can be examined for their brightness. Then those image blocks that are associated with the road can be recognized with the aid of a corresponding histogram. To determine the pavement surface, the individual image blocks, which are arranged, for example, in several columns and several lines, can be examined. The individual columns can be examined from bottom to top using a corresponding histogram. It can be checked, for example, whether there is a predetermined number of contiguous image blocks, for example three contiguous image blocks. In this case the pavement surface can then be determined. It can also be provided that the center of the at least one of the image blocks associated with the plaster surface in the series is determined. This point can then be viewed as a boundary of the patch surface within an image.

Such a technique could indicate that a limit corresponding to that in 2nd and 4th shown grid G contains pixels that are located on the pavement surface, so that pixels outside this limit are excluded from an analysis for the presence of pavement markings.

As well as surfaces of 3-dimensional objects within the field of view of the camera 3rd understand that other areas of any, through the camera 3rd captured image can also be excluded from the analysis for the presence of plaster marks. For example, as in 2nd and 4th shown any area known to be from the rear surface 5 of the vehicle 1 that appears in the captured image must always be excluded from an analysis for the presence of plaster markings.

As in WO2017 / 042224 (Ref: SDE1766), various techniques can be merged with those referenced and described above to define a final boundary for a pixel map within an image that contains only those pixels on a patch surface that need to be analyzed for the presence of patch marks .

Although the above-described embodiments have been described in terms of capturing multiple images from a given camera, composite images may exist when a vehicle is in different locations and / or orientations to identify 3-dimensional objects within a scene and around the surfaces To exclude from such objects from an analysis for the presence of plaster markings, it goes without saying that, in variants of such embodiments, several images can be captured essentially simultaneously by several cameras or a camera array with known projection models, which are placed close to one another in a known spatial and alignment relationship are to provide several images with clear fields of view of the surroundings of the vehicle, for which a depth map can then be generated for at least a few pixels within those images, around the surfaces of 3-dimensional Identify objects within a scene.

In any event, after the pixels within a scene that could potentially contain patch marks are identified, any suitable technique, such as that referenced above, can be used to identify such patch marks within a given image.

While any such technique can be used as normal to identify patch marks, it is understood that because it can be assumed with more confidence that the pixels being analyzed do not belong to the surface of a 3-dimensional object that is from such tuning technique can actually be relaxed to pick up potentially patch marks that may not be as clearly defined as would be required for a technique that could not make this assumption. As such, the present method can provide greater utility in the operation of driver assistance systems.

In any case, assuming that any marking identified within a captured image is on the plaster surface, i.e. has a known z coordinate, it can be seen how the image coordinates for any identified plaster marking can be transformed from the image plane into the world coordinate system - the inverse of the transformation of the polyline C from the world coordinate system into the image plane.

With those in the same world coordinate system as the vehicle 1 defined markings can now be seen how these can be used by any driver assistance system to control the vehicle 1 to support.

Note that embodiments of the present invention need not be limited to using information from a given image to determine the location of pavement marks within an environment of a vehicle. In particular, in the case where a vehicle moves relatively slowly within an environment, space coordinates for a plaster marking detected in an image can overlap the section of a world coordinate space that is in a subsequent image of the environment 6 is recorded. Thus, the detection of a patch marking within several of a sequence of images can contribute to a degree of confidence in the patch marking, which can then be used by a driver assistance system to consider whether the patch marking should be treated as valid or not. For example, using this technique, an image of a blinking light or a reflection on a pavement could be filtered out from consideration by a driver assistance system.

In other embodiments, knowledge of the likely geometry of a parking space / space can be used to determine which pavement markings might and may not include the outline of a parking space / space, so that those that do not include valid pavement markings can be excluded to be delivered to a driver assistance system. For example, only straight plaster markings that run parallel or orthogonal to other plaster markings on the horizontal plane could be retained after such filtering.

Thus, summarized with reference to FIG 5 in embodiments of the present invention, an evaluation unit 4th at least two images of substantially the same environment from one or more cameras 3rd , Step 50 . The evaluation unit uses the images to identify at least one point on a surface of a 3-dimensional object within the field of view of the one or more cameras, step 52 . The evaluation unit 4th excludes pixels within an image behind the surface of the 3-dimensional object from the analysis for the presence of patch marks, step 54 . The remaining pixels can be further filtered to produce a final pixel map to be analyzed for the presence of patch marks, step 57 . At least a portion of the last pixel map is then analyzed to identify patch marks, step 56 , and these in turn can be filtered, step 59 Before the last set of patch marks identified for a given image of the environment can then be used for any driver assistance system to control the vehicle 1 to support.

QUOTES INCLUDE IN THE DESCRIPTION

This list of documents listed by the applicant has been generated automatically and is only included for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• US 2008136673 A [0005]
• KR 2008/0024776 A [0006]
• US 2013265429 A [0007]
• US 2014112542 A [0008]
• US 2016307052 A [0009]
• US 2017096167 A [0010]
• DE 102018113559 [0011]
• WO 2016/146559 [0027, 0040]
• WO 2017/042224 [0041, 0044]

Claims (13)

A method for recognizing a plaster marking in an image of an environment of a vehicle (1), comprising: Capturing (50) at least two images of substantially the same surroundings from one or more cameras (3), the images being captured from at least one of: a unique camera position or orientation; Identifying (52) within each image at least one point on a surface of a 3-dimensional object within the field of view of the one or more cameras; Excluding (54) pixels within an image of the at least two images, located behind the surface of the 3-dimensional object relative to the camera position, for the image from analysis for the presence of patch marks; Analyzing (56) at least a portion of non-excluded pixels from the image to identify patch marks; and Providing details for at least some of the identified patch markings to a driver assistance system to support the control of the vehicle (1). Procedure according to Claim 1 , further comprising: filtering (57) the non-excluded pixels to exclude further pixels from the image from analyzing for the presence of patch marks. Procedure according to Claim 2 wherein the filtering includes excluding pixels within the image that are known to correspond to a surface of the vehicle. Procedure according to Claim 2 wherein the filtering includes identifying a boundary within the image that contains pixels whose appearance indicates that they correspond to a plaster surface, and further excluding pixels within the image outside the boundary from analyzing for the presence of plaster marks. Procedure according to Claim 1 , further comprising: filtering (59) the identified patch marks to exclude one or more identified patch marks from being delivered to the driver assistance system. Procedure according to Claim 5 wherein the filtering includes applying geometric rules to exclude the one or more identified patch marks. Procedure according to Claim 5 , wherein the filtering includes tracking a history of an identified patch mark in a sequence of images captured in front of the image to determine a level of confidence for each identified patch mark and to exclude any identified patch mark with a confidence level below a threshold. Procedure according to Claim 1 wherein identifying (52) within each image of at least one point on a surface of a 3-dimensional object comprises: locating a set of one or more characteristic pixels in each of the at least two images; Determining a coordinate in a 3-dimensional space for corresponding characteristic pixels from the at least two images; and selecting pixels with a height characteristic over a threshold value as points on a surface of a 3-dimensional object accordingly. Procedure according to Claim 8 , further comprising: determining at least one boundary on a two-dimensional plane that separates the points on a surface of a 3-dimensional object from the camera location; and mapping the boundary into the image plane to form at least a portion of a boundary for the excluded pixels. Computer program product for executing a method according to one of the preceding claims, if the computer program product is executed on a programmable computer device. Camera system (2) with at least one camera (3) and an evaluation unit (4), which is designed to carry out a method according to one of the Claims 1 to 9 . Vehicle control device, comprising the camera system according to Claim 11 and a driver assistance system responsive to receiving the identified patch marks to aid in the control of the vehicle. Motor vehicle (1) with a camera system (2) Claim 11 .

Images