DE102018220018A1 - Method for object detection using a vehicle sensor system - Google Patents

Abstract

In a method for object detection using sensor systems of vehicles (11, 12), a plurality of vehicles (11, 12) are networked with one another. Each vehicle (11, 12) has a sensor system (111, 112; 1111, 1112) for the detection of objects (13). The data and / or information recorded by the individual vehicles (11, 12) when objects are detected are merged with one another and checked for plausibility.

Description

The present invention relates to a method for object detection by means of sensor systems of vehicles, several vehicles being networked with one another and each of the vehicles having a sensor system for detecting objects. Furthermore, the invention relates to a computer program for performing the method, a machine-readable storage medium on which the computer program is stored, and an electronic control device for performing the method.

State of the art

The term object recognition generally describes a method which can recognize an object by means of optical, acoustic or other physical methods. In particular, the detection of objects by means of camera systems is already widely used. Especially in the automotive sector, various camera-based systems with functions for the detection of objects are used. For example, traffic signs or the lane can be recognized automatically. The detection of pedestrians is of particular importance. Object detection can be used for a wide range of functions and is required in particular for various assistance functions up to automated driving.

In general, object recognition methods are based on a specific characteristic of an object type, for example a person, which is evaluated on the basis of a training data record. The training data record contains different forms of an individual object type, with characteristic features being generated in a training step from several training images in a development and / or application phase of the method. An overall description of the object type can be created from a collection of such characteristic features. To recognize a certain type of object (such as a vehicle, motorcycle or pedestrian), a large number of image examples are required for the respective type, which represent different forms of the respective object under different environmental conditions. Furthermore, a large number of counterexamples are required so that characteristic, image-based properties of the respective object can be determined via machine learning.

In order to be able to clearly assign a specific feature vector to an object type in operation, it is generally necessary to train a classifier. Characteristic numbers that describe the respective object are used for the actual object recognition. Using the classifier, the feature numbers can be converted into a number using many nested and automatically generated weights, which indicates the probability of the object to be classified in the current search window. The trained object examples usually result in a rigid structure of preprocessing and object detection based on fixed filter masks and weights.

The German publication DE 10 2016 003 969 A1 describes a method for recording properties of an environment, wherein a plurality of motor vehicles acquire observation data of the environment via respective sensor devices. A stationary server device sends query messages to the motor vehicles as a function of a minimum requirement criterion, whereupon the motor vehicles send those acquired observation data to the server device that meet the minimum requirement criterion.

Disclosure of the invention

Advantages of the invention

The invention provides a method for object detection by means of sensor systems of vehicles, the method starting from a plurality of vehicles that are networked with one another. Each of the vehicles has at least one sensor system for the detection of objects. The essence of the invention is that the data and / or information captured by the individual vehicles in the detection of objects are fused together and checked for plausibility. The information that is acquired and / or ascertained on the basis of the detection of the objects is, in particular, an object recognition and / or an object classification in the sense described at the beginning. The object recognition itself and / or the object classification can take place either in the vehicle itself or outside of the vehicles at a central point. The object detections, which are based on the data recorded in the individual vehicles, can thus be compared with one another and, if necessary, corrected. This method solves the problem that object recognition and in particular object classification are trained differently and can therefore lead to different results. For example, an object recognition can be designed in relation to a person recognition in such a way that all people are primarily recognized, even if the recognition is uncertain. However, this has the disadvantage that, for example, ghost objects or ghost persons that are not actually present are also recognized. These false positive results lead to unnecessary braking, for example in the case of assistance functions or automated driving, which in particular can lead to avoidable accidents on expressways. A somewhat opposite interpretation of object detection can be that people are primarily recognized only if the detection is very reliable. This has the disadvantage that, for example, people in poor lighting conditions or poses or under unusual circumstances are not recognized. In the worst case, such a false negative result means that automatic braking systems are not activated and people can be injured.

The term “object detection” here means that sensor data that a vehicle receives via its sensor system are processed and evaluated with a view to detecting objects. This object recognition can include, for example, an extraction of characteristic data, which are evaluated, for example, using a classifier. The object detection can also include, however, that data from the sensor system of the vehicle is only forwarded, for example to a cloud or a local provider, in order to be further evaluated elsewhere in the sense of the actual object detection.

The method according to the invention solves the problem of false positive or false negative results of an object recognition in that the individual object recognizations or individually recorded data are compared to one another to a certain extent when objects are detected by individual vehicles, by the data and / or information for object recognition being merged and be checked for plausibility. The advantages of the method come into play in particular if a classification is provided for object recognition, which can be improved by the method according to the invention in such a way that fewer false positive and less false negative results are generated. The method according to the invention is used with particular advantage in connection with object detection in the case of moving or stationary objects, preferably in the case of object detection of moving objects such as people, animals or other vehicles. The use of the method for object recognition of people and possibly animals is particularly advantageous, the possibilities of the method according to the invention not being limited to such object recognition.

A semantic map is used with particular advantage for the method according to the invention, the plausibility-checked data and / or information, in particular the objects recognized on the basis of the object recognition, being stored in the semantic map.
In the context of the present invention, a semantic map can be understood to mean a metric map which, in addition to geometric and / or topographical information relating to 2D measuring points and / or 3D measuring points, additionally contains semantic names of the respective measuring points. In other words, a semantic map can be understood to mean a metric or geometric and / or topographic map that has additional information in the form of names or inscriptions of points on the map. The vehicles themselves are preferably also located in the semantic map. Thus, a certain object at a certain location in the environment and thus also in the semantic map can be classified by the sensor systems of different vehicles and thus also from different perspectives with possibly different lighting conditions. The recognition and / or classification accuracy of the objects is increased by a fusion and plausibility check of the data and / or information on the same local objects. The vehicles themselves can be located in the semantic map using methods known per se, for example by cross-bearing known landmarks or template matching or scan matching of current sensor measurements, for example by ultrasound or radar or lidar. The objects can be localized, for example, using stereo cameras on the vehicles, the distance of the objects being determined by means of the disparity, so that the position in the semantic map can be determined in relation to the location of the vehicles. In this case, it can be expedient if additional correction services are used in order to improve the positional accuracy of the localized objects.

It is further preferred that the semantic map is georeferenced. Methods known per se can also be used for this, for example using GNSS (global navigation satellite system) or RTK (Real Time Kinematics). The map creation itself can be created in particular by means of the environmental sensors of the vehicles or possibly the infrastructure by creating a three-dimensional environment model or a three-dimensional environment map. Here, by means of machine vision or pattern recognition and the like, for example, stationary (e.g. buildings, streets / paths, construction pits, light poles, traffic signs, trees, building materials etc.) and non-stationary (e.g. stationary machines, standing vehicles, standing people etc. ) or moving objects (running people, moving vehicles, working machines etc.) can be recognized and classified. These recognized objects are recorded in a semantic map, which may also be georeferenced. The Vehicles themselves can locate themselves or assist themselves in this semantic map and, if necessary, also navigate. As an alternative to creating an environmental model or an environmental map on the basis of sensor data of the vehicle, a GNSS-based map can alternatively or optionally also be used.

The prerequisite for the described method is that the individual vehicles are directly (vehicle to vehicle - V2V) and / or networked via a traffic infrastructure (V2X). It can optionally be provided that other road users, such as two-wheelers (motorcycles, bicycles or other) and possibly pedestrians with corresponding mobile devices, are also networked with the vehicles. It is important that the vehicles have sensor systems, whereby the object recognition and, if necessary, the classification are based on the data generated thereby. Camera systems, in particular mono cameras, or, in a particularly preferred manner, stereo camera systems, and corresponding algorithms for image processing and, if appropriate, machine vision, can be provided as sensor systems. Such camera systems are already widely used for object detection, particularly in the automotive sector. For example, the object can be recognized on the basis of data from a front camera of a motor vehicle. Alternatively or additionally, the object can also be detected, for example, by using radar, lidar or ultrasound. In addition, it is possible for the vehicles to have systems which enable an interpretation of the behavior of objects and / or a prediction of the behavior of objects. The systems can, for example, be designed so that gestures and intentions of people can be interpreted and their behavior can be predicted. This can also be done on the basis of camera systems, image processing algorithms and machine vision.

The data and / or information for object recognition can be merged in particular by a spatial fusion of extracted features of object recognition. Then, in particular, a trained classifier can then be carried out with these merged features, so that the result of the classifier represents a plausibility-checked result of the object recognition. The data and / or information can be merged and processed further locally in the individual vehicles or at a central location.

In a preferred embodiment of the method, the plausibility check of the data and / or information takes place using statistical methods. If, for example, two or more vehicles have recognized an object and classified it as a person, whereas a third vehicle at the corresponding location on the semantic map has not recognized any person at this time, the third vehicle can correct its object recognition accordingly and react if necessary. Conversely, if, for example, only one vehicle recognized a person and the other two did not, it is most likely not a person. The plausibility check of the object detection can take place to a certain extent on the basis of mean values taking into account the object detection by several vehicles.
In other words, the plausibility check of the data and / or information recorded by the individual vehicles in the detection of objects is carried out by comparing the data and / or information recorded by different vehicles in order to obtain consistent data and / or information and of the consistent data and / or to determine information deviating data and / or information.

If the object recognition takes place on the basis of a classifier, it can advantageously be provided that the classifier is retrained on the basis of the fusion and plausibility check of the various object recognizations or data of individual vehicles. As a result, online learning or online retraining of the classifier takes place to a certain extent.

The data and / or information for object recognition can be merged and / or checked for plausibility in various ways. For example, this can be done in a cloud or in a base station of a provider. Furthermore, it is possible for the data and / or information to be merged and / or checked for plausibility in the individual vehicles, a relatively large computing effort being required for this in the individual vehicles.

If a classifier (object classifier) is retrained in the manner described, this retrained classifier can be made available to all vehicles, for example via a cloud or the service provider, if a classifier is provided in each of the vehicles. Alternatively, for example, the image information with the label “This is a person” can be sent from the object recognition of the individual vehicle, for example, to the cloud or to the service provider, the classifier being trained, for example, in the cloud or at the service provider. In this way, the classifier can be continuously improved and more and more automatically labeled image data can be made available.

The invention further comprises a computer program which is set up to carry out the steps of the described method. Furthermore, the invention comprises a machine-readable storage medium on which such a computer program is stored. Finally, the invention comprises an electronic control device which is set up to receive data and / or information from a plurality of vehicles which are networked with one another, the data and / or information being detected when objects are detected by means of sensor systems of the vehicles; and to merge the received data and / or information with one another and to plausibility check in order to provide the merged and plausibility check data and / or information to the vehicles. This can be, in particular, a motor vehicle control unit that is used, for example, in the context of driver assistance systems or in the context of autonomous driving.

Further features and advantages of the invention result from the following description of exemplary embodiments in conjunction with the drawing. The individual features can be implemented individually or in combination with one another.

• 1 Skizze einer Verkehrssituation zur Illustrierung einer Ausgestaltung des erfindungsgemäßen Verfahrens.

The drawing shows:

• 1 Sketch of a traffic situation to illustrate an embodiment of the method according to the invention.

Description of exemplary embodiments

1 illustrates a traffic situation in which the method according to the invention is used. A street is shown 10th on which a first vehicle 11 and a second vehicle 12 are traveling in the opposite direction, with the vehicles moving towards each other. In the road section between the vehicles 11 and 12 crosses a person 13 the street. The vehicles 11 and 12 are each equipped with a sensor system that is set up for object detection. For example, front cameras can be provided as sensor systems, the triangular fields 111 and 112 the field of vision of the front cameras of the vehicles 11 respectively 12 represent. Alternatively or optionally in addition, for example radar-based or ultrasound-based sensor systems can be provided, the rectangular fields 1111 and 1112 the field of view of a radar or ultrasound for the vehicle 11 respectively 12 represent. The second vehicle 12 the person can under the assumed environmental and lighting conditions 13 recognize using the front camera or another sensor system and classify it as a person. Thereupon, for the vehicle 12 a corresponding entry can be made in a semantic map. In this example adopted is for the first vehicle 11 the view is hampered, for example, by rising clouds of fog, so that no corresponding feature extraction is possible and no clear object detection can be carried out. In this case, driver assistance systems, such as emergency braking, would not be triggered without further measures. To solve this problem, the method according to the invention provides that the vehicle 11 by accessing the semantic map into which by vehicle 12 a corresponding entry has been made, which can access information about the existing person. So vehicle has 11 a sure knowledge of the presence of the crossing person 13 , so that appropriate driver assistance systems can react and, for example, emergency braking can be initiated.

Other situations that can lead to the object detection of a vehicle not being able to be carried out correctly can occur, for example, due to a low position of the sun, so that a front camera is faced with backlighting, which makes feature extraction difficult and can lead to a false negative result in the case of object detection . An unfavorable position of the sun can, for example, also cause a shadow to be cast, which is incorrectly recognized as a person when an object is recognized (false positive).

It is particularly advantageous if the object identifications, which are based on the data of the individual vehicles, are continuously entered into the semantic map, so that corresponding information can be continuously merged and checked for plausibility. If, for example, several vehicles have already classified pedestrians as they drive past, the respective people have been recognized from different perspectives, under different lighting conditions and in possibly different poses, etc. According to this embodiment of the method, all of this information is permanently updated and localized in the semantic map and is merged and checked for plausibility in this map. Other vehicles can access this information so that object recognition with incorrect results can be largely avoided or corrected. The fusion or plausibility check of the object recognition in the semantic map can, for example, take place in such a way that all vehicles involved send their object information and their location to the cloud. This is where the merger or plausibility check takes place, the result of which all the vehicles involved can access immediately. Alternatively, the Object information and its location, which are recorded by the individual vehicles, are sent to the closest base station of the provider. This procedure is comparable to the cloud solution, although generally lower latency times can be expected than with the cloud solution. As a further possibility, the same semantic map can be provided in each vehicle, which is loaded by the respective vehicle in advance from a cloud, for example, when driving into a certain global area. Here, all object information and their locations are exchanged between the individual vehicles. With this solution, the fusion or plausibility check of object detection can take place in every vehicle itself. This has the advantage that it enables significantly lower latency times than the other variants described.

Another advantage of the continuous entry in the semantic map is that it also makes object tracking easier, so that, for example, a person cannot be lost during tracking. This can also be used to check the plausibility of a recognized or not recognized or classified object.

QUOTES INCLUDE IN THE DESCRIPTION

This list of documents listed by the applicant has been generated automatically and is only included for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• DE 102016003969 A1 [0005]

Claims (14)

Method for object detection using sensor systems of vehicles (11, 12), a plurality of vehicles (11, 12) being networked with one another and each of the vehicles (11, 12) using a sensor system (111, 112; 1111, 1112) for detection of objects (13), characterized in that the data and / or information acquired by the individual vehicles (11, 12) for the detection of objects are merged with one another and checked for plausibility. Procedure according to Claim 1 , characterized in that the information recorded during the detection of objects (13) comprises object detection and / or object classification. Procedure according to Claim 1 or Claim 2 , characterized in that the objects (13) are moving objects. Method according to one of the preceding claims, characterized in that the plausibility-checked data and / or information are stored in a semantic map. Procedure according to Claim 4 , characterized in that the vehicles (11, 12) themselves are localized in the semantic map. Procedure according to Claim 4 or Claim 5 , characterized in that the semantic map is georeferenced. Method according to one of the preceding claims, characterized in that algorithms for image processing and / or machine vision and / or interpretation of behavior of objects and / or prediction of behavior of objects are provided in the detection of objects (13). Method according to one of the preceding claims, characterized in that the fusion of data and / or information takes place by spatial fusion of extracted features in the case of object recognition and, if appropriate, execution of a trained classifier. Method according to one of the preceding claims, characterized in that the plausibility check of data and / or information is carried out using statistical methods. Method according to one of the preceding claims, characterized in that a classifier is carried out for object recognition, the classifier being retrained. Method according to one of the preceding claims, characterized in that - the fusion and / or plausibility check of the data and / or information for object recognition takes place in a cloud or - that the fusion and / or plausibility check for the data and / or information for object recognition takes place in a base station a provider or - that the merger and / or plausibility check of the data and / or information for object detection takes place in the individual vehicles. Computer program which is set up the steps of a method according to one of the Claims 1 to 11 perform. Machine-readable storage medium on which a computer program Claim 12 is saved. Electronic control device that is set up to - to receive data and / or information from a plurality of vehicles (11, 12) networked with one another, the data and / or information when detecting objects (13) by means of sensor systems (111, 112; 1111, 1112) of the vehicles ( 11, 12) are detected; and - merge the received data and / or information with one another and plausibility check in order to provide the merged and plausibility check data and / or information to the vehicles (11, 12).

Images