DE102018116036A1 - Training a deep convolutional neural network for individual routes - Google Patents

Abstract

The present invention relates to a method for training a deep convolutional neural network for processing image data for use in a driving support system of a vehicle using training data which contain route information, with the steps of providing an initially trained deep convolutional neural network with general model data a set of annotated training data containing route information, setting up the deep convolutional neural network with the individual model data assigned to the route based on the general model data, performing a training step of the deep convolutional neural network with the individual model data assigned to the route using the respective set of annotated training data and Storage of the individual model data of the deep convolutional neural network assigned to the route. The present invention also relates to a method for using a deep convolutional neural network for processing image data in a driving support system of a vehicle based on route information of a desired route, with the steps for identifying the route, setting up the deep convolutional neural network with the individual model data assigned to the route and processing the image data acquired while driving along the route using the deep convolutional neural network with individual model data associated with the route. The present invention further relates to a driving support system for performing one of the above methods.

Description

The present invention relates to a method for training a deep convolutional neural network for processing image data for use in a driving support system of a vehicle using training data containing route information, with the steps of providing an initially trained deep convolutional neural network with general model data and providing it a set of annotated training data containing route information.

The present invention also relates to a method for using a deep convolutional neural network for processing image data in a driving support system of a vehicle based on route information of a desired route.

The present invention further relates to a driving support system for performing any of the above methods.

Autonomous and semi-autonomous driving are becoming increasingly important in the automotive industry. Prototypes for autonomous driving are already being developed and tested, sometimes even under real driving situations. Autonomous driving is seen as a breakthrough technology in the automotive sector.

Autonomous and semi-autonomous driving is based on knowledge of the surroundings of the vehicle. Based on this knowledge, the driving support system can e.g. Identify dangers to the vehicle or third parties and act in a suitable way to defuse dangerous driving situations. For this purpose, various types of environmental sensors are installed in vehicles to monitor the surroundings of the vehicle. Such environmental sensors can have any combination of sensors from ultrasonic sensors, LIDAR-based sensors, radar sensors or optical cameras.

Processing image data from an optical camera, and particularly video data that contains a sequence of many images per second, is a challenging task.

A huge amount of data has to be processed in real time in order to reliably record the surroundings of the vehicle. However, the data processing resources of the vehicle are limited in terms of the space for accommodating the processing units and also in terms of the available electrical power. Even if these technical problems were solved, the price for the resources would remain limited in order to be able to offer vehicles at a reasonable price.

Neural networks are a very effective means of processing image data. Applications of neural networks for image processing are typically based on convolutional neural networks (CNNs) and in particular on deep convolutional neural networks (DCNNs). The use of such network types has shown promising results at an acceptable price.

When using deep convolutional neural networks, some questions arise in the development of suitable structures of neural networks, especially in the development from scratch. These included, to name a few, a definition of a suitable neural network type, the determination of a number and a position of inputs, the determination of a number of hidden layers to be used in a deep convolutional neural network, and that Determine a number of required output neurons. These questions are important because if the neural network is too large / too small, the neural network could possibly over- or under-determine the image data. As a result, the neural network would not learn appropriately to generalize appropriately from the training data provided. In such cases, even if the error in the training data is suppressed to a very small value, the error may be too large when new image data, i.e. different image data from the training data previously used for training the neural network are presented. In this case, the neural network has saved the training data, but has not learned to generalize from the training data provided, which would be necessary for such new situations with new image data. Because of this, when using such neural networks, suitable training with suitable training data that can be applied to different neural networks is extremely important in order to evaluate and compare the performance of the different neural networks.

Many companies are constantly developing their own processes and systems to enable autonomous and semi-autonomous driving applications, such as an autopilot function that is to be used in upcoming hardware generations in their vehicles. However, the use of neural networks in autonomous or semi-autonomous driving applications depends on the provision of one trained neural network that can be used in the vehicles. Therefore, the availability of suitable training data is an essential basis for autonomous and semi-autonomous driving applications. For this reason, companies that are involved in the development of such autonomous and semi-autonomous driving applications are constantly collecting image data with their vehicle fleet that are on the road under various conditions, for example environmental conditions, traffic conditions and the like. Unfortunately, in order to use the image data as training data, annotations are typically required that are created by a human. Annotating the image data is therefore very time-consuming and expensive.

In addition, data collection plays a key role in building neural networks. If the training data is too extensive, the neural network could have problems learning to a desired level of accuracy. Therefore, the designer wants e.g. identify the most significant elements of the training data from the entire training data set. This significant training data can relate to images that contain a large number of features to be recorded. Accordingly, only these most significant elements of the training data are used for training the neural network, and insignificant training data are removed from the training data record.

Driving assistance is most desirable for the most monotonous routes. This applies in particular to routes that are driven frequently, sometimes even daily. Such frequent commuting routes include, for example, a route from home to an office or from home to office, a route from home to a supermarket, a route from home to a park, etc. Since these are the most frequently traveled routes, they occur most accidents on these routes, and there is an increased need to drive the frequent commuting routes safely. In addition, these frequent commuting routes may bore a driver or owner of the vehicle, so autonomous driving functions, particularly an autopilot function, are highly desirable for driving the vehicle.

It is therefore extremely important to enable these commuting routes to be driven autonomously and safely.

In this context, the document concerns EP 2213980 B1 a method for operating a navigation device of a motor vehicle. The navigation device determines the position and / or the travel route of the motor vehicle on the basis of geographic basic data available on the motor vehicle side. At least one geographical item of information about a region of the motor vehicle environment is determined from the sensor data of at least one vehicle-side sensor device. The at least one geographic information is stored in a storage device as an additional geographic data element and used to supplement and / or update the geographic basic data.

The document DE 10030932 A1 relates to a method in which a vehicle with a position determination system records its position via a route and stores it in a memory. The measured route is transferred to a database, and a computer processes the data, ideally with other vehicle data, to check the data and analyze it statistically in order to obtain a revised road map. The latter can then be transmitted to users via telecommunications.

According to the DE 10258470 B4 A navigation system for a motor vehicle has a database with a digital map and a position detection unit. The vehicle also has equipment for sensing vehicle operating parameters and / or its surroundings. These parameters can be used to derive information about the type of road and / or its direction. This information is then compared to available data derived from the digital map data and can be used to update the database if necessary. The navigation database of the motor vehicle is updated using a learning method in which acquired vehicle data are used to update existing database data.

The document DE 10354910 B4 relates to a self-controlling driving system for an expressway and a method for its control. A large number of traffic signs are installed one after the other on the left and on the right side of the expressway. An imaging device depicts the traffic signs arranged along the expressway. An image processor analyzes at least one image, distinguishes the street signs and reads the contents of the signs contained in the street signs. A navigation database stores map information of the expressway. An electronic control unit based on artificial intelligence determines a current position and a road condition on the basis of the map information stored in the navigation database and the character contents read by the image processor. An integrated ECU controls a steering operation, a speed reduction / increase operation and other operations associated with the self-driving operation related, according to a result of the determination.

The document DE 19916967 C1 relates to a method for updating a data processing-supported traffic route network map which contains route data about a traffic route network and associated attribute data. Using at least one test vehicle, current route and attribute data from the respective passable section of the route are recorded, and the traffic route network map is updated on the basis of this recorded data.

The document DE 10 2013 211696 A1 relates to a method for completing and / or updating a digital road map, comprising the steps of: a) capturing a digital image of a route section using a camera of a motor vehicle located on the route section; b) determining a current position of the motor vehicle in the width direction of the track section and / or at least one property of the track section based on the image by means of a computing device of the motor vehicle; and c) transmitting to a server a data record which contains information about a geographical position of the image and information about the at least one specific property of the route section and / or the position in the width direction. Steps a) to c) are carried out by several motor vehicles in the area, and the digital road map is completed and / or updated on the basis of the received data records by means of the server.

The document DE 10 2015 010542 A1 relates to a method for generating a digital environment map for a vehicle, an environment detection being carried out by means of environment sensors of the vehicle and / or at least one further vehicle. Based on acquired sensor data, map information is generated which is stored in the area map. The environment map is updated automatically, objects and / or obstacles arranged in the environment being detected by means of at least one environment sensor of the vehicle and / or the at least one further vehicle. During the detection of the objects and / or obstacles, a route actually driven by the vehicle and / or the at least one further vehicle is recorded. Depending on a comparison of positions of the route actually traveled with positions of a route in the area map, updated map information is generated and stored in the area map.

The present invention has for its object to provide a method for training a deep convolutional neural network for processing image data for use in a driving support system of a vehicle using training data which contain route information, a method for applying a deep convolutional neural network for processing image data in a driving support system of a vehicle based on route information of a desired route and a driving support system for executing any of the above methods of the above-mentioned type, which enable efficient training of a deep convolutional neural network and improve the performance using the deep convolutional neural network. Another task is to enable efficient training of a DCNN installed in a vehicle or in a driving support system.

This task is solved by the independent claims. Advantageous refinements are specified in the subclaims.

In particular, the present invention provides a method for training a deep convolutional neural network for processing image data for use in a driving support system of a vehicle using training data containing route information, with the steps for providing an initially trained deep convolutional network with general model data Providing a set of annotated training data containing route information, setting up the deep convolutional neural network with the individual model data assigned to the route based on the general model data, performing a training step of the deep convolutional neural network with the individual model data assigned to the route using the respective set of annotated Training data, and storing the individual model data of the deep convolutional neural network assigned to the route ,

The present invention also provides a method for using a deep convolutional neural network for processing image data in a driving support system of a vehicle based on route information of a desired route, with the steps for identifying the route and establishing the deep convolutional neural network associated with the route individual model data and processing the image data acquired while driving along the route using the deep convolutional neural network with the individual model data associated with the route.

The present invention further provides a driving support system for performing any of the above methods.

The basic idea of the invention is to provide a simple but efficient improvement for deep convolutional neural networks (DCNN), in particular for everyday driving situations which relate to repeated use of the same routes and are also referred to as commuting routes. Such commuting routes include, for example, a route from home to an office or from office to home, a route from home to a supermarket, a route from home to a park, etc. An individual training of the deep convolutional neural network is therefore carried out for these routes, so that the training step provides corresponding model data for the deep convolutional neural network that are highly suitable for driving on these routes. With the special training of the DCNN for these routes, which cover a large part of the driving of the vehicle, accidents on these routes can be better avoided. This is all the more important because driving along these routes is done by people, e.g. by the owner or driver of the vehicle is considered boring. The attention of the driver or owner of the vehicle often decreases when he is bored, so that autonomous functions, in particular an autopilot function for autonomous driving, are also highly desirable in order to increase traffic safety.

The neural network that is trained with the proposed method is a deep convolutional neural network (DCNN) for processing image data. DCNNs are a special implementation of more general convolutional neural networks (CNN). CNNs have an input and an output level as well as several hidden layers. The hidden layers of a CNN typically consist of convolutional layers, pooling layers, fully connected layers and standardization layers. The convolutional layers apply a convolution operation to the input and forward the result to the next layer. The convolution emulates a single neuron's response to visual stimuli.

Such DCNNs are increasingly used in connection with driving applications in vehicles. Therefore, the DCNNs in particular e.g. used in vehicle driving support systems. The driving support systems include driver assistance systems that assist a human driver of the vehicle in various driving situations, or systems that support autonomous driving of the vehicle by providing inputs to reliably handle different traffic situations.

The step of providing an initially trained deep convolutional neural network with general model data relates to providing a pre-trained DCNN. The DCNN can be trained before it is transmitted to the vehicle with standard training data. These standard training data are independent of a route traveled by a driver or owner of the vehicle. The pre-trained DCNN is ready for use in the driver support system.

The step of providing a set of annotated training data containing route information relates to providing the training data with labels with respect to the features shown. The labels can be added manually, e.g. by a human person, or automatically using e.g. an already trained DCNN or other technology. The route information relates to some kind of identification of the route. The identification can be added manually, e.g. by the owner or driver of the vehicle, or automatically, e.g. based on location data.

The step of setting up the deep convolutional neural network with the individual model data assigned to the route based on the general model data relates to providing the DCNN for training the DCNN for a specific route.

Performing the training step of the deep convolutional neural network with the individual model data assigned to the route using the respective set of annotated training data refers to training the deep convolutional neural network that is set up with the individual model data using the respective set of annotated training data that refers to this route. Therefore, special training can be carried out that covers possible features along this route, thereby providing a highly efficient DCNN when driving this route.

The step of storing the deep convolutional neural network individual model data associated with the route refers to storing the individual model data for later use when the vehicle is used to drive the particular route. Verification is preferably performed to verify whether an error in the training data using the individual model data is lower than the error in previous individual model data.

The method of applying a deep convolutional neural network to process image data in a driving support system of a vehicle based on route information of a desired route relates to using the vehicle with the driving support system using the DCNN.

The step of identifying the route can be carried out in advance, e.g. when route planning is carried out using a navigation system of the vehicle. Alternatively, the route can be e.g. based on position data, in particular satellite position data, based on a recognition of orientation points or based on any other recognition of the route.

The step for setting up the deep convolutional neural network with the individual model data assigned to the route relates to the activation of the DCNN with the individual model data of the respective route.

The step of processing the image data that is acquired when driving on the route using the deep convolutional neural network with the individual model data assigned to the route relates to an application of the DCNN in the driving support system.

According to a modified embodiment of the invention, the step of providing a set of annotated training data containing route information, identifying a driving route, providing image data as training data from at least one camera when driving the route, and automatically annotating the image data to provide the set of annotated Training data on. The driving route can be based on user input or automatically based e.g. be identified based on location information of at least one starting point and one end point of the route. The route can be identified before or after driving on the route. The image data can be provided when driving the route with a specific vehicle, e.g. with the vehicle in which the DCNN is implemented. However, the image data can also be provided if the route is traveled by other vehicles, even without the DCNN being implemented in the specific vehicle. In addition, the step of automatically annotating the image data to provide the set of annotated training data enables the set of annotated training data to be generated without human interaction. Therefore, the generation of valid training data can be improved and the time and money typically spent on manual annotation of training data can be reduced. The step of automatically annotating the image data to provide the set of annotated training data can be performed at any location, e.g. using the vehicle's driving support system. Therefore, the entire step of providing the set of annotated training data containing route information can be performed by and within the vehicle. No data transmission or data transmission between the driving support system and an external server is required.

According to a modified embodiment of the invention, the step of providing image data from at least one camera when driving the route comprises labeling the provided image data with position information, in particular with satellite-based position information. In addition to identifying the route that the vehicle travels when the image data is generated, the position information can also be e.g. be used for the purpose of compiling certain routes based on combinations of route sections that have been traveled previously to collect training data and to provide the set of annotated training data. Therefore, it is not even necessary to follow a particular route to provide a set of annotated training data that includes the route. With a sufficiently large database of image data, a set of annotated training data can be provided for essentially every route. The set of annotated training data can be made available as soon as the vehicle is used for the first time. In addition, the position information can be used to process the image data and to determine different conditions for providing the image data from the same location. Therefore, image data provided in similar locations can be identified and filtered to select image data with different properties, e.g. with different lighting conditions, different weather conditions, different traffic conditions or other properties. This will reduce data redundancy in the set of annotated training data and the training step of the DCNN can be carried out in an efficient manner. The image data can be labeled with position information based on location information, e.g. is provided by GPS, Galileo, GLONASS or Beidou satellite positioning systems.

According to a modified embodiment of the invention, the step of automatically annotating the image data automatically includes the steps of processing a set of non-annotated training data through the deep convolutional neural network to generate a set annotated data, calculating confidence metrics for the set of automatically labeled data and automatically relabeling the set of automatically labeled data based on the calculated confidence metrics to generate the set of annotated training data.

The proposed use of the DCNN for the automatic annotation of the image data is simple and time-efficient since the use of the initially trained DCNN for processing the image data to generate the set of automatically labeled data already uses the DCNN. The confidence metrics can also be calculated automatically and in an efficient manner. The confidence metrics for the set of automatically labeled data enable a simple assessment of the relevance of the labels generated by the DCNN. If the confidence metrics indicate a high degree of confidence in the automatically labeled data generated by the DCNN, the automatic relabeling step does not change the labels. Otherwise, it is necessary to change the labeling and automatically relabel the set of automatically labeled data. This can refer to modifying all labels or just modifying some labels associated with the set of automatically labeled data. Relabeling is not as efficient as processing the image data using the DCNN. The effort of relabeling, i.e. however, the use of the less efficient labeling process is only required in cases where the DCNN cannot process the set of annotated training data in a sufficiently reliable manner, as indicated by the confidence metrics. The deep convolutional neural network may be the initially trained deep convolutional neural network with the general model data, or the deep convolutional neural network with the individual model data associated with the route that the image data corresponds to.

According to a modified embodiment of the invention, the step of computing confidence metrics for the set of automatically labeled data comprises applying a confidence metric algorithm based on Computer Vision. Computer vision is generally powerful and has already reached a high quality level for recognizing features. Various methods for computing confidence metrics are known in this area that are used to calculate confidence metrics for the set of automatically labeled data.

According to a modified embodiment of the invention, the step of automatically relabeling the set of automatically labeled data based on the calculated confidence metrics comprises applying a labeling algorithm based on Computer Vision. Computer vision is generally powerful and has already reached a high quality level for recognizing features. In the event that the confidence of the automatic labeling process by the DCNN is not sufficient, Computer Vision can be used to add the necessary labels to the respective image data.

According to a modified embodiment of the invention, the step for setting up the convolutional neural network with the individual model data assigned to the route comprises generating the individual model data assigned to the route as a copy of the general model data or loading the previously stored individual model data of the deep convolutional data assigned to the route neural network. When the procedure is run for the first time, no individual model data is available. Since the DCNN has received suitable training to provide the general model data, this general model can be used as a starting point for training the DCNN for the particular route. As soon as such training has been carried out for a specific route, individual model data of the deep convolutional neural network assigned to the route have been generated and stored so that they are available. This individual model data can thus be used as the basis for further training of the DCNN for the specific route.

According to a modified embodiment of the invention, carrying out a training step of the deep convolutional neural network with the individual model data assigned to the route using the respective set of annotated training data comprises training last layers of the deep convolutional neural network, in particular only the last layer. Training only the last layers, especially the last layer of the DCNN, reduces the number of parameters to be saved for the individual model data. With the given general model data of the DCNN, which have been generated based on a training, it can be assumed that the DCNN is functioning correctly at least in general. In view of the specific routes, however, a further improvement can be achieved for these specific routes without modification of the upper layers of the DCNN, ie with only minor adjustments to the DCNN based on the general model data. Therefore, the training step can be carried out even in systems with processing restrictions, either in terms of processing power or storage capacity. The step furthermore preferably has for storing the individual model data of the deep convolutional assigned to the route neural network only storing the last layers of the deep convolutional neural network, in particular only the last layer. The remaining model data can be found in the general model data provided for the DCNN. Therefore, little storage is required to store the individual model data for each route, which can be important in systems with memory restrictions. Therefore, when setting up the convolutional neural network with the individual model data assigned to the route, it is only necessary to generate, for example, the individual model data or to load the individual model data of the DCNN only for its last layers, in particular only for the last layer.

For systems without such limitations, it may be preferable to perform the training step for all layers of the DCNN to further improve the overall performance of the DCNN.

According to a modified embodiment of the invention, the method is implemented as an in-vehicle method in which the training step of the deep convolutional neural network with the individual model data assigned to the route using the respective set of annotated training data for training the deep convolutional neural network intended for a specific vehicle is performed. Therefore, the vehicle, in particular its driving support system, e.g. be adapted to an individual owner or driver of the vehicle and to certain routes that are repeatedly driven by the vehicle. Therefore, the DCNN is optimally trained for the needs of the owner or driver of the vehicle in order to provide the highest possible level of safety and comfort for every owner or driver of the vehicle. The set of annotated training data can be provided externally, e.g. when the owner or driver of the vehicle defines preferred routes and receives the set of annotated training data therefor that is supplied by an external provider. This enables immediate training of the DCNN. In this case, the set of annotated training data can be provided by a storage device connected to the vehicle or via a data connection between a server and the vehicle. The data connection is a wired or wireless data connection. The server can e.g. provided as a cloud server over the Internet.

Performing the training step is time consuming and may require the full processing power of the vehicle's driving support system. Therefore, when performing the training step, the vehicle may need to be driven manually so that the same hardware of the driving support system can be reused. No dedicated hardware is required to train the DCNN. To avoid such situations, it is preferred that the DCNN training step be performed when the vehicle is not moving to ensure full support of the driving support system.

According to a modified embodiment of the invention, the step of providing a set of annotated training data containing route information comprises providing the set of annotated training data containing route information using at least one camera of the vehicle. Therefore, the annotated training data is provided based on image data generated using the vehicle's camera for which the DCNN is trained. Therefore, the method can be carried out entirely inside the vehicle, i.e. through the vehicle's driving support system. Therefore, the method can be carried out as an in-vehicle method. This has the additional advantage that image data for training the DCNN are provided using the same hardware as when using the DCNN. This means that hardware influences on the performance of the DCNN can be excluded. A satellite receiver of the vehicle for receiving satellite position data of the vehicle can be used to identify the route and / or to label the image data.

According to a modified embodiment of the invention, the step of providing the set of annotated training data containing route information using at least one camera of the vehicle comprises accumulating training data and evaluating whether the set of annotated training data contains sufficient training data. Training data obtained by only traveling the route once typically does not provide a suitable set of annotated training data. It is therefore necessary to provide training data from multiple trips of the route in order to obtain a suitable set of annotated training data. The training data can, for example, be accumulated if the owner of the vehicle travels a route several times using his vehicle, ie if the vehicle is used under typical driving conditions. The method may include an additional step in which, upon successful evaluation of the set of annotated training data that contains sufficient training data, a driver or owner of the vehicle is prompted to perform the training step of the deep convolutional neural network with the accumulating training data kick off. The step in which, after successfully evaluating the set of annotated training data containing sufficient training data, the driver or owner of the vehicle is asked to start carrying out the training step of the deep convolutional neural network with the accumulating training data can be carried out independently for each route , Therefore, the training step can be carried out at different times, for example based on a frequency of driving a route.

According to a modified embodiment of the invention, the step of providing the set of annotated training data containing route information using at least one camera of the vehicle comprises filtering the training data based on a statistical analysis of the training data. It is helpful to feed the training data to the DCNN under various conditions in order to get an efficiently trained DCNN. Therefore the training data e.g. in the event of traffic jams, differently structured road surfaces, different lighting conditions or different traffic scenarios such as overtaking, changing lanes, roundabouts, parking areas or others. Filtering the training data based on the statistical analyzes therefore enables efficient training of the DCNN, since the different conditions ensure that the DCNN is trained appropriately. The statistical analyzes can e.g. using a histogram of individual features of the respective training data. The step of filtering the training data based on a statistical analysis of the training data is preferred.

According to a modified embodiment of the invention, the step for executing a training step of the deep convolutional neural network with the individual model data assigned to the route using the respective set of annotated training data comprises the steps for determining scene statistics for the set of annotated training data, classifying the set of annotated training data based on the scene statistics and automatically selecting training data from the set of annotated training data based on the classification. The scene statistics relate to a statistical determination of features that are contained in the training data. The features can be any type of relevant features that can be captured. The features can include the annotations of the training data, i.e. labels added to them. The characteristics can relate to static characteristics without movement information, or to dynamic characteristics that contain movement information, such as on a walking pedestrian, or more specifically on a pedestrian walking from left to right. The classification of the training data refers to a classification based on the occurrence of features within the training data. The classification can e.g. Identify training data that contain a particular type of feature and / or a certain number of features. For example, such classifications can include training data with a high feature density, training data with a special type of features, e.g. Pedestrians or vehicles, with or without dynamic information, e.g. those that do not move, that move, or that move from right to left. In addition, the training data can be classified and selected to separate them into training data for training, for a test and for validation of the DCNN.

According to a modified embodiment of the invention, the step of determining scene statistics for the set of annotated training data comprises determining label statistics of objects labeled on at least one type of label, a position of labels and / or properties, preferably determining statistics of label combinations , The labels are therefore evaluated in order to determine scene statistics. The label statistics of at least one information element from a type of label, a position of labels and / or properties of labeled objects relate to different aspects of the labels, all of which can be used to statistically identify common characteristics of different training data. The label statistics are preferably determined for several or all information elements from available types of labels, positions of the labels and / or properties of labeled objects. The determination of statistics of label combinations can be based on statistics of a combination of several labels e.g. in the same picture, e.g. refer to a certain number of pedestrians or a certain number of vehicles within the image. Additionally or alternatively, the statistics of label combinations can refer to a combination of several information elements under the type of labels, the position of the labels and / or the properties of labeled objects.

According to a modified embodiment of the invention, the step of determining label statistics of a type of label comprises performing a histogram analysis of the type of labels from the set of annotated training data. Therefore, label statistics are provided as histograms for different types of labels. This makes it easy to identify similar training data using a histogram analysis. In addition, there can be one for each label type Histogram of the luminance and the color properties are provided in order to enable a further differentiation between label types. This makes it possible not only to identify simple label types, such as pedestrians or vehicles, but also to distinguish between complex label types, such as a pedestrian / vehicle in the shade, in sunlight or at night.

According to a modified embodiment of the invention, the step of determining label statistics of a position of labels comprises dividing images into blocks of equal size. The blocks are used as a basis for identifying the position of the respective label. Preferably the equally sized blocks are NxN pixels in size, where N is typically a large number, e.g. N = 64. Based on the blocks of the same size, the position of a label can e.g. be defined using a simple block number. Preferably, histogram analysis of labels that overlap each block is performed to obtain position statistics of each annotated label type.

According to a modified embodiment of the invention, the step for classifying the set of annotated training data based on the scene statistics comprises classifying the set of annotated training data with additional consideration of odometry information of the vehicle. The odometry information relates to the fact that the vehicle generates the underlying video images in order to provide the training data at the point in time at which the underlying video images are recorded. Therefore, additional information regarding the video images is provided which can help to classify the annotated training data efficiently. The odometry information relates to information regarding movement of the vehicle, e.g. the speed, the steering angle or the like. This information enables a further classification of the training data that goes beyond a classification that is carried out exclusively on the basis of the image data.

According to a modified embodiment of the invention, the step of classifying the set of annotated training data based on the scene statistics includes identifying hardware used when the set of annotated training data is obtained. The hardware particularly relates to the camera that is used to obtain the set of annotated training data and that can directly affect an appearance of the set of annotated training data. Therefore, a camera type and / or a camera resolution or other camera properties can be evaluated. In addition, the vehicle used can also influence the appearance of the set of annotated training data, for example in the case of the appearance of the vehicle in the images or a position of the camera on the vehicle.

According to a modified embodiment of the invention, the step for automatically selecting training data from the set of annotated training data comprises in each case equalizing training data from the set of annotated training data based on the classification between different classes. The balancing enables reliable training of the DCNN with all classes of training data. Therefore, similar performance of the DCNN can be achieved for all traffic situations, even taking into account changing conditions, e.g. Environmental conditions. Such a balancing of the training data helps to generalize the performance of the DCNN for a certain route for different traffic situations, including different lighting conditions. The various traffic situations can include traffic lights, sharp bends, roundabouts, high pedestrian traffic, a supermarket car park, to name but a few. Even if these scenarios are not necessarily very common or even very rare, it is important that DCNN also be trained to handle these scenarios reliably, which can occur when driving on a route. Preferably, the equalization of the training data comprises the removal of training data, which relate to common driving situations, e.g. driving on a freeway without many objects and with similar lighting conditions. Therefore, high performance of the DCNN can be achieved for different possible scenarios for the particular route to be traveled.

According to a modified embodiment of the invention, the method has a step for filtering out unsuitable training data from the set of annotated training data based on the scene statistics for the set of annotated training data and / or based on the classification of the set of annotated training data. Unsuitable training data refer to training data e.g. without labels or with an inappropriately small number of labels. Training the DCNN with such training data essentially does not help to effectively increase its performance. Therefore, omitting training with such training data improves training efficiency.

These and other aspects of the invention will be apparent and explained from the embodiments described below. Individual features disclosed in the embodiments may form an aspect of the present invention, alone or in combination. Characteristics of the various embodiments can be transferred from one embodiment to another embodiment.

• 1 ein Ablaufdiagramm eines Verfahrensschritts zum Bereitstellen eines anfangs trainierten tiefen konvolutionellen neuronalen Netzwerks mit allgemeinen Modelldaten gemäß einem Verfahren zum Trainieren eines tiefen konvolutionellen neuronalen Netzwerks zum Verarbeiten von Bilddaten zur Anwendung in einem Fahrunterstützungssystem eines Fahrzeugs unter Verwendung von Trainingsdaten, die Routeninformation enthalten, gemäß einer ersten bevorzugten Ausführungsform;
• 2 ein Ablaufdiagramm eines Verfahrensschritts zum Bereitstellen eines Satzes annotierter Trainingsdaten, die Routeninformation enthalten, gemäß dem Verfahren zum Trainieren eines tiefen konvolutionellen neuronalen Netzwerks zum Verarbeiten von Bilddaten zur Anwendung in einem Fahrunterstützungssystem eines Fahrzeugs unter Verwendung von Trainingsdaten, die Routeninformation enthalten, gemäß der ersten Ausführungsform;
• 3 ein Ablaufdiagramm eines individuellen Trainings des DCNN gemäß dem Verfahren zum Trainieren eines tiefen konvolutionellen neuronalen Netzwerks zum Verarbeiten von Bilddaten zur Anwendung in einem Fahrunterstützungssystem eines Fahrzeugs unter Verwendung von Trainingsdaten, die Routeninformation enthalten, gemäß der ersten Ausführungsform; und
• 4 ein Ablaufdiagramm eines Verfahrens zum Anwenden eines tiefen konvolutionellen neuronalen Netzwerks zum Verarbeiten von Bilddaten in einem Fahrunterstützungssystem eines Fahrzeugs basierend auf Routeninformation einer gewünschten Route gemäß einer zweiten Ausführungsform.

Show it:

• 1 a flowchart of a method step for providing an initially trained deep convolutional neural network with general model data according to a method for training a deep convolutional neural network for processing image data for use in a driving support system of a vehicle using training data containing route information, according to a first preferred embodiment;
• 2 a flowchart of a process step for providing a set of annotated training data containing route information according to the method for training a deep convolutional neural network for processing image data for use in a vehicle driving support system using training data containing route information according to the first embodiment;
• 3 a flowchart of an individual training of the DCNN according to the method for training a deep convolutional neural network for processing image data for use in a driving support system of a vehicle using training data containing route information according to the first embodiment; and
• 4 a flowchart of a method for applying a deep convolutional neural network for processing image data in a driving support system of a vehicle based on route information of a desired route according to a second embodiment.

The 1 to 3 together show a method for training a deep convolutional neural network for processing image data for use in a driving support system of a vehicle using training data containing route information, according to a first preferred embodiment. The method is carried out using the vehicle's driving support system.

The neural network trained according to the proposed method is a deep convolutional neural network (DCNN) for processing image data. The DCNN is a special implementation of a convolutional neural network (CNN) that has an input and an output layer together with several hidden layers that are arranged between the input layer and the output layer. The hidden layers of a CNN typically have convolutional layers, pooling layers, fully connected layers and normalization layers. The convolutional layers perform a convolution function with regard to the input and pass the result on to the next layer. The convolution emulates the response of an individual neuron to visual stimuli.

The DCNN is used in a driving application in a vehicle. For this purpose, the DCNN is used in a vehicle driving support system. The driving support system is a system that supports a human driver of the vehicle in various driving situations, or a system that supports autonomous driving of the vehicle by providing inputs for a reliable handling of different traffic situations.

The process starts with step S100 , step S100 refers to providing an initially trained deep convolutional neural network using an initial set of annotated training data. The annotated training data are manually annotated training data, ie a person has added labels to the provided image data.

step S100 includes three individual substeps in this embodiment S102 . S104 and S106 , as discussed in detail below.

step S102 relates to determining scene statistics for the initial set of annotated training data. The scene statistics relate to a statistical determination of features contained in the training data. The features include static features without movement information and dynamic features with movement information, such as a walking pedestrian or even a pedestrian who walks from left to right. The characteristics can contain the annotations of the training data, for example labels added to them.

The scene statistics are carried out taking into account label types, label positions and properties of labeled objects. The label statistics are evaluated with regard to a label type, a label position and properties of labeled objects. Statistics of a combination of labels are also determined, i.e. statistics of a combination of several labels e.g. in the same picture, for example a certain number of pedestrians or a certain number of vehicles within the picture.

Label statistics include performing a histogram analysis of the label type from the set of annotated training data regarding different label types. For each label type, this includes providing a histogram of the luminance and color properties.

In addition, to determine label statistics of a position of labels, the images are divided into blocks of equal size. The blocks are used as the basis for identifying the position of the corresponding labels. The blocks of the same size have a size of NxN pixels, where N = 64. Based on the blocks of the same size, the position of a label is e.g. defined as a simple block number. Histogram analysis of labels that overlap each of the blocks is performed to extract position statistics of each annotated label type.

step S104 Refers to classifying the set of annotated training data based on the scene statistics.

The classification of the annotated training data refers to a classification that is based on the occurrence of features within the training data based on the histogram. The classification enables the identification of training data that contain a certain type of feature and / or a certain number of features. Such classifications include training data with a high feature density, training data containing a certain type of feature, e.g. Pedestrians or vehicles, with or without dynamic information, such as not moving, in motion, moving from right to left.

step S106 refers to the automatic selection of training data from the set of annotated training data based on the classification. This includes balancing training data from the set of annotated training data based on the classification between different classes.

step S106 includes filtering out unsuitable training data from the set of annotated training data based on the scene statistics for the set of annotated training data and based on the classification of the set of annotated training data. Unsuitable training data refer to training data eg without labels or with an inappropriately small number of labels.

The training of the deep convolutional neural network is known from conventional DCNNs. The selected training data from the initial set of annotated training data of the DCNN is supplied to the DCNN to perform the training. step S106 stops when the DCNN shows sufficient performance.

In the next step S108 general DCNN model data are provided which are transmitted to and stored in the vehicle's driving support system.

The in 2 step shown S110 refers to providing a set of automatically annotated training data containing route information, as discussed in detail below. Therefore, the training data are provided with labels in relation to the features contained. step S110 has individual sub-steps S112 . S114 . S116 . S118 and S120 as discussed in detail below.

According to step S110 a camera of the vehicle is used to provide image data while driving along the route. The image data are labeled with satellite-based position information using a receiver for GPS signals. Furthermore, a driving route is identified based on the location information while driving along the route. step S110 is carried out using the vehicle's driving support system.

step S112 refers to the processing of a set of non-annotated training data by the initially trained deep convolutional neural network to produce a set of automatically labeled data. The stored model data of the initially trained DCNN are used to process the set of non-annotated training data and to generate a set of automatically labeled data.

step S114 Refers to computing confidence metrics for the set of automatically labeled data. The confidence metrics are calculated using a confidence vision algorithm based on Computer Vision. The confidence metric algorithm provides a confidence value as an indication of the degree of confidence. The confidence value is a probability that indicates a valid acquisition of features and a valid labeling of the data based on the acquired features. An assessment of the confidence metrics for the automatically labeled data is performed.

Based on the evaluation result of the confidence metrics, the annotation is displayed or not.

step S116 refers to filtering the training data based on a statistical analysis of the training data. Therefore can Traffic congestion, differently structured road surfaces, different lighting conditions or different traffic scenarios such as overtaking, changing lanes, roundabouts, parking areas or others are identified and used to filter the training data. Furthermore, the image data is processed based on the position information in order to determine different conditions for the image data from the same location. Image data provided at similar locations are identified and filtered in order to select image data with different properties, for example with different lighting conditions, different weather conditions, different traffic conditions or others, in the same location or in similar locations.

step S118 refers to automatically relabeling the set of automatically labeled data based on the calculated confidence metrics to generate the set of annotated training data. If the confidence metrics indicate a high degree of confidence in the automatically labeled data generated by the DCNN, the labels are not changed. Otherwise, the labels are modified using a labeling algorithm based on Computer Vision.

step S120 refers to saving the set of annotated training data for further use, eg for the subsequent training of the DCNN. The labeled training data is accumulated every time the owner of the vehicle travels a particular route using the vehicle, that is, when the vehicle is used under typical driving conditions. The training data are accumulated with respect to the respective route.

step S120 also includes an evaluation process if the set of annotated training data contains sufficient training data. A notification to the driver or owner of the vehicle can be generated as soon as the set of annotated training data contains sufficiently annotated training data to train the DCNN for an individual route, followed by processing to step S130 progresses.

3 shows an individual training of the DCNN based on the set of annotated training data for a particular route.

In detail, step refers S130 on identifying the route of the set of annotated training data. Depending on the route, various subsequent parallel paths of the method are carried out based on the identified route. The paths are executed identically, but using different model data for routes # 1 to #n of the DCNN and training data based on the identified route. Each parallel path contains steps S132 to S138 on.

step S132 relates to a step of determining scene statistics for the set of annotated training data. The scene statistics relate to a statistical determination of features that are contained in the training data. Therefore, label statistics of at least one information element are determined from a type of label, a position of labels and / or properties of labeled objects, preferably statistics of label combinations for several or all information elements from available types of labels, positions of the labels and / or properties of labeled objects are determined , The same applies here with reference to step S102 specified details. Therefore, a histogram analysis of the type of labels is carried out from the set of annotated training data. In addition, a histogram of the luminance and color properties is provided for each label type in order to enable a further distinction to be made between label types.

step S134 refers to classifying the set of annotated training data based on the scene statistics and automatically selecting training data from the set of annotated training data based on the classification. The same applies here with reference to step S104 specified details.

step S136 refers to setting up the deep convolutional neural network with individual model data that the one in step S130 are assigned to the identified route based on the general model data. This provides the DCNN for training on a particular route. When the method is carried out for the first time, the individual model data associated with the route are provided as a copy of the general model data. Otherwise, the individual model data of the deep convolutional neural network assigned to the route, which were previously saved, are loaded.

step S138 relates to a training step of the deep convolutional neural network with the individual model data assigned to the route using the respective set of annotated training data. In this embodiment, this includes training only the last layer of the DCNN. step S138 is according to the above description of the process step S106 executed.

step S140 refers to storing the individual model data of the deep convolutional neural network assigned to the route. Therefore, only the model data for the last layer of the DCNN saved, the remaining layers of the DCNN remain unchanged.

In this embodiment, all process steps S130 to S140 using the vehicle's driving support system that used the DCNN. The method is therefore implemented as an in-vehicle method.

The present invention also provides a method for applying a deep convolutional neural network for processing image data in a driving support system of a vehicle based on route information of a desired route, which is shown in the flowchart of 4 is shown. This procedure relates to using the vehicle with the driver assistance system that uses the DCNN.

The process begins with process step S200 , step S200 refers to identifying the route. step S200 is executed as with reference to step S130 has been described in detail.

step S210 refers to the establishment of the deep convolutional neural network with the individual model data assigned to the route. step S210 is carried out as described in detail with reference to S136. Therefore, the DCNN is activated with the individual model data of the respective route.

step S220 refers to processing the image data acquired when the route is traveled by the vehicle and using the driving support system that uses the deep convolutional neural network with the individual model data associated with the route. Therefore, an application of the DCNN in the driving support system is provided.

QUOTES INCLUDE IN THE DESCRIPTION

This list of documents listed by the applicant has been generated automatically and is only included for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• EP 2213980 B1 [0014]
• DE 10030932 A1 [0015]
• DE 10258470 B4 [0016]
• DE 10354910 B4 [0017]
• DE 19916967 C1 [0018]
• DE 102013211696 A1 [0019]
• DE 102015010542 A1 [0020]

Claims (16)

Method for training a deep convolutional neural network for processing image data for use in a driving support system of a vehicle using training data which contain route information, comprising the steps: Providing an initially trained deep convolutional neural network with general model data; Providing a set of annotated training data containing route information; Establishing the deep convolutional neural network with the route-assigned individual model data based on the general model data; Performing a training step of the deep convolutional neural network with the individual model data assigned to the route using the respective set of annotated training data; and Storage of the individual model data of the deep convolutional neural network assigned to the route. Procedure according to Claim 1 , characterized in that the step of providing a set of annotated training data containing route information, identifying a driving route, providing image data as training data from at least one camera when driving the route, and automatically annotating the image data to provide the set of annotated training data. Procedure according to Claim 2 , characterized in that the step of providing image data from at least one camera when driving the route comprises labeling the provided image data with position information, in particular with satellite-based position information. Procedure according to Claim 2 or 3 , characterized in that the step of automatically annotating the image data comprises the steps of: processing a set of non-annotated training data through the deep convolutional neural network to generate a set of automatically labeled data; Calculate confidence metrics for the set of automatically labeled data, and automatically relabel the set of automatically labeled data based on the calculated confidence metrics to generate the set of annotated training data. Procedure according to Claim 4 , characterized in that the step of calculating confidence metrics for the set of automatically labeled data comprises applying a confidence vision algorithm based on Computer Vision. Procedure according to Claim 4 or 5 , characterized in that the step of automatically relabeling the set of automatically labeled data based on the computed confidence metrics comprises applying a labeling algorithm based on Computer Vision. Method according to one of the preceding claims, characterized in that the step of setting up the convolutional neural network with the individual model data assigned to the route, generating the individual model data assigned to the route as a copy of the general model data or loading the previously stored individual model data of the deep convolutional neural network. Method according to one of the preceding claims, characterized in that performing a training step of the deep convolutional neural network with the individual model data assigned to the route using the respective set of annotated training data, training the last layers of the deep convolutional neural network, in particular only the last layer , having. Method according to one of the preceding claims, characterized in that the method is implemented as an in-vehicle method, the training step of the deep convolutional neural network with the individual model data assigned to the route using the respective set of annotated training data for training that provided for a specific vehicle deep convolutional neural network. Procedure according to Claim 9 , characterized in that the step of providing a set of annotated training data containing route information comprises providing the set of annotated training data containing route information using at least one camera of the vehicle. Procedure according to Claim 10 , characterized in that the step of providing of the set of annotated training data containing route information, using at least one camera of the vehicle, accumulating training data and evaluating whether the set of annotated training data contains enough training data. Procedure according to Claim 10 or 11 , characterized in that the step of providing the set of annotated training data containing route information using at least one camera of the vehicle comprises filtering the training data based on a statistical analysis of the training data. Method according to one of the preceding claims, characterized in that the step for executing a training step of the deep convolutional neural network with the individual model data assigned to the route using the respective set of annotated training data comprises the steps of: determining scene statistics for the set of annotated training data; Classifying the set of annotated training data based on the scene statistics; and automatically selecting training data from the set of annotated training data based on the classification. Procedure according to Claim 13 , characterized in that the step of determining scene statistics for the set of annotated training data comprises determining label statistics of at least one of a type of label, a position of labels and / or properties of labels, preferably determining statistics of label combinations. Method for applying a deep convolutional neural network for processing image data in a driving support system of a vehicle based on route information of a desired route, comprising the steps: Identifying the route; Establishing the deep convolutional neural network with the individual model data assigned to the route; and Processing the image data acquired when the route is traveled using the deep convolutional neural network with individual model data associated with the route. Driving support system for executing a method according to one of the Claims 1 to 15 ,

Images