DE102022208049A1 - Method for determining a parking space using a convolutional neural network - Google Patents

Abstract

The invention relates to a method for determining parking information describing at least one parking space (9) using a sensor device (2) of a vehicle (1), the sensor device (2) having at least one environment sensor (3) and a computing device (4), comprising Steps: - Acquiring measurement data describing an environment of the vehicle (1) with the at least one environment sensor (1), the measurement data being assigned to one or more objects in the environment of the vehicle (1), - Creating at least one of the surroundings of the vehicle (1 ) descriptive environment map (5) from the measurement data, the measurement data in the environment map being spatially specified in relation to the vehicle (1), - evaluating the at least one environment map (5) by the computing device (4), the environment map (5) is fed to a folding neural network, the neural network classifying and localizing existing parking spaces (9) from the environment map (5) in the vicinity of the vehicle (1), the neural network using regression to determine the relative arrangement of the parking spaces (9) to the Vehicle (1) is determined, and generating the parking information from the classified and localized parking spaces (9) and their relative arrangement.

Description

The invention relates to a method for determining parking information describing at least one parking space using a sensor device of a vehicle, the sensor device having at least one environment sensor and a computing device. The invention further relates to a method for training a convolutional neural network, a sensor device, a vehicle, a computer program product and a computer-readable storage medium.

Modern vehicles can use environmental sensors to obtain information about various objects in the vehicle's surroundings. These objects can be, for example, other road users such as moving or stationary third-party vehicles or immovable structures such as curbs, walls, markings or the like. The information obtained in this way can be used to provide different driver assistance functions.

For example, further information regarding available parking spaces in the area surrounding the vehicle can be derived from an environment description based on such object information. For this purpose, algorithms are generally used which determine free parking spaces from the objects detected in the environment and/or their arrangement according to defined rules and dependencies. The rules are defined, for example, through search tables and/or through individual case queries or if conditions.

This approach has the disadvantage that creating and checking these comparatively complex and extensive rules is time-consuming and that parking space detection is basically limited to the scenarios queried by the rules. Furthermore, further development of the rules and their adaptation to different vehicles and/or different requirements for parking space determination can only be implemented with great effort.

So-called deep learning methods, which provide for the evaluation of the image data using artificial intelligence, are known from other areas of technology for the machine evaluation of image data or for computer-based vision.

An example of such a method is that in the article by Redmon, Joseph, et al. “You only look once: Unified, real-time object detection.” Proceedings of the IEEE conference on computer vision and pattern recognition (2016) “You only look once method” (also referred to as YOLO method), which enables the simultaneous classification and localization of objects in images by using a grid in a single neural network. The neural network determines class probabilities assigned to the objects as well as the frames surrounding the objects in the image and localizing the object. The frames each surround an individual classified object, with the edges of the frames running parallel to the edges of the rectangular, evaluated image.

The invention is based on the object of specifying an improved method for determining a parking space, which in particular enables improved adaptability of parking space detection to different requirements as well as improved precision in parking space detection.

• - Erfassen von ein Umfeld des Fahrzeugs beschreibenden Messdaten mit dem wenigstens einen Umfeldsensor, wobei die Messdaten einem oder mehreren Objekten im Umfeld des Fahrzeugs zugeordnet sind,
• - Erstellen wenigstens einer das Umfeld des Fahrzeugs beschreibenden Umfeldkarte aus den Messdaten, wobei die Messdaten in der Umfeldkarte räumlich in Bezug zu dem Fahrzeug angegeben sind,
• - Auswerten der wenigstens einen Umfeldkarte durch die Recheneinrichtung, wobei die Umfeldkarte einem faltenden neuronalen Netzwerk zugeführt wird, wobei das neuronale Netzwerk aus der Umfeldkarte im Umfeld des Fahrzeugs vorhandene Parklücken klassifiziert und lokalisiert, wobei das neuronale Netz mittels Regression jeweils die relative Anordnung der Parklücken zu dem Fahrzeug ermittelt, und
• - Erzeugen der Parkinformation aus den klassifizierten und lokalisierten Parklücken sowie ihrer relativen Anordnung.

This object is achieved according to the invention by a method of the type mentioned at the outset, the method comprising the following steps:

• - Acquiring measurement data describing an environment of the vehicle with the at least one environment sensor, the measurement data being assigned to one or more objects in the area surrounding the vehicle,
• - Creating at least one environment map describing the environment of the vehicle from the measurement data, the measurement data being specified in the environment map spatially in relation to the vehicle,
• - Evaluating the at least one environment map by the computing device, the environment map being fed to a convolutional neural network, the neural network classifying and localizing parking spaces present in the area surrounding the vehicle from the environment map, the neural network determining the relative arrangement of the parking spaces by means of regression the vehicle is determined, and
• - Generating the parking information from the classified and localized parking spaces as well as their relative arrangement.

The method according to the invention is based on measurement data which are determined using one or more environmental sensors of a vehicle. The measurement data describes objects in the area surrounding the vehicle. The objects can in particular be third-party vehicles, infrastructure objects, buildings or structures as well as road and/or parking lot markings. In particular, the measurement data describes objects that can generally limit parking spaces. The measurement data can be recorded using one or more different types of environmental sensors.

Depending on the type of environment sensor, the measurement data can, for example, be measuring points to which a distance to the vehicle is assigned. Such measuring points can be generated, for example, using ultrasonic sensors, radar sensors or lidar sensors. In the case of a sensor designed as a camera, the measurement data as an object can describe, for example, a road or parking lot marking and its distance or arrangement relative to the vehicle.

An environment map is then created from the recorded measurement data, which in particular describes the current environment of the vehicle. For this purpose, the measurement data is specified spatially in relation to the vehicle, for example in a rectangular map in which the vehicle is in the center. The measurement data can be entered, for example, in a two-dimensional map, which depicts the plane extending along the vehicle's longitudinal direction and the vehicle's transverse direction. The environment map can be generated, for example, as an occupancy map. The environment map can preferably have a spatial grid into which the measurement data is entered or to which the measurement data is assigned.

In this way, the environment of the vehicle described by the measurement data can be mapped on the environment map.

The invention is based on the knowledge that such an environment map can be viewed as an image or that deep learning methods used for image evaluation can also be applicable to such an environment map. Accordingly, according to the invention, a convolutional neural network (CNN) is used to evaluate the environment map. This is preferably stored in the computing device of the sensor device so that the environment map can be fed to it as input data.

With the help of the neural network, the environment map is evaluated with regard to parking spaces in the area surrounding the vehicle or free parking spaces in the area surrounding the vehicle. To do this, the neural network classifies and locates parking spaces in the area surrounding the vehicle based on the surrounding map. In particular, only a single convolutional neural network is used for the evaluation, which carries out both the classification and the localization of the parking spaces. In other words, the neural network carries out an image evaluation process based on the YOLO method on the environment map. It is possible for a further neural network to be connected downstream of the folding neural network for evaluating the output data determined by the folding neural network.

In addition, the neural network uses regression to determine the relative arrangement of the parking spaces to the vehicle. In other words, in comparison to the classic YOLO method described at the beginning, the localization of a detected parking space is not only indicated by inserting a rectangular frame that encloses the parking space and is always aligned along the edges of the image, but instead the relative position or .the relative position and the relative orientation of the parking space to the vehicle. In the case of a rectangular parking space, for example, an angle by which the parking space is rotated relative to the vehicle can be determined. The parking space can advantageously be localized taking into account the information required for a particularly assisted or at least partially automated parking process.

The parking spaces can be classified in particular by assigning a classification probability to different sections of the surrounding map. In other words, the classification can be carried out by specifying a classification probability for different sub-areas of the environment map, the classification probability indicating the probability with which the sub-section is a parking space, i.e. a free parking space for the vehicle.

Parking information is then formed from the parking spaces classified by the neural network and their relative arrangement. For example, the parking information can contain at least the parking spaces whose classification probability is above a predetermined limit value, with the relative arrangement to the vehicle also being specified for these parking spaces.

The parking information can then be used to control an actuator of the vehicle and/or a display device of the vehicle. The actuator of the vehicle can, for example, be an actuator of the vehicle designed for transverse guidance and/or longitudinal guidance. The at least one actuator can be, for example, a drive motor of the vehicle and/or a steering drive of the vehicle.

Depending on the parking information, for example, automated or partially automated control of a parking process of the vehicle can take place, in particular by controlling at least one actuator of the vehicle. The vehicle can move forward and/or backward during the parking process, make steering angle changes to a set steering angle between and/or during these movements and, in particular, also carry out several movements or pulls in different directions. The parking process can in particular be carried out in such a way that the vehicle is positioned in at least one parking space described by the parking information.

In addition or as an alternative to the actuator, at least one display device of the vehicle can also be controlled, so that, for example, a driver of the vehicle can be made aware of the parking spaces determined in the area surrounding the vehicle, for example by means of a corresponding pictorial representation. The display device can be, for example, a display arranged in an interior of the vehicle, a heads-up display, a virtual side mirror and/or the like. The representation on the display device can be used, for example, to enable the driver to select one of several identified parking spaces as the destination of a parking process. Additionally or alternatively, aids such as virtual lines and markings for moving the vehicle into a determined and/or selected parking space can be displayed on the display device, which enable the driver to control a movement of the vehicle into this parking space.

The vehicle can be, for example, a motor vehicle such as a passenger car or a truck. Furthermore, the vehicle can also be a vehicle combination, which has, for example, a towing vehicle and one or more trailers. Alternatively, the method can also be used on other types of vehicles.

The sensor device of the vehicle can have one or more environment sensors. The surrounding sensors are arranged on the vehicle so that they can record the immediate surroundings of the vehicle. The sensor device further comprises a computing device, which can be part of the vehicle. Alternatively, it is possible for the computing direction to be a computing direction external to the vehicle, which communicates, for example, with the at least one environment sensor via a, in particular, wireless communication connection. Parking information determined by a vehicle-external computing device can also be transmitted back to the vehicle via the communication connection, so that the previously described control of the actuator and/or the display device of the vehicle can take place depending on the parking information.

The creation of the environment map from the measurement data of the at least one environment sensor can also be done by the computing device. Alternatively, this step can also be carried out by a further computing device of the vehicle and/or a further computing device external to the vehicle, with the further computing device transmitting the environment map to the computing device.

The neural network used to evaluate the environment map is a convolutional neural network (CNN), which is stored in the computing device or a storage unit of the computing device. The environment map is evaluated using artificial intelligence or a deep learning method. The folding neural network is set up or trained to classify and locate existing parking spaces in the area surrounding the vehicle from the map of the surroundings and to use regression to determine the relative arrangement of the parking spaces to the vehicle. In this way, the parking space can be determined based on the surrounding map using a process that can be adapted and improved using machine learning. This makes it possible to flexibly adapt the parking space determination to different scenarios by appropriately training the neural network fit. This advantageously provides an easy-to-implement option for adapting the parking space determination.

For the purposes of the invention, the term “machine learning” (ML) refers to the use of algorithms for analyzing training data, learning from this training data and then determining or predicting an initially unknown property of input data to be evaluated. Both controlled learning and uncontrolled learning can be used in this context. Among other things, the labeling strategy or the labeling of training data can be crucial for its analysis.

The term “deep learning” (DL) in the sense of the invention includes a cascade of several layers of nonlinear processing units (usually artificial neural networks) for the extraction and transformation of features (or for the extraction and transformation of parameters). The term “artificial neural network” (ANN) or “convolutional neural network” (CNN) includes networks of artificial neurons that are connected to each other in the abstract, as in a nervous system of a living being, in terms of information processing. The neurons can be represented as nodes and their connections as edges in a diagram. The backmost (node) layer of the network is called the output layer and the “invisible” (node) layers that are at the front are called hidden layers.

Such artificial neural networks can be single-layer (one output layer), two-layer (one output layer and one hidden layer to improve abstraction) or multi-layer (at least one output layer and several hidden layers to improve abstraction). In addition, with regard to their data transmission, these can be designed to be forward-facing (feedforward) and/or with backward-facing edges (recurrent connections) (feedback; feedback network).

For the purposes of the invention, the term “computer-implemented method” refers to a flowchart that can be implemented and carried out on the basis of a data processor. The data processor, e.g. a computer, a computer network or other programmable device, can process data using programmable calculation rules. With regard to the method, essential properties can be realized, e.g. through a new program, new programs, algorithms or similar.

According to the invention, it can be provided that the environment map contains a map grid and the measurement data are assigned to the cells of the map grid, with for each cell the number of measurement points of the measurement data assigned to the cell, the height of an object described by the measurement data assigned to the cell, the type of of the object described by the measurement data assigned to the cell and / or at least one signal property is specified by the measurement data assigned to the cell.

For example, for measurement data that comes from a type of sensor that evaluates reflected echoes of emitted signals, the number of echoes reflected from an object in the cell can be stored as signal properties. A sensor of this type can be, for example, an ultrasonic sensor, a radar sensor or a lidar sensor. The echoes can in particular each represent a measuring point. Furthermore, it is possible that information about the height of the object on which the echo was reflected can also be determined from the echoes or measuring points from such sensors. Height information describing this height of the object can also be assigned to the individual cells of the map grid of the surrounding map for each measuring point or averaged over several measuring points.

For example, the intensity of the echoes or measuring points assigned to the cell can be stored in the environment map as a signal property. The intensity can be specified for the individual measuring points or averaged. Depending on the sensor type, further properties of the measurement data, in particular properties that enable conclusions to be drawn about an object type and/or a detection quality, can also be entered in the environment map.

Furthermore, the type of an object described by the measurement data for the respective cell can also be specified in the environment map or assigned to the individual cells. The type of an object can be determined from the measurement data, for example from measurement data obtained via an environmental camera. In this way, objects that cannot or only have ultrasonic, radar or lidar sensors can also be detected that are difficult to detect, such as road or parking lot markings, must be entered into the area map.

This information is assigned in particular to those cells whose position corresponds to the object position determined via the sensor device. In particular, no measurement data can be assigned to cells of the environment map that correspond to a section of space in the area surrounding the vehicle that is not blocked by an object, so that they remain free.

According to the invention, it can be provided that the neural network is set up to overlay the environment map with an evaluation grid comprising several cells, with each cell being assigned at least one frame, with a classification as a parking space and / or an orientation for each of the frames in the parking information a parking space described by the measurement data and assigned to the frame is specified relative to the vehicle.

The frame, which can also be referred to as a box, has in particular a rectangular shape. It is possible for the frames of several cells of the evaluation grid to be positioned and/or aligned in the same way with respect to the cells. The frames each represent initial assumptions for possible parking spaces, to which a classification information, for example a classification probability, for classification as a parking space can be assigned by the neural network. This also enables at least a basic localization of possible parking spaces in the surrounding map. Furthermore, at least for frames classified as parking spaces, additional properties that describe the exact positioning and orientation of the parking space and/or their respective width and length can be determined by means of regression by the neural network.

In a preferred embodiment of the invention, it can be provided that an evaluation grid is used in which several differently positioned and/or differently oriented frames are assigned to each cell. The frames can, for example, be two rectangular frames arranged rotated by 90° relative to one another. This simplifies the determination of the relative alignment of the parking space to the vehicle, since it can initially be assumed that the frame already corresponds more closely to a parking space described by the surrounding map or has a higher probability of classification. It is also possible that each cell of the evaluation rater contains more than two frames and/or that they are oriented in a different way to one another. The orientation of the parking space can be determined by the frames or localization and/or orientation parameters can be determined by several differently positioned and/or differently oriented frames. The more frames are used, the more precisely the localization and determination of the relative arrangement of the parking space in relation to the vehicle can be carried out.

According to the invention, it can be provided that the parking information for at least some of the classified parking spaces additionally contains position information, the position information describing a target position for the vehicle that is assigned to the respective parking space and lies within the parking space. The position information can be used to take into account that the parking space is generally larger, i.e. offers more parking space than is occupied by the vehicle after parking. The target position can therefore in particular lie completely within the parking space and, for example, maintain a predetermined safety distance from the edges or edges of the parking space. The target position can also be described, for example, by a rectangle which has a shorter edge length than the parking space and/or is rotated at an angle relative to the parking space. The position information can also advantageously be determined using the convolutional neural network, in particular also using regression.

In a preferred embodiment of the invention it can be provided that the classification of the parking space includes a classification as a perpendicular parking space and as a parallel parking space. In addition to evaluating the environment map, the determination of the relative arrangement of a parking space to the vehicle can also be advantageously used to classify a parking space as a perpendicular parking space and/or as a parallel parking space. A parallel parking space describes a parking space in which you can park alongside the direction of travel of a street on which the vehicle is located. Accordingly, a transverse parking space refers to a parking space in which the vehicle can be parked perpendicular to the direction of travel of a street on which the vehicle is located. A distinction between a perpendicular parking space and a parallel parking space can be made, for example, by providing two rectangular frames rotated by 90° relative to each other in the evaluation grid.

In a preferred embodiment of the invention, it can be provided that a sensor device is used which comprises at least one ultrasonic sensor, at least one lidar sensor, at least one radar sensor and/or at least one surroundings camera as at least one environment sensor. The sensor device can in particular comprise several of the aforementioned sensor types and/or combinations of the aforementioned sensor types. The respective measurement data from different sensors can in particular be entered or merged into the same environment map.

• - Bereitstellen wenigstens eines Trainingsdatensatzes umfassend mehrere jeweils das Umfeld des Fahrzeugs beschreibende Trainings-Umfeldkarten aus Trainings-Messdaten, wobei die Trainings-Messdaten in der Trainings-Umfeldkarte räumlich in Bezug zu einem Fahrzeug angegeben sind,
• - Zuweisen einer wenigstens eine Parklücke in dem Umfeld des Fahrzeugs beschreibenden Grundwahrheit jeder der Trainings-Umfeldkarten, wobei die Grundwahrheit zumindest einen Abschnitt der Trainings-Umfeldkarte als eine Parklücke klassifiziert und lokalisiert, wobei die Grundwahrheit die relative Orientierung der Parklücke zu dem Fahrzeug angibt,
• - Überlagern der Trainings-Umfeldkarten jeweils mit einem mehrere Zellen umfassenden Auswertungsraster, wobei jeder Zelle des Auswertungsrasters wenigstens ein Rahmen zugewiesen ist,
• - Ermitteln des jeweiligen Rahmens mit dem größten Überlapp zu der wenigstens einen, durch die Grundwahrheit beschriebenen Parklücke, und
• - Optimieren des neuronalen Netzes hinsichtlich einer Übereinstimmung in der relativen Anordnung zwischen dem jeweiligen Rahmen mit dem größten Überlapp und der durch die Grundwahrheit beschriebenen Parklücke.

The invention further relates to a method for training a convolutional neural network, comprising the steps:

• - Providing at least one training data set comprising several training environment maps from training measurement data, each describing the environment of the vehicle, the training measurement data being specified spatially in relation to a vehicle in the training environment map,
• - Assigning a ground truth describing at least one parking space in the area surrounding the vehicle to each of the training environment maps, the ground truth classifying and localizing at least a section of the training environment map as a parking space, the ground truth indicating the relative orientation of the parking space to the vehicle,
• - Overlaying the training environment maps with an evaluation grid comprising several cells, with at least one frame being assigned to each cell of the evaluation grid,
• - Determining the respective frame with the largest overlap to the at least one parking space described by the ground truth, and
• - Optimize the neural network with regard to a match in the relative arrangement between the respective frame with the largest overlap and the parking space described by the ground truth.

The method according to the invention for training a folding neural network can be used in particular for training a neural network used in a method according to the invention for determining parking information describing at least one parking space. Through the training, the internal parameters of the neural network can be adjusted so that it can be set up or trained for the evaluation of the environment maps created from the measurement data according to the method according to the invention for determining parking information describing at least one parking space.

The creation of the training environment maps from the training measurement data is carried out in particular analogous to the creation of the environment maps from the measurement data, as described above. The training measurement data are indicated in the training environment map in relation to the position of a vehicle. The training measurement data can, for example, have been determined at an earlier point in time by means of a sensor device of a vehicle comprising at least one environment sensor and can then be used for training. Alternatively, training measurement data created and/or calculated in a different way can also be used.

A ground truth is then assigned to each of the training environment maps, with the ground truth describing one or more parking spaces in the environment map. In addition, the ground truth also indicates the location for locating the parking space and the relative orientation of the parking space to the vehicle described in the surrounding map. The ground truth represents the result to which the evaluation of the neural network is optimized during training.

The training environment maps are then overlaid during training with an evaluation grid comprising several cells, which can also be referred to as a training evaluation grid in relation to the training of the neural network. At least one frame or at least one box is assigned to the cells of the evaluation grid. These represent initial assumptions for possible parking spaces.

The frame that has the largest overlap with a parking space described by the ground truth is then selected from the plurality of frames in the evaluation grid. In particular, the frame with the largest overlap is determined for each parking space described in the ground truth. The overlap can also be referred to as Intersection Over Union (IoU) and indicates the proportion to which each of the frames in the evaluation grid overlaps with one of the parking spaces described by the ground truth.

After determining the one or more frames with the largest overlap to one or more parking spaces described by the ground truth, the neural network is optimized with regard to a match in the relative arrangement between the respective frame with the largest overlap and the one described by the ground truth. parking space assigned to the frame. In this way, the neural network can be trained efficiently for the recognition of parking spaces in maps of the surrounding area, in particular for use in a method according to the invention for determining parking information describing at least one parking space.

According to the invention, it can be provided that, in order to determine the overlap, it is determined in which cell the center of the at least one parking space described by the ground truth lies, with the frame of this cell being selected as the frame with the largest overlap, the area-related overlap of which with that indicated by The parking space described in the basic truth is the largest. If it is first determined in which cell of the evaluation grid the center of the parking space or a parking space described by the ground truth is located, the number of frames for which the overlap is determined can be reduced, so that fewer calculation steps are required and thus more efficient training is possible is.

In a preferred embodiment of the invention, it can be provided that an evaluation grid is used in which several differently positioned and/or differently oriented frames are assigned to each cell and/or that the classification of the parking space includes a classification as a perpendicular parking space and as a parallel parking space.

Analogous to the determination of the parking information describing at least one parking space, several frames per cell of the evaluation grid can also be used during training. These frames can in particular be rectangular and arranged at different positions and/or with different orientations to the cards of a rectangular cell of the evaluation grid. These frames can be used, for example, to make a distinction between a perpendicular parking space and a parallel parking space and thus also to train the neural network with regard to a corresponding classification or distinction.

According to the invention, it can be provided that the ground truth for the at least one parking space described by the ground truth additionally contains position information, the position information describing a target position for the vehicle assigned to the respective parking space and lying within the parking space, the optimization of the neural network also with regard to Position information is provided. In this way, it can be achieved that the neural network is also trained with regard to position information located within the parking space, so that this can also be determined by the neural network.

The invention further relates to a sensor device comprising at least one environment sensor and a computing device, wherein the computing device is set up to carry out a method according to the invention for determining parking information describing at least one parking space.

The invention further relates to a vehicle comprising at least one sensor device according to the invention.

The invention also relates to a computer program product comprising instructions which cause a computing device to carry out a method according to the invention for determining parking information describing at least one parking space.

In addition, the invention relates to a computer-readable storage medium comprising a computer program product according to the invention. The computer-readable storage medium can be, for example, a non-transient data carrier, for example a CD, a DVD, a floppy disk, a flash memory or the like.

All of the advantages and refinements described above for the method according to the invention for determining parking information describing at least one parking space also apply accordingly to the method according to the invention for training a convolutional neural network and vice versa.

Furthermore, all advantages and refinements described in relation to one of the methods according to the invention also apply correspondingly to the sensor device according to the invention, the vehicle according to the invention, the computer program product according to the invention and the computer-readable storage medium according to the invention and vice versa.

• 1 ein Ausführungsbeispiel eines erfindungsgemäßen Fahrzeugs umfassend ein Ausführungsbeispiel einer erfindungsgemäßen Sensoreinrichtung,
• 2 eine Darstellung einer Umfeldkarte zur Erläuterung eines Ausführungsbeispiels eines erfindungsgemäßen Verfahrens zur Ermittlung einer wenigstens eine Parklücke beschreibenden Parkinformation und
• 3 eine Darstellung einer mit einem Auswerteraster überlagerten Umfeldkarte zur Erläuterung des Ausführungsbeispiels eines erfindungsgemäßen Verfahrens zur Ermittlung einer wenigstens eine Parklücke beschreibenden Parkinformation sowie eines Ausführungsbeispiels zum Trainieren eines neuronalen Netzes.

Further advantages and details of the invention emerge from the exemplary embodiments described below and from the drawings. These are schematic representations and show:

• 1 an embodiment of a vehicle according to the invention comprising an embodiment of a sensor device according to the invention,
• 2 a representation of an environment map to explain an exemplary embodiment of a method according to the invention for determining parking information describing at least one parking space and
• 3 a representation of an environment map superimposed with an evaluation grid to explain the exemplary embodiment of a method according to the invention for determining parking information describing at least one parking space and an exemplary embodiment for training a neural network.

In 1 an exemplary embodiment of a vehicle 1 according to the invention is shown. The vehicle 1 can be, for example, a motor vehicle such as a passenger car or a truck. Furthermore, the vehicle 1 can also be a vehicle combination, which has, for example, a towing vehicle and one or more trailers. Alternatively, the method can also be used on other types of vehicles.

The vehicle 1 includes a sensor device 2, which has several environment sensors 3 and a computing device 4. The environment sensors 3 of the sensor device 2 are designed, for example, as ultrasonic sensors. Additionally or alternatively, at least some of the surroundings sensors 3 can also be designed as a surroundings camera, as a radar sensor and/or as a lidar sensor.

The computer device 4 of the sensor device 2 is shown as part of the vehicle 1 in the present exemplary embodiment. It is possible that the computing device 4 is integrated into one of the environment sensors 3 or that it is a computing device external to the vehicle. A vehicle-external computing device can communicate with the environment sensors 3 and/or other devices of the vehicle 1 via a particularly wireless communication connection.

Using the surrounding sensors 3, measurement data from the surroundings of the vehicle 1 can be determined. The measurement data describes objects that are in the vicinity of the vehicle 1. The objects can be, for example, other road users such as other vehicles or similar. Furthermore, immovable objects, such as infrastructure objects, parts of buildings or similar, can also be recorded as objects.

The computing device 4 of the sensor device 2 is designed to carry out a method for determining parking information describing at least one parking space in the vicinity of the vehicle 1. Depending on the parking information, for example, an automated or partially automated control of a parking process of the vehicle 1 can take place, in particular by controlling at least one actuator (not shown) of the vehicle 1. The vehicle 1 can move forward and / or backward during the parking process, carry out steering movements and in particular, carry out several partial movements or moves. The parking process can in particular be carried out in such a way that the vehicle is positioned in at least one parking space described by the parking information.

In addition or as an alternative to the actuator, a display device (not shown) of the vehicle 1 can also be controlled, so that, for example, a driver of the vehicle 1 can be made aware of the parking spaces determined in the area surrounding the vehicle, for example by means of a corresponding pictorial representation. The display device can be, for example, a display arranged in an interior of the vehicle 1, a heads-up display, a virtual side mirror or the like.

The representation on the display device can be used, for example, to enable the driver to select one of several identified parking spaces as the destination of a parking process. Additionally or alternatively, aids can be used to move the vehicle 1 into a determined one and/or selected parking space are displayed on the display device, which enable the driver to move the vehicle 1 into this parking space.

To determine the parking information with the computing device 4 or to determine the parking information in a computer-implemented method, measurement data describing the surroundings or the surroundings of the vehicle 1 are first determined with one or more of the surroundings sensors 3, the measurement data being assigned to one or more objects in the surroundings of the Vehicle 1 are assigned. At least one environment map 5 describing the environment of the vehicle 1 is then created from the measurement data, the measurement data being indicated spatially in relation to the vehicle 1 in the environment map 5.

In 2 an exemplary embodiment of such an environment map 5 is shown. The environment map 5 is a rectangular map in which the vehicle 1 is arranged in the middle. In the environment map 5, the measurement data are given spatially in relation to the vehicle 1. The measurement data are shown here as measurement points 6, which can be determined, for example, by environmental sensors 3 designed as ultrasonic sensors. The measuring points 6 of the measurement data each describe a signal echo that was reflected by an object in the vicinity of the vehicle 1.

The individual measuring points 6 each include a distance measurement in relation to the vehicle 1, which can be carried out, for example, from a transit time measurement. In addition, a direction can also be determined for each of the measuring points 6, so that the reflection locations or the objects assigned to the measuring points 6 can be placed in spatial relation to the vehicle 1.

The environment map 5 further comprises a map grid 7, with the measurement data being assigned to the individual cells 8 of the map grid 7. For each of the cells 8, for example, the number of measurement points 6 of the measurement data assigned to the cell 8 can be determined and stored in the environment map 5. Furthermore, the height of an object described by the measurement data assigned to the cell 8, the type of such an object and/or at least one further signal property of the measurement data assigned to the cell can also be specified for the individual cells 8.

The signal property can be, for example, the intensity of the individual measuring points 6. It is possible for a signal property such as the intensity to be specified, for example averaged for all measuring points of one of the cells 8. This can also be done accordingly for information regarding the height of an object and/or the type of an object.

In addition to the measurement points 6 of environment sensors 2 designed as ultrasonic sensors, measurement points 6 of environment sensors 3 designed as radar sensors and / or lidar sensors can also be entered into the environment map 5 accordingly. Furthermore, objects determined via other types of sensors, for example surroundings sensors 3 designed as surroundings cameras, can also be entered in the surroundings map 5. In this way, objects such as road markings, parking lot markings or the like can also be entered in the environment map 5.

The at least one environment map 5 is then evaluated by the computing device 4, the environment map 5 being fed to a folding neural network implemented by the computing device 4, the neural network classifying and localizing parking spaces 9 present in the area surrounding the vehicle 1 from the environment map 5, wherein the neural network uses regression to determine the relative arrangement of the parking spaces 9 to the vehicle 1. From the classified parking spaces 9 and their relative arrangement to the vehicle 1, the computing device 4 or another computing device (not shown) then generates the parking information describing at least one parking space 9 in the surroundings of the vehicle 1. Advantageously, the environment map 5 can be processed like a two-dimensional image file by the neural network.

The neural network is set up to overlay the environment map 5 with an evaluation grid 11 comprising several cells 10, with each cell 10 being assigned at least one frame 12. An environment map 5 overlaid with an evaluation grid 11 is in 3 shown.

In the present case, each cell 10 comprises two rectangular frames 12, 13, which are arranged rotated by 90°. The frames 12, 13 are therefore oriented differently with respect to the cell 10. It is possible for additional frames (not shown) to be assigned to the cells 10 for example, have the same orientation as one of the frames 12, 13, but are arranged offset along one of the edges of the cell 10 in the x direction and / or in the y direction.

In the parking information, a classification as a parking space and/or an orientation of a parking space 9 described by the measurement data and assigned to the frame 12, 13 relative to the vehicle 1 is specified for each of the frames 12, 13 of all cells 10 of the evaluation grid 11. This makes it possible for the classification of the parking space 9 to include a classification as a perpendicular parking space and as a parallel parking space. A parking space 9 can be classified as such if a classification probability determined by the neural network is above a predetermined limit value. Accordingly, a parking space can be classified as a parallel parking space or as a perpendicular parking space if the corresponding classification probability assigned to one of the frames 12, 13 is above a predetermined limit value.

The neural network additionally determines, at least for the frames 12, 13 classified as parking space 9, the relative arrangement of the parking space 9 to the vehicle 1 by means of regression. In this way, an exact localization of the parking space 9 in relation to the vehicle 1 is determined. Furthermore, different orientations of the parking spaces or different relative arrangements between vehicle 1 and a determined parking space 9 can be taken into account in this way.

The parking information derived from the output of the neural network can additionally contain position information for at least some of the recognized parking spaces 9, the position information describing a target position for the vehicle 1 that is assigned to the respective parking space 9 and lies within the parking space 9.

ausgedrückt werden kann. Dabei sind die dem Rahmen 12 zugeordneten Werte beispielsweise mit dem Index „1“ und die dem Rahmen 13 zugeordneten, größtenteils aus Übersichtsgründen nicht dargestellten Werte mit dem Index „2“ gekennzeichnet. Bei den einzelnen Werten bzw. Komponenten des Vektors beschreiben:

• P: einen Indikator (z.B. 0 oder 1), ob es sich bei dem entsprechenden Rahmen 12, 13 um eine Parklücke 9 handelt,
• b_x, b_y: die Koordinaten in x-Richtung und y-Richtung eines Mittelpunkts der ermittelten Parklücke 9, zum Beispiel als Prozentangabe bezogen auf die Größe der Zelle 10 ausgehend von der jeweils unteren linken Ecke,
• b_w, b_h: die Breite und die Höhe der ermittelten Parklücke 9, angegeben beispielsweise als Prozentangabe in Bezug zu der Breite und der Höhe der Zelle 10 des Auswerterasters 11,
• be: Winkel bzw. Verdrehung der Parklücke 9 in Bezug zu der Zelle 10,
• t_x, t_y: Koordinaten eines Mittelpunkts einer Zielposition innerhalb der ermittelten Parklücke 9, angegeben zum Beispiel als Prozentangabe in Bezug zu der Zelle 10 und relativ zu deren Mittelpunkt,
• te: Winkel der Zielposition, angegeben zum Beispiel in Bezug zu der ermittelten Parklücke 9.
• C_par, C_perp, C_void: Die Klassenwahrscheinlichkeiten für das Vorliegen einer Längsparklücke (C_par), für das Vorliegen einer Querparklücke (C_perp) und für das Vorliegen keiner Parklücke (C_void).

The parking information generated can, for example, be a tensor which indicates for each of the frames 12, 13 whether it is a parking space 9 or not or whether it is a perpendicular parking space or a parallel parking space. Further information can be provided for each frame 12, 13 classified as a parking space 9. For each cell 10 of the evaluation grid 11, the parking information can contain, for example, a vector y _cell , which can be used as $$y_{cell}={\left[\begin{array}{cccccccccccccc}{P}_1& b_{x1}& b_{y1}& b_{w1}& b_{H1}& b_{\theta 1}& t_{x1}& t_{y1}& t_{\theta 1}& C_{part1}& C_{per1}& C_{vOid}& {\mathrm{<figure-callout\; id="2"\; label="P"\; filenames="DE102022208049A1\_0003.png"\; state="\{\{state\}\}">P</figure-callout>}}_2& \dots \end{array}\right]}^T$$

can be expressed. The values assigned to the frame 12 are marked, for example, with the index “1” and the values assigned to the frame 13, which are largely not shown for reasons of clarity, are marked with the index “2”. Describe the individual values or components of the vector:

• P: an indicator (e.g. 0 or 1) as to whether the corresponding frame 12, 13 is a parking space 9,
• b _x , b _y : the coordinates in the x direction and y direction of a center point of the determined parking space 9, for example as a percentage based on the size of the cell 10 starting from the lower left corner,
• b _w , b _h : the width and height of the determined parking space 9, specified, for example, as a percentage in relation to the width and height of cell 10 of the evaluation grid 11,
• be: angle or rotation of the parking space 9 in relation to the cell 10,
• t _x , t _y : coordinates of a center point of a target position within the determined parking space 9, specified for example as a percentage in relation to the cell 10 and relative to its center point,
• te: Angle of the target position, specified for example in relation to the determined parking space 9.
• C _par , C _perp , C _void : The class probabilities for the presence of a parallel parking space (C _par ), for the presence of a perpendicular parking space (C _perp ) and for the existence of no parking space (C _void ).

The indicator P can, for example, be set to one if one of the probabilities C _par or C _perp is above a predetermined limit, or set to zero if the probability C _void is above a predetermined limit.

The information t _x , t _y and t _θ form position information which describes the target position for the vehicle 1 that is assigned to the respective parking space and lies within the parking space.

The parking information output, for example, in the form of a tensor has a dimensionality of A × B × (y _cell ) or A × B × (n ∗ i),
where A and B represent the number of rows and columns of the evaluation grid and their product therefore indicates the number of cells in the evaluation grid and y _cell is the vector assigned to a cell, which generally has n ∗ i components, where n is the number of frames per Cell and i describes the number of information determined per frame in the cell. In the example above for y _cell , n = 2 and i = 12.

In an exemplary embodiment of a method for training the folding neural network, at least one training data set comprising several training environment maps each describing the environment of the vehicle 1 is provided from training measurement data. The measurement data in the training environment map are spatially specified in relation to a vehicle 1. The training method serves in particular to train the convolutional neural network implemented in the computing device 4 to carry out the exemplary embodiment described above.

Each of the training environment maps is assigned a ground truth describing at least one parking space 14 in the surroundings of the vehicle 1, the ground truth classifying and localizing at least a section of the training environment map as a parking space 14. In addition, the ground truth indicates the relative orientation of the parking space 14 to the vehicle. In 3 a parking space 14 specified by the ground truth is shown. The training environment maps basically correspond to the environment maps 5, as described above.

The training environment maps are also used during training, as described above and as in 3 is shown, each with an evaluation grid 11 comprising several cells 10, each of the cells 10 of the evaluation grid 11 being assigned at least one frame 12, 13. The respective frame 12, 13 with the largest overlap to the at least one parking space 14 described by the ground truth is then determined. The neural network is then checked for a match in the relative arrangement between the respective frame 12, 13 with the largest overlap and the parking space 14 described by the ground truth is optimized.

To determine the overlap, it is determined in which of the cells 10 of the evaluation grid 11 the center point 15 of the at least one parking space 14 described by the ground truth lies. This is in the in 3 The example shown is cell 10, which is in the second row and the fifth column.

Subsequently, the frame 12, 13 in this cell is selected as the frame with the largest overlap, whose overlap with the parking space 14 described by the ground truth is largest. To determine the respective overlap, a value for its area-related overlap or its Intersection Over Union (IoU) is determined for each of the frames 12, 13 in the cell in which the center 15 lies.

In the in 3 In the example shown, the overlap with the vertically oriented frame 12 is larger, so that this is chosen as the goal of optimizing the neural network. Based on the choice of the frame 12, 13 and the arrangement of the vehicle 1 in the surrounding map, it can be determined or trained accordingly whether it is a parallel parking space or, as in the present case, a perpendicular parking space.

It is possible for an evaluation grid 11 to be used in which more than two frames 12, 13 are assigned to each cell 10. The multiple frames 12, 13 can be positioned differently and/or oriented differently. Different dimensions, such as a different width, a different length, a different position of the center of the frame 12 and the parking space 14 with respect to the cell 10 as well as the relative orientation (angle θ) between the frame 12 and the parking space 14, which are determined by the neural network are also determined using regression and are also trained.

The orientation of the parking space 14 can be approximated by the frames 12, 13 or, by using two or more frames 12, 13 per cell 10, differently positioned and/or differently oriented localization and/or orientation parameters can be initially specified. The more frames 12, 13 are used, the more precisely the neural network can localize and determine the relative arrangement of the parking space 14 in relation to the vehicle.

Furthermore, the ground truth for the at least one parking space 14 can additionally contain position information, the position information describing a target position for the vehicle 1 that is assigned to the respective parking space 14 and lies within the parking space, the optimization of the neural network also taking place with regard to the position information. This can be done accordingly by training the determination of the parameters tx, ty and tθ.

Optimizing the neural network can be done using a loss function. The loss function can minimize error squares for all variables of the vector y _cell or for all variables i, which should be able to be determined in relation to one of the frames 12, 13. Additionally or alternatively, a softmax function can also be used or only the errors of the class probabilities Cpar, Cperp and Cvoid and/or the classification probability P _n can be minimized.

An example of a loss function is ℓ(ŷ,y). $$\begin{array}{c}\mathcal{l}\left(\widehat{y},y\right)={\left({\widehat{y}}_1-y_1\right)}^2+{\left({\widehat{y}}_2-y_2\right)}^2+\cdots +{\left({\widehat{y}}_n-y_n\right)}^2,wenn{y}_1=1\\ \mathcal{l}\left(\widehat{y},y\right)={\left({\widehat{y}}_1-y_1\right)}^2,wenn{y}_1=0.\end{array}$$

If y ₁ = 1, it means that the frame 12 assigned the index “1” has been classified as a parking space. In this case, the loss function is used to calculate the squared error of each variable i, ie each component y _n of the vector y _cell . If y ₁ = 0, this means that the frame 12 assigned to the index "1" was classified as "not a parking space", in which case only the squared error in the indicator or the classification probability P _n , i.e. the component y ₁ of the vector y _cell , is minimized.

Claims (15)

Method for determining parking information describing at least one parking space (9) using a sensor device (2) of a vehicle (1), the sensor device (2) having at least one environment sensor (3) and a computing device (4), comprising the steps: - Acquiring measurement data describing an environment of the vehicle (1) with the at least one environment sensor (1), the measurement data being assigned to one or more objects in the environment of the vehicle (1), - Creating at least one environment map (5) describing the environment of the vehicle (1) from the measurement data, the measurement data being specified in the environment map spatially in relation to the vehicle (1), - Evaluating the at least one environment map (5) by the computing device (4), the environment map (5) being fed to a folding neural network, the neural network from the environment map (5) identifying parking spaces (9) in the area surrounding the vehicle (1). ) classified and localized, the neural network using regression to determine the relative arrangement of the parking spaces (9) to the vehicle (1), and - Generating the parking information from the classified and localized parking spaces (9) as well as their relative arrangement. Procedure according to Claim 1 , characterized in that the environment map (5) contains a map grid (7) and the measurement data are assigned to the cells (8) of the map grid (7), with the number of measuring points (8) assigned to the cell (8) for each cell (8). 6) the measurement data, the height of an object described by the measurement data assigned to the cell (8), the type of an object described by the measurement data assigned to the cell (8) and / or at least one signal property of the measurement data assigned to the cell (8) is specified . Procedure according to Claim 1 or 2 , characterized in that the neural network is set up to overlay the environment map (5) with an evaluation grid (11) comprising several cells (10), with each cell (10) being assigned at least one frame (12, 13), whereby in the parking information for each of the frames (12, 13) a classification as a parking space and/or an orientation of a parking space described by the measurement data and assigned to the frame (12, 13) is specified relative to the vehicle (1). Procedure according to Claim 3 , characterized in that an evaluation grid (11) is used, in which each cell (10) is assigned several differently positioned and/or differently oriented frames (12, 13). Method according to one of the preceding claims, characterized in that the parking information for at least some of the classified parking spaces (9) additionally contains position information, the position information being a target position for the respective parking space (9) assigned and within the parking space (9). Vehicle (1) describes. Method according to one of the preceding claims, characterized in that the classification of the parking space (9) includes a classification as a perpendicular parking space and as a parallel parking space. Method according to one of the preceding claims, characterized in that a sensor device (2) is used, which comprises at least one ultrasonic sensor, at least one lidar sensor, at least one radar sensor and/or at least one environment camera as at least one environment sensor (3). Method for training a convolutional neural network, in particular for use in a method according to one of the preceding claims, comprising the steps: - Providing at least one training data set comprising several training environment maps each describing the environment of a vehicle (1) from training measurement data, the training measurement data being specified in the training environment map spatially in relation to the vehicle (1), - Assigning a ground truth describing at least one parking space (14) in the area surrounding the vehicle (1) to each of the training environment maps, the ground truth classifying and localizing at least a section of the training environment map as a parking space (14), the ground truth being the relative indicates the orientation of the parking space (14) to the vehicle (1), - Overlaying the training environment maps with an evaluation grid (11) comprising several cells (10), with at least one frame (12, 13) being assigned to each cell (10) of the evaluation grid (11), - Determining the respective frame (12, 13) with the largest overlap to the at least one parking space (14) described by the ground truth, - Optimizing the neural network with regard to a match in the relative arrangement between the respective frame (12, 13) with the largest overlap and the parking space (14) described by the ground truth. Procedure according to Claim 8 , characterized in that in order to determine the overlap, it is determined in which cell (10) the center point (15) of the at least one parking space (14) described by the ground truth is located, the frame (12, 13) of this cell (10 ) is chosen as the frame (12, 13) with the largest overlap, whose overlap with the parking space (14) described by the ground truth is largest. Procedure according to Claim 8 or 9 , characterized in that an evaluation grid (11) is used, in which each cell (10) is assigned several differently positioned and/or differently oriented frames (12, 13) and/or that the classification of the parking space described by the ground truth (14 ) includes a classification as a perpendicular parking space and a parallel parking space. Procedure according to one of the Claims 8 until 10 , characterized in that the ground truth for the at least one parking space (14) additionally contains position information, the position information describing a target position for the vehicle (1) assigned to the respective parking space (14) and lying within the parking space (14), the Optimization of the neural network also takes place with regard to the position information. Sensor device comprising at least one environment sensor (3) and a computing device (4), the computing device (4) being used to carry out a method according to one of Claims 1 until 7 is set up. Vehicle comprising at least one sensor device (2). Claim 12 . Computer program product comprising instructions which cause a computing device (4) to carry out a method according to one of the Claims 1 until 7 to carry out. Computer-readable storage medium comprising a computer program product Claim 14 .

Images