WO2024027884A1

WO2024027884A1 - Method for determining a parking space with the aid of a convolutional neural network

Info

Publication number: WO2024027884A1
Application number: PCT/DE2023/200153
Authority: WO
Inventors: Mohanad Youssef
Original assignee: Continental Autonomous Mobility Germany GmbH
Priority date: 2022-08-03
Filing date: 2023-07-25
Publication date: 2024-02-08
Also published as: DE102022208049A1

Abstract

The invention relates to a method for determining an item of parking information, that describes at least one parking space (9), using a sensor device (2) of a vehicle (1), the sensor device (2) having at least one environment sensor (3) and a computing device (4), said method comprising the steps: - using the at least one environment sensor (1) to capture measurement data that describes an environment of the vehicle (1), the measurement data being associated with one or more objects in the environment of the vehicle (1); - based on the measurement data, creating at least one environment map (5) that describes the environment of the vehicle (1), the measurement data being indicated in the environment map spatially in relation to the vehicle (1); - analysing the at least one environment map (5) using the computing device (4), the environment map (5) being fed to a convolutional neural network, the neural network classifying and localising parking spaces (9) that are present in the environment of the vehicle (1) based on the environment map (5), the neural network determining the arrangement of each of the parking spaces (9) relative to the vehicle (1) by means of regression, and - generating the parking information based on the classified and localised parking spaces (9) and their relative arrangement.

Description

202107831 1 Description Method for determining a parking space using a folding neural network The invention relates to a method for determining parking information describing at least one parking space using a sensor device of a vehicle, the sensor device having at least one environment sensor and a computing device. The invention further relates to a method for training a convolutional neural network, a sensor device, a vehicle, a computer program product and a computer-readable storage medium. Modern vehicles can use environmental sensors to obtain information about various objects in the vehicle's surroundings. These objects can be, for example, other road users such as moving or stationary third-party vehicles or immovable structures such as curbs, walls, markings or the like. The information obtained in this way can be used to provide different driver assistance functions. For example, further information regarding available parking spaces in the area surrounding the vehicle can be derived from an environment description based on such object information. For this purpose, algorithms are generally used which determine free parking spaces from the objects detected in the environment and/or their arrangement according to defined rules and dependencies. The rules are defined, for example, through search tables and/or through individual case queries or if conditions. This approach has the disadvantage that creating and checking these comparatively complex and extensive rules is time-consuming and that parking space detection is basically limited to the scenarios queried by the rules. Furthermore, further development of the rules and their adaptation to different vehicles and/or different requirements for parking space determination can only be implemented with great effort. 202107831 2 So-called deep learning methods, which provide for the evaluation of the image data using artificial intelligence, are known from other areas of technology for the machine evaluation of image data or for computer-based vision. An example of such a method is that in the article by Redmon, Joseph, et al. "You only look once: Unified, real-time object detection." Proceedings of the IEEE conference on computer vision and pattern recognition (2016) describes the “You-Only-Look-Once method” (also known as the YOLO method), which uses a grid in a single neural network to achieve simultaneous classification and localization of objects in images. The neural network determines class probabilities assigned to the objects as well as the frames surrounding the objects in the image and localizing the object. The frames each surround an individual classified object, with the edges of the frames running parallel to the edges of the rectangular, evaluated image. The invention is based on the object of specifying an improved method for determining a parking space, which in particular enables improved adaptability of parking space detection to different requirements as well as improved precision in parking space detection. This object is achieved according to the invention by a method of the type mentioned at the outset, the method comprising the following steps: ^ Acquiring measurement data describing an environment of the vehicle with the at least one environment sensor, the measurement data being assigned to one or more objects in the area surrounding the vehicle, ^ Creating at least one environment map describing the environment of the vehicle from the measurement data, the measurement data in the environment map being specified spatially in relation to the vehicle, ^ Evaluating the at least one environment map by the computing device, the environment map being fed to a convolutional neural network, where the neural network classifies and locates existing parking spaces in the area surrounding the vehicle from the environment map, whereby the 202107831 3 neural network uses regression to determine the relative arrangement of the parking spaces to the vehicle, and ^ generate the parking information from the classified and localized parking spaces and their relative arrangement. The method according to the invention is based on measurement data which are determined using one or more environmental sensors of a vehicle. The measurement data describes objects in the area surrounding the vehicle. The objects can in particular be third-party vehicles, infrastructure objects, buildings or structures as well as road and/or parking lot markings. In particular, the measurement data describes objects that can generally limit parking spaces. The measurement data can be recorded using one or more different types of environmental sensors. Depending on the type of environment sensor, the measurement data can, for example, be measuring points to which a distance to the vehicle is assigned. Such measuring points can be generated, for example, using ultrasonic sensors, radar sensors or lidar sensors. In the case of a sensor designed as a camera, the measurement data as an object can describe, for example, a road or parking lot marking and its distance or arrangement relative to the vehicle. An environment map is then created from the recorded measurement data, which in particular describes the current environment of the vehicle. For this purpose, the measurement data is specified spatially in relation to the vehicle, for example in a rectangular map in which the vehicle is in the center. The measurement data can be entered, for example, in a two-dimensional map, which depicts the plane extending along the vehicle's longitudinal direction and the vehicle's transverse direction. The environment map can be generated, for example, as an occupancy map. The environment map can preferably have a spatial grid into which the measurement data is entered or to which the measurement data is assigned. 202107831 4 In this way, the environment of the vehicle described by the measurement data can be mapped on the environment map. The invention is based on the knowledge that such an environment map can be viewed as an image or that deep learning methods used for image evaluation can also be applicable to such an environment map. Accordingly, according to the invention, a convolutional neural network (CNN) is used to evaluate the environment map. This is preferably stored in the computing device of the sensor device so that the environment map can be fed to it as input data. With the help of the neural network, the environment map is evaluated with regard to parking spaces in the area surrounding the vehicle or free parking spaces in the area surrounding the vehicle. To do this, the neural network classifies and locates parking spaces in the area surrounding the vehicle based on the surrounding map. In particular, only a single convolutional neural network is used for the evaluation, which carries out both the classification and the localization of the parking spaces. In other words, the neural network carries out an image evaluation process based on the YOLO method on the environment map. It is possible for the convolutional neural network to be followed by another neural network for evaluating the output data determined by the convolutional neural network. In addition, the neural network uses regression to determine the relative arrangement of the parking spaces to the vehicle. In other words, in comparison to the classic YOLO method described at the beginning, the localization of a detected parking space is not only indicated by inserting a rectangular frame that encloses the parking space and is always aligned along the edges of the image, but instead the relative position or .the relative position and the relative orientation of the parking space to the vehicle. In the case of a rectangular parking space, for example, an angle can be determined by which the parking space is 202107831 5 is rotated relative to the vehicle. The parking space can advantageously be localized taking into account the information required for a particularly assisted or at least partially automated parking process. The parking spaces can be classified in particular by assigning a classification probability to different sections of the surrounding map. In other words, the classification can be carried out by specifying a classification probability for different sub-areas of the environment map, the classification probability indicating the probability with which the sub-section is a parking space, ie a free parking space for the vehicle. Parking information is then formed from the parking spaces classified by the neural network and their relative arrangement. For example, the parking information can contain at least the parking spaces whose classification probability is above a predetermined limit value, with the relative arrangement to the vehicle also being specified for these parking spaces. The parking information can then be used to control an actuator of the vehicle and/or a display device of the vehicle. The actuator of the vehicle can, for example, be an actuator of the vehicle designed for transverse guidance and/or longitudinal guidance. The at least one actuator can be, for example, a drive motor of the vehicle and/or a steering drive of the vehicle. Depending on the parking information, for example, automated or partially automated control of a parking process of the vehicle can take place, in particular by controlling at least one actuator of the vehicle. The vehicle can move forward and/or backward during the parking process, make steering angle changes to a set steering angle between and/or during these movements and 202107831 6 in particular also carry out several movements or pulls in different directions. The parking process can in particular be carried out in such a way that the vehicle is positioned in at least one parking space described by the parking information. In addition or as an alternative to the actuator, at least one display device of the vehicle can also be controlled, so that, for example, a driver of the vehicle can be made aware of the parking spaces determined in the area surrounding the vehicle, for example by means of a corresponding pictorial representation. The display device can be, for example, a display arranged in an interior of the vehicle, a heads-up display, a virtual side mirror and/or the like. The representation on the display device can be used, for example, to enable the driver to select one of several identified parking spaces as the destination of a parking process. Additionally or alternatively, aids such as virtual lines and markings for moving the vehicle into a determined and/or selected parking space can be displayed on the display device, which enable the driver to control a movement of the vehicle into this parking space. The vehicle can be, for example, a motor vehicle such as a passenger car or a truck. Furthermore, the vehicle can also be a vehicle combination, which has, for example, a towing vehicle and one or more trailers. Alternatively, the method can also be used on other types of vehicles. The sensor device of the vehicle can have one or more environment sensors. The surrounding sensors are arranged on the vehicle so that they can record the immediate surroundings of the vehicle. The sensor device further comprises a computing device, which can be part of the vehicle. Alternatively, it is possible for the computing direction to be a computing direction external to the vehicle, which is connected, for example, to the at least one environment sensor via a particularly wireless one 202107831 7 Communication link communicated. Parking information determined by a vehicle-external computing device can also be transmitted back to the vehicle via the communication connection, so that the previously described control of the actuator and/or the display device of the vehicle can take place depending on the parking information. The creation of the environment map from the measurement data of the at least one environment sensor can also be done by the computing device. Alternatively, this step can also be carried out by a further computing device of the vehicle and/or a further computing device external to the vehicle, with the further computing device transmitting the environment map to the computing device. The neural network used to evaluate the environment map is a convolutional neural network (CNN), which is stored in the computing device or a storage unit of the computing device. The environment map is evaluated using artificial intelligence or a deep learning method. The folding neural network is set up or trained to classify and locate existing parking spaces in the area surrounding the vehicle from the map of the surroundings and to use regression to determine the relative arrangement of the parking spaces to the vehicle. In this way, the parking space can be determined based on the surrounding map using a process that can be adapted and improved using machine learning. This makes it possible to flexibly adapt the parking space determination to different scenarios by appropriately training the neural network. This advantageously provides an easy-to-implement option for adapting the parking space determination. For the purposes of the invention, the term “machine learning” (ML) refers to the use of algorithms for analyzing training data, learning from this training data and then determining or predicting an initially unknown property of input data to be evaluated. In 202107831 8 Both controlled learning and uncontrolled learning can be used in this context. Among other things, the labeling strategy or the labeling of training data can be crucial for its analysis. The term “deep learning” (DL) in the sense of the invention includes a cascade of several layers of nonlinear processing units (usually artificial neural networks) for the extraction and transformation of features (or for the extraction and transformation of parameters). The term "artificial neural network" (ANN) or "convolutional neural network" (CNN) includes networks of artificial neurons that are connected to each other in the abstract, as in a nervous system of a living being, in terms of information processing. The neurons can be represented as nodes and their connections as edges in a diagram. The backmost (node) layer of the network is called the output layer and the "invisible" (node) layers that are at the front are called hidden layers. Such artificial neural networks can be single-layer (one output layer), two-layer (one output layer and one hidden layer to improve abstraction) or multi-layer (at least one output layer and several hidden layers to improve abstraction). In addition, with regard to their data transmission, these can be designed to be forward-facing (feedforward) and/or with backward-facing edges (recurrent connections) (feedback; feedback network). For the purposes of the invention, the term “computer-implemented method” refers to a flowchart that can be implemented and carried out on the basis of a data processor. The data processor, such as a computer, a computer network, or other programmable device, can process data using programmable calculation rules. With regard to the method, essential properties can be realized, for example through a new program, new programs, algorithms or similar. 202107831 9 According to the invention, it can be provided that the environment map contains a map grid and the measurement data are assigned to the cells of the map grid, with for each cell the number of measurement points of the measurement data assigned to the cell, the height of an object described by the measurement data assigned to the cell, the Type of an object described by the measurement data assigned to the cell and / or at least one signal property is specified by the measurement data assigned to the cell. For example, for measurement data that comes from a type of sensor that evaluates reflected echoes of emitted signals, the number of echoes reflected from an object in the cell can be stored as signal properties. A sensor of this type can be, for example, an ultrasonic sensor, a radar sensor or a lidar sensor. The echoes can in particular each represent a measuring point. Furthermore, it is possible that information about the height of the object on which the echo was reflected can also be determined from the echoes or measuring points from such sensors. Height information describing this height of the object can also be assigned to the individual cells of the map grid of the surrounding map for each measuring point or averaged over several measuring points. For example, the intensity of the echoes or measuring points assigned to the cell can be stored in the environment map as a signal property. The intensity can be specified for the individual measuring points or averaged. Depending on the sensor type, further properties of the measurement data, in particular properties that enable conclusions to be drawn about an object type and/or a detection quality, can also be entered in the environment map. Furthermore, the type of an object described by the measurement data for the respective cell can also be specified in the environment map or assigned to the individual cells. The type of an object can be determined from the measurement data, for example from measurement data obtained via an environmental camera. In this way, objects that can be detected via ultrasound, radar 202107831 10 or lidar sensors cannot be detected or can only be detected with difficulty, such as road or parking lot markings, are entered in the surrounding map. This information is assigned in particular to those cells whose position corresponds to the object position determined via the sensor device. In particular, no measurement data can be assigned to cells of the environment map that correspond to a section of space in the area surrounding the vehicle that is not blocked by an object, so that they remain free. According to the invention, it can be provided that the neural network is set up to overlay the environment map with an evaluation grid comprising several cells, with each cell being assigned at least one frame, with a classification as a parking space and / or an orientation in the parking information for each of the frames a parking space described by the measurement data and assigned to the frame is specified relative to the vehicle. The frame, which can also be referred to as a box, has in particular a rectangular shape. It is possible for the frames of several cells of the evaluation grid to be positioned and/or aligned in the same way with respect to the cells. The frames each represent initial assumptions for possible parking spaces, to which a classification information, for example a classification probability, for classification as a parking space can be assigned by the neural network. This also enables at least a basic localization of possible parking spaces in the surrounding map. Furthermore, at least for frames classified as parking spaces, additional properties that describe the exact positioning and orientation of the parking space and/or their respective width and length can be determined by means of regression by the neural network. In a preferred embodiment of the invention, it can be provided that an evaluation grid is used in which each cell has several different ones 202107831 11 positioned and/or differently oriented frames are assigned. The frames can, for example, be two rectangular frames arranged rotated by 90° relative to one another. This simplifies the determination of the relative alignment of the parking space to the vehicle, since it can initially be assumed that the frame already corresponds more closely to a parking space described by the surrounding map or has a higher probability of classification. It is also possible that each cell of the evaluation rater contains more than two frames and/or that they are oriented in a different way to one another. The orientation of the parking space can be determined by the frames or localization and/or orientation parameters can be determined by several differently positioned and/or differently oriented frames. The more frames are used, the more precisely the localization and determination of the relative arrangement of the parking space in relation to the vehicle can be carried out. According to the invention, it can be provided that the parking information for at least some of the classified parking spaces additionally contains position information, the position information describing a target position for the vehicle that is assigned to the respective parking space and lies within the parking space. The position information can be used to take into account that the parking space is generally larger, that is, it offers more parking space than is occupied by the vehicle after parking. The target position can therefore in particular lie completely within the parking space and, for example, maintain a predetermined safety distance from the edges or edges of the parking space. The target position can also be described, for example, by a rectangle which has a shorter edge length than the parking space and/or is rotated at an angle relative to the parking space. The position information can also advantageously be determined using the convolutional neural network, in particular also using regression. In a preferred embodiment of the invention it can be provided that the classification of the parking space is a classification as a perpendicular parking space and as 202107831 12 parallel parking space included. In addition to evaluating the environment map, the determination of the relative arrangement of a parking space to the vehicle can also be advantageously used to classify a parking space as a perpendicular parking space and/or as a parallel parking space. A parallel parking space describes a parking space in which you can park alongside the direction of travel of a street on which the vehicle is located. Accordingly, a transverse parking space refers to a parking space in which the vehicle can be parked perpendicular to the direction of travel of a street on which the vehicle is located. A distinction between a perpendicular parking space and a parallel parking space can be made, for example, by providing two rectangular frames rotated by 90° relative to each other in the evaluation grid. In a preferred embodiment of the invention, it can be provided that a sensor device is used which comprises at least one ultrasonic sensor, at least one lidar sensor, at least one radar sensor and/or at least one surroundings camera as at least one environment sensor. The sensor device can in particular comprise several of the aforementioned sensor types and/or combinations of the aforementioned sensor types. The respective measurement data from different sensors can in particular be entered or merged into the same environment map. Furthermore, the invention relates to a method for training a folding neural network, comprising the steps: - Providing at least one training data set comprising several training environment maps each describing the environment of the vehicle from training measurement data, the training measurement data being spatially in the training environment map Reference to a vehicle are specified, - assigning a ground truth describing at least one parking space in the area surrounding the vehicle to each of the training environment maps, the ground truth classifying and localizing at least a section of the training environment map as a parking space, the ground truth being the relative orientation of the indicates parking space for the vehicle, 202107831 13 - Overlaying the training environment maps each with an evaluation grid comprising several cells, with at least one frame being assigned to each cell of the evaluation grid, - determining the respective frame with the largest overlap to the at least one parking space described by the ground truth, and - optimizing of the neural network with regard to a match in the relative arrangement between the respective frame with the largest overlap and the parking space described by the ground truth. The method according to the invention for training a folding neural network can be used in particular for training a neural network used in a method according to the invention for determining parking information describing at least one parking space. Through the training, the internal parameters of the neural network can be adjusted so that it can be set up or trained for the evaluation of the environment maps created from the measurement data according to the method according to the invention for determining parking information describing at least one parking space. The creation of the training environment maps from the training measurement data is carried out in particular analogous to the creation of the environment maps from the measurement data, as described above. The training measurement data are indicated in the training environment map in relation to the position of a vehicle. The training measurement data can, for example, have been determined at an earlier point in time by means of a sensor device of a vehicle comprising at least one environment sensor and can then be used for training. Alternatively, training measurement data created and/or calculated in a different way can also be used. A ground truth is then assigned to each of the training environment maps, with the ground truth describing one or more parking spaces in the environment map. In addition, the ground truth also gives the location for locating the parking space as well as the relative orientation of the parking space to the 202107831 14 vehicle described in the area map. The ground truth represents the result to which the evaluation of the neural network is optimized during training. The training environment maps are then overlaid during training with an evaluation grid comprising several cells, which can also be referred to as a training evaluation grid in relation to the training of the neural network. At least one frame or at least one box is assigned to the cells of the evaluation grid. These represent initial assumptions for possible parking spaces. The frame that has the largest overlap with a parking space described by the ground truth is then selected from the majority of frames in the evaluation grid. In particular, the frame with the largest overlap is determined for each parking space described in the ground truth. The overlap can also be referred to as Intersection Over Union (IoU) and indicates the proportion to which each of the frames in the evaluation grid overlaps with one of the parking spaces described by the ground truth. After determining the one or more frames with the largest overlap to one or more parking spaces described by the ground truth, the neural network is optimized with regard to a match in the relative arrangement between the respective frame with the largest overlap and the one described by the ground truth. parking space assigned to the frame. In this way, the neural network can be trained efficiently for the recognition of parking spaces in maps of the surrounding area, in particular for use in a method according to the invention for determining parking information describing at least one parking space. According to the invention, it can be provided that, in order to determine the overlap, it is determined in which cell the center of the at least one parking space described by the ground truth lies, the frame of this 202107831 15 cell is chosen as the frame with the largest overlap, whose area-related overlap with the parking space described by the ground truth is largest. If it is first determined in which cell of the evaluation grid the center of the parking space or a parking space described by the ground truth is located, the number of frames for which the overlap is determined can be reduced, so that fewer calculation steps are required and thus more efficient training is possible is. In a preferred embodiment of the invention, it can be provided that an evaluation grid is used in which several differently positioned and/or differently oriented frames are assigned to each cell and/or that the classification of the parking space includes a classification as a perpendicular parking space and as a parallel parking space. Analogous to the determination of the parking information describing at least one parking space, several frames per cell of the evaluation grid can also be used during training. These frames can in particular be rectangular and arranged at different positions and/or with different orientations to the cards of a rectangular cell of the evaluation grid. These frames can be used, for example, to make a distinction between a perpendicular parking space and a parallel parking space and thus also to train the neural network with regard to a corresponding classification or distinction. According to the invention, it can be provided that the ground truth for the at least one parking space described by the ground truth additionally contains position information, the position information describing a target position for the vehicle assigned to the respective parking space and lying within the parking space, the optimization of the neural network also with regard to Position information is provided. In this way, it can be achieved that the neural network is also trained with regard to position information located within the parking space, so that this can also be determined by the neural network. 202107831 16 The invention further relates to a sensor device comprising at least one environment sensor and a computing device, wherein the computing device is set up to carry out a method according to the invention for determining parking information describing at least one parking space. The invention further relates to a vehicle comprising at least one sensor device according to the invention. The invention also relates to a computer program product comprising instructions which cause a computing device to carry out a method according to the invention for determining parking information describing at least one parking space. In addition, the invention relates to a computer-readable storage medium comprising a computer program product according to the invention. The computer-readable storage medium can be, for example, a non-transient data carrier, for example a CD, a DVD, a floppy disk, a flash memory or the like. All of the advantages and refinements described above for the method according to the invention for determining parking information describing at least one parking space also apply accordingly to the method according to the invention for training a convolutional neural network and vice versa. Furthermore, all advantages and refinements described in relation to one of the methods according to the invention also apply correspondingly to the sensor device according to the invention, the vehicle according to the invention, the computer program product according to the invention and the computer-readable storage medium according to the invention and vice versa. 202107831 17 Further advantages and details of the invention emerge from the exemplary embodiments described below and from the drawings. These are schematic representations and show: an environment map superimposed with an evaluation grid to explain the exemplary embodiment of a method according to the invention for determining parking information describing at least one parking space and an exemplary embodiment for training a neural network. An exemplary embodiment of a vehicle 1 according to the invention is shown in FIG. The vehicle 1 can be, for example, a motor vehicle such as a passenger car or a truck. Furthermore, the vehicle 1 can also be a vehicle combination, which has, for example, a towing vehicle and one or more trailers. Alternatively, the method can also be used on other types of vehicles. The vehicle 1 includes a sensor device 2, which has several environment sensors 3 and a computing device 4. The environment sensors 3 of the sensor device 2 are designed, for example, as ultrasonic sensors. Additionally or alternatively, at least some of the surroundings sensors 3 can also be designed as a surroundings camera, as a radar sensor and/or as a lidar sensor. The computer device 4 of the sensor device 2 is shown as part of the vehicle 1 in the present exemplary embodiment. It is possible that the computing device 4 is integrated into one of the environment sensors 3 or that it 202107831 18 is a computing device external to the vehicle. A vehicle-external computing device can communicate with the environment sensors 3 and/or other devices of the vehicle 1 via a particularly wireless communication connection. Using the surrounding sensors 3, measurement data from the surroundings of the vehicle 1 can be determined. The measurement data describes objects that are in the vicinity of the vehicle 1. The objects can be, for example, other road users such as other vehicles or similar. Furthermore, immovable objects, such as infrastructure objects, parts of buildings or similar, can also be recorded as objects. The computing device 4 of the sensor device 2 is designed to carry out a method for determining parking information describing at least one parking space in the vicinity of the vehicle 1. Depending on the parking information, for example, an automated or partially automated control of a parking process of the vehicle 1 can take place, in particular by controlling at least one actuator (not shown) of the vehicle 1. The vehicle 1 can move forward and / or backward during the parking process, carry out steering movements and in particular, carry out several partial movements or moves. The parking process can in particular be carried out in such a way that the vehicle is positioned in at least one parking space described by the parking information. In addition or as an alternative to the actuator, a display device (not shown) of the vehicle 1 can also be controlled, so that, for example, a driver of the vehicle 1 can be made aware of the parking spaces determined in the area surrounding the vehicle, for example by means of a corresponding pictorial representation. The display device can be, for example, a display arranged in an interior of the vehicle 1, a heads-up display, a virtual side mirror or the like. The representation on the display device can be used, for example, to enable the driver to select one of several identified parking spaces as the destination of a parking operation. Additionally or alternatively 202107831 19, aids for moving the vehicle 1 into a determined and/or selected parking space can be displayed on the display device, which enable the driver to move the vehicle 1 into this parking space. To determine the parking information with the computing device 4 or to determine the parking information in a computer-implemented method, measurement data describing the surroundings or the surroundings of the vehicle 1 are first determined with one or more of the surroundings sensors 3, the measurement data being assigned to one or more objects in the surroundings of the Vehicle 1 are assigned. At least one environment map 5 describing the environment of the vehicle 1 is then created from the measurement data, the measurement data being indicated spatially in relation to the vehicle 1 in the environment map 5. An exemplary embodiment of such an environment map 5 is shown in FIG. The environment map 5 is a rectangular map in which the vehicle 1 is arranged in the middle. The measurement data are indicated spatially in relation to the vehicle 1 in the environment map 5. The measurement data are shown here as measurement points 6, which can be determined, for example, by environmental sensors 3 designed as ultrasonic sensors. The measuring points 6 of the measurement data each describe a signal echo that was reflected by an object in the vicinity of the vehicle 1. The individual measuring points 6 each include a distance measurement in relation to the vehicle 1, which can be carried out, for example, from a transit time measurement. In addition, a direction can also be determined for each of the measuring points 6, so that the reflection locations or the objects assigned to the measuring points 6 can be placed in spatial relation to the vehicle 1. The environment map 5 further comprises a map grid 7, with the measurement data being assigned to the individual cells 8 of the map grid 7. For each of the cells 8, for example, the number of measurement points 6 of the measurement data assigned to the cell 8 can be determined and stored in the environment map 5. Furthermore, the height of an object described by the measurement data assigned to the cell 8 and the type of such an object can also be added to the individual cells 8 202107831 20 and/or at least one further signal property of the measurement data assigned to the cell can be specified. The signal property can be, for example, the intensity of the individual measuring points 6. It is possible for a signal property such as the intensity to be specified, for example averaged for all measuring points of one of the cells 8. This can also be done accordingly with information regarding the height of an object and/or the type of an object. In addition to the measuring points 6 of environment sensors 2 designed as ultrasonic sensors, measurement points 6 of environment sensors 3 designed as radar sensors and/or lidar sensors can also be included in the environment map 5 be entered. Furthermore, objects determined via other types of sensors, for example surroundings sensors 3 designed as surroundings cameras, can also be entered in the surroundings map 5. In this way, objects such as road markings, parking lot markings or the like can also be entered in the environment map 5. The at least one environment map 5 is then evaluated by the computing device 4, the environment map 5 being fed to a folding neural network implemented by the computing device 4, the neural network classifying and localizing parking spaces 9 present in the area surrounding the vehicle 1 from the environment map 5, wherein the neural network uses regression to determine the relative arrangement of the parking spaces 9 to the vehicle 1. From the classified parking spaces 9 and their relative arrangement to the vehicle 1, the computing device 4 or another computing device (not shown) then generates the parking information describing at least one parking space 9 in the surroundings of the vehicle 1. Advantageously, the environment map 5 can be processed like a two-dimensional image file by the neural network. The neural network is set up to overlay the environment map 5 with an evaluation grid 11 comprising several cells 10, with each cell 10 202107831 21 at least one frame 12 is assigned. An environment map 5 superimposed with an evaluation grid 11 is shown in Fig.3. In the present case, each cell 10 comprises two rectangular frames 12, 13, which are arranged rotated by 90°. The frames 12, 13 are therefore oriented differently with respect to the cell 10. It is possible for the cells 10 to also be assigned additional frames (not shown), which, for example, have the same orientation as one of the frames 12, 13, but along one of the edges of the cell 10 in the x direction and/or in y direction are arranged offset. In the parking information, a classification as a parking space and/or an orientation of a parking space 9 described by the measurement data and assigned to the frame 12, 13 relative to the vehicle 1 is specified for each of the frames 12, 13 of all cells 10 of the evaluation grid 11. This makes it possible for the classification of the parking space 9 to include a classification as a perpendicular parking space and as a parallel parking space. A parking space 9 can be classified as such if a classification probability determined by the neural network is above a predetermined limit value. Accordingly, a parking space can be classified as a parallel parking space or as a perpendicular parking space if the corresponding classification probability assigned to one of the frames 12, 13 is above a predetermined limit value. The neural network additionally determines, at least for the frames 12, 13 classified as parking space 9, the relative arrangement of the parking space 9 to the vehicle 1 by means of regression. In this way, an exact localization of the parking space 9 in relation to the vehicle 1 is determined. Furthermore, different orientations of the parking spaces or different relative arrangements between vehicle 1 and a determined parking space 9 can be taken into account in this way. The parking information derived from the output of the neural network can also be used for at least some of the detected parking spaces 9 202107831 22 contain position information, wherein the position information describes a target position for the vehicle 1 that is assigned to the respective parking space 9 and lies within the parking space 9. The parking information generated can, for example, be a tensor which indicates for each of the frames 12, 13 whether it is a parking space 9 or not or whether it is a perpendicular parking space or a parallel parking space. Further information can be provided for each frame 12, 13 classified as a parking space 9. For each cell 10 of the evaluation grid 11, the parking information can contain, for example, a vector ^ _^^^^ , which can be used as

can be expressed. The values assigned to the frame 12 are marked, for example, with the index “1” and the values assigned to the frame 13, which are largely not shown for reasons of clarity, are marked with the index “2”. The individual values or components of the vector describe: P: an indicator (e.g. 0 or 1), whether the corresponding frame 12, 13 is a parking space 9, bx, by: the coordinates in the x direction and y- Direction of a center point of the determined parking space 9, for example as a percentage based on the size of the cell 10 starting from the lower left corner, bw, bh: the width and the height of the determined parking space 9, specified for example as a percentage based on the width and the height of the cell 10 of the evaluation grid 11, bθ: angle or rotation of the parking space 9 in relation to the cell 10, tx, ty: coordinates of a center point of a target position within the determined parking space 9, specified, for example, as a percentage in relation to the Cell 10 and relative to its center, 202107831 23 tθ: Angle of the target position, specified for example in relation to the determined parking space 9. Cpar, Cperp, Cvoid: The class probabilities for the presence of a parallel parking space (Cpar), for the presence of a perpendicular parking space (Cperp) and for the existence of no parking space (Cvoid). For example, the indicator P can be set to one if one of the probabilities Cpar or Cperp is above a predetermined limit, or set to zero if the probability Cvoid is above a predetermined limit. The information tx, ty and tθ form position information which describes the target position for the vehicle 1 that is assigned to the respective parking space and lies within the parking space. The parking information output, for example, in the form of a tensor has a dimensionality of ^ × ^ × (^ _^^^^ ) or ^ × ^ × (" ∗ $), where A and B are the number of rows and columns of the evaluation grid represent and their product thus indicates the number of cells in the evaluation grid and ^ _^^^^ is the vector assigned to each cell, which generally has " ∗ $ components, where " is the number of frames per cell and $ is the number of frames per frame in The information determined in the cell describes. In the above example for ^ _^^^^ is " = 2 and $ = 12. In an exemplary embodiment of a method for training the convolutional neural network, at least one training data set is first comprising several each describing the environment of the vehicle 1 Training environment maps provided from training measurement data. The measurement data in the training environment map are spatially specified in relation to a vehicle 1. The training method serves in particular to train the convolutional neural network implemented in the computing device 4 to carry out the exemplary embodiment described above. 202107831 24 Each of the training environment maps is assigned a ground truth describing at least one parking space 14 in the surroundings of the vehicle 1, the ground truth classifying and localizing at least a section of the training environment map as a parking space 14. In addition, the ground truth indicates the relative orientation of the parking space 14 to the vehicle. A parking space 14 specified by the ground truth is shown in FIG. The training environment maps basically correspond to the environment maps 5, as described above. Also during training, the training environment maps, as described above and as shown in FIG . The respective frame 12, 13 with the largest overlap to the at least one parking space 14 described by the ground truth is then determined. The neural network is then checked for a match in the relative arrangement between the respective frame 12, 13 with the largest overlap and the parking space 14 described by the ground truth is optimized. To determine the overlap, it is determined in which of the cells 10 of the evaluation grid 11 the center point 15 of the at least one parking space 14 described by the ground truth lies. In the example shown in Figure 3, this is cell 10, which is in the second row and the fifth column. Subsequently, the frame 12, 13 in this cell is selected as the frame with the largest overlap, whose overlap with the parking space 14 described by the ground truth is largest. To determine the respective overlap, a value for its area-related overlap or its Intersection Over Union (IoU) is determined for each of the frames 12, 13 in the cell in which the center 15 lies. 202107831 25 In the example shown in Fig. 3, the overlap with the vertically oriented frame 12 is larger, so that this is chosen as the goal of optimizing the neural network. Based on the choice of the frame 12, 13 and the arrangement of the vehicle 1 in the surrounding map, it can be determined or trained accordingly whether it is a parallel parking space or, as in the present case, a perpendicular parking space. It is possible for an evaluation grid 11 to be used in which more than two frames 12, 13 are assigned to each cell 10. The multiple frames 12, 13 can be positioned differently and/or oriented differently. Different dimensions, such as a different width, a different length, a different position of the center of the frame 12 and the parking space 14 with respect to the cell 10 as well as the relative orientation (angle θ) between the frame 12 and the parking space 14, which are determined by the neural network are also determined using regression and are also trained. The orientation of the parking space 14 can be approximated by the frames 12, 13 or, by using two or more frames 12, 13 per cell 10, differently positioned and/or differently oriented localization and/or orientation parameters can be initially specified. The more frames 12, 13 are used, the more precisely the neural network can localize and determine the relative arrangement of the parking space 14 in relation to the vehicle. Furthermore, the ground truth for the at least one parking space 14 can additionally contain position information, the position information describing a target position for the vehicle 1 that is assigned to the respective parking space 14 and lies within the parking space, the optimization of the neural network also taking place with regard to the position information. This can be done accordingly by training the determination of the parameters ^', ^^ and ^(. 202107831 26 Optimizing the neural network can be done using a loss function. The loss function can minimize error squares for all variables of the vector ^ _^^^^ or for all variables $, which should be able to be determined in relation to one of the frames 12, 13. Additionally or alternatively, a softmax function can also be used or only the errors of the class probabilities ^)*+, ^),+) and ^-.$/ and/or the classification probability ^ ₀ can be minimized. An example of a loss function ℓ(^2, ^) is ℓ(^2, ^) = (^2 _^ − ^ _^ ) ^{^} + (^2 _^ − ^ _^ ) ^{^} + ⋯ + (^2 ₀ − ^ ₀ ) ^{^} , 8,"" ^ _^ = 1 ℓ(^2, ^) = (^2 _^ − ^ _^ ) ^{^} , 8,"" ^ _^ = 0. If ^ _^ = 1, it means that the Frame 12 assigned to index “1” was classified as a parking space. In this case, the loss function is used to calculate the squared error of each variable $ ie each component ^ ₀ of the vector ^ _^^^^ . If ^ _^ = 0, this means that the frame 12 assigned to the index "1" was classified as "not a parking space", in which case only the squared error in the indicator or the classification probability ^ ₀ , i.e. the component ^ _^ of the vector ^ _^^^^ , is minimized.

Claims

202107831 27 Claims 1. Method for determining parking information describing at least one parking space (9) with a sensor device (2) of a vehicle (1), the sensor device (2) having at least one environment sensor (3) and a computing device (4), comprising the steps: ^ Acquiring measurement data describing an environment of the vehicle (1) with the at least one environment sensor (1), the measurement data being assigned to one or more objects in the environment of the vehicle (1), ^ Creating at least one of the surroundings of the vehicle ( 1) descriptive environment map (5) from the measurement data, the measurement data in the environment map being spatially specified in relation to the vehicle (1), ^ evaluating the at least one environment map (5) by the computing device (4), the environment map (5 ) is fed to a folding neural network, the neural network classifying and localizing existing parking spaces (9) from the environment map (5) in the vicinity of the vehicle (1), the neural network using regression to determine the relative arrangement of the parking spaces (9). the vehicle (1), and ^ generating the parking information from the classified and localized parking spaces (9) as well as their relative arrangement. 2. The method according to claim 1, characterized in that the environment map (5) contains a map grid (7) and the measurement data are assigned to the cells (8) of the map grid (7), with the number of cells for each cell (8). (8) assigned measuring points (6) of the measurement data, the height of an object described by the measurement data assigned to the cell (8), the type of an object described by the measurement data assigned to the cell (8) and / or at least one signal property from the cell ( 8) assigned measurement data is specified. 202107831 28 3. Method according to claim 1 or 2, characterized in that the neural network is set up to overlay the environment map (5) with an evaluation grid (11) comprising several cells (10), each cell (10) having at least one Frame (12, 13) is assigned, wherein in the parking information for each of the frames (12, 13) a classification as a parking space and / or an orientation of a parking space described by the measurement data and assigned to the frame (12, 13) relative to the vehicle (1) is specified. 4. The method according to claim 3, characterized in that an evaluation grid (11) is used, in which each cell (10) is assigned several differently positioned and / or differently oriented frames (12, 13). 5. Method according to one of the preceding claims, characterized in that the parking information for at least some of the classified parking spaces (9) additionally contains position information, the position information being a target position assigned to the respective parking space (9) and lying within the parking space (9). for the vehicle (1). 6. Method according to one of the preceding claims, characterized in that the classification of the parking space (9) includes a classification as a perpendicular parking space and as a parallel parking space. 7. The method according to any one of the preceding claims, characterized in that a sensor device (2) is used, which comprises at least one ultrasonic sensor, at least one lidar sensor, at least one radar sensor and / or at least one environment camera as at least one environment sensor (3). 202107831 29 8. Method for training a folding neural network, in particular for use in a method according to one of the preceding claims, comprising the steps: ^ Providing at least one training data set comprising several training environment maps from training, each describing the environment of a vehicle (1). -Measurement data, the training measurement data being specified in the training environment map spatially in relation to the vehicle (1), ^ assigning a ground truth describing at least one parking space (14) in the area surrounding the vehicle (1) to each of the training environment maps, wherein the ground truth classifies and localizes at least a section of the training environment map as a parking space (14), the ground truth indicating the relative orientation of the parking space (14) to the vehicle (1), ^ overlaying the training environment maps each with a plurality of cells (10) comprehensive evaluation grid (11), with each cell (10) of the evaluation grid (11) being assigned at least one frame (12, 13), ^ determining the respective frame (12, 13) with the greatest overlap to the at least one, parking space (14) described by the ground truth, ^ optimizing the neural network with regard to a match in the relative arrangement between the respective frame (12, 13) with the largest overlap and the parking space (14) described by the ground truth. 9. The method according to claim 8, characterized in that for determining the overlap it is determined in which cell (10) the center point (15) of the at least one parking space (14) described by the ground truth is located, the frame (12, 13) this cell (10) is selected as the frame (12, 13) with the largest overlap, the overlap of which with the parking space (14) described by the ground truth is largest. 202107831 30 10. Method according to claim 8 or 9, characterized in that an evaluation grid (11) is used, in which each cell (10) is assigned several differently positioned and/or differently oriented frames (12, 13) and/or that the classification of the parking space (14) described by the ground truth includes a classification as a perpendicular parking space and as a parallel parking space. 11. The method according to one of claims 8 to 10, characterized in that the ground truth for the at least one parking space (14) additionally contains position information, the position information being a target position assigned to the respective parking space (14) and lying within the parking space (14). for the vehicle (1), the optimization of the neural network also taking place with regard to the position information. 12. Sensor device comprising at least one environment sensor (3) and a computing device (4), the computing device (4) being set up to carry out a method according to one of claims 1 to 7. 13. Vehicle comprising at least one sensor device (2) according to claim 12. 14. Computer program product comprising instructions which cause a computing device (4) to carry out a method according to one of claims 1 to 7. 15. Computer-readable storage medium comprising a computer program product according to claim 14.