DE102021127748A1 - PROCEDURE, COMPUTER PROGRAM PRODUCT, PARKING ASSISTANCE SYSTEM AND VEHICLE - Google Patents

Abstract

A method for operating a parking assistance system (110) for a vehicle (100) is proposed. The method comprises: Receiving (S1) a first image (210) of an area (200) of the vehicle (100), which is recorded from a first recording position (201), and a second image (220) of the area (200) of the Vehicle (100), which is detected from a second recording position (202), determining (S2) an object (230, 240) in the received first image (210) and in the received second image (220), determining (S3) a respective bounding box (231A, 231B, 241A, 241B) encompassing the identified object (230, 240) in the first image and in the second image, the respective bounding box (231A, 231B, 241A, 241 B) having a respective first and a respective second vertical boundary line (232A, 232B, 233A, 233B, 242A, 242B, 243A, 243B), each of which defines a corner point (C1-1, C1-2, C2-1, C2-2) of the detected object (230 , 240) in the respective image (210, 220), and carrying out a triangulation of at least one corner point (C1-1, C1-2, C2-1, C2-2) of the determined object (230, 240) on the basis of a distance ( DIST) from the first recording position (201) to the second recording position (202) and the respective vertical boundary line (232A, 232B, 233A, 233B, 242A, 242B, 243A, 243B) which defines the corner point (C1-1, C1-2 , C2-1, C2-2).

Description

The present invention relates to a method, a computer program product, a parking assistance system and a vehicle.

Parking assistance systems are known which have an automated parking function. In order to find a suitable parking space, the vehicle to be parked drives past a row of parked vehicles, for example, and detects the vehicles in its vicinity. If a large gap is detected between two other parked vehicles, this gap can be recognized as a possible parking space. Such a system can be based on ultrasonic sensors, for example.

The problem here can be that ultrasonic sensors do not provide reliable detections in some situations and that the range of the ultrasonic sensors is limited. For this reason, additional camera images are evaluated, for example, in order to record the surroundings of the vehicle more precisely. However, the problem now arises of combining the data from the camera with the data from the ultrasonic sensors (sensor fusion). One possibility for this is to enter all measurement data in a two-dimensional or three-dimensional map, with distances to the objects contained in the camera images having to be determined on the basis of the camera images.

It is known to determine a distance to objects on the basis of a number of images that were recorded from different positions. This method is based on a triangulation of specific points in the images. One difficulty is finding matching object points in different images. This can be done using different methods.

A first method is based on the so-called optical flow, which can be determined on the basis of two images recorded one after the other for a specific scene. The two images essentially show the same scene, it being possible for individual elements of the image to be displaced due to the element's own movement and for the entire image to be displaced and rotated due to the camera's own movement. Based on optical flow measurements, a distance from the camera can be determined for each pixel. With this method, the images must have been captured by one camera at different times; it is not possible to evaluate images captured by different cameras in this way. However, the change in the image content must not exceed a certain level, since otherwise it is no longer possible to establish a correlation between the images. In particular, this means that the time between the images must not be too large.

In a second method, specific features are determined in a respective image. By tracking a particular feature in multiple images, it is possible to triangulate that feature to determine the feature's distance from the camera. Images from different cameras cannot be combined with this method either, since there is no guarantee that matching features will be recorded in the images.

A problem with these known methods is not only that the methods are limited to images from a specific camera, but also that tracking a feature over larger moving distances of the camera is not reliable, since features can have a different appearance due to the change in the viewing angle . In addition, it is not possible to reliably recognize a feature at a later point in time if it has been lost in the meantime. Furthermore, these known methods are computationally intensive and, for efficient execution, require hardware that is specifically adapted to the respective system, such as specific ASICs (application-specific integrated circuits). Therefore, these methods can only be updated to a very limited extent and cannot simply be transferred to other systems, such as another camera and/or another lens and the like, but must be adapted in each case in a laborious manner. This makes their implementation complicated and expensive.

Against this background, an object of the present invention is to improve the operation of a parking assistance system for a vehicle.

• Empfangen eines ersten Bildes einer Umgebung des Fahrzeugs, das von einer ersten Aufnahmeposition aus erfasst ist, und eines zweiten Bildes der Umgebung des Fahrzeugs, das von einer zweiten Aufnahmeposition aus erfasst ist,
• Ermitteln eines Objekts in dem empfangenen ersten Bild und in dem empfangenen zweiten Bild,
• Bestimmen eines jeweiligen, das ermittelte Objekt umfassenden Begrenzungsrahmens in dem ersten Bild und in dem zweiten Bild, wobei der jeweilige Begrenzungsrahmen eine jeweilige erste und eine jeweilige zweite vertikale Begrenzungslinie umfasst, die jeweils einen Eckpunkt des ermittelten Objekts in dem jeweiligen Bild kennzeichnet, und
• Durchführen einer Triangulation zumindest eines Eckpunkts des ermittelten Objekts auf Basis eines Abstands von der ersten Aufnahmeposition zu der zweiten Aufnahmeposition und der jeweiligen vertikalen Begrenzungslinie, die den Eckpunkt in dem ersten Bild und in dem zweiten Bild kennzeichnet, zum Ermitteln einer relativen Position des Eckpunkts zu der ersten Aufnahmeposition und der zweiten Aufnahmeposition.

According to a first aspect, a method for operating a parking assistance system for a vehicle is proposed. The procedure includes:

• receiving a first image of an area surrounding the vehicle, which is captured from a first recording position, and a second image of the area surrounding the vehicle, which is recorded from a second recording position,
• detecting an object in the received first image and in the received second image,
• determining a respective bounding box comprising the identified object in the first image and in the second image, the respective bounding box comprising a respective first and a respective second vertical boundary line, each of which characterizes a corner point of the identified object in the respective image, and
• Carrying out a triangulation of at least one corner point of the determined object on the basis of a distance from the first recording position to the second recording position and the respective vertical boundary line, which characterizes the corner point in the first image and in the second image, to determine a relative position of the corner point to the first pick-up position and the second pick-up position.

This method has the advantage that a triangulation can be carried out at least for the two corner points or lateral boundaries of the object that are visible from a respective location, so that their respective relative position is determined, even if the recording positions and/or times of the Image acquisition are significantly different. The known methods, which are based on the optical flow or on feature triangulation, cannot be used, or cannot be used reliably, in this situation. The method is based on the finding that the vertical boundary line of a bounding box essentially always marks a corner or lateral edge of the object, even for images that were captured from different locations, and is therefore a suitable feature for triangulating the corner of the object to perform.

In particular, the images are recorded by a camera or by a plurality of cameras of the vehicle.

The object in the respective received image can be determined by means of an object recognition routine, in particular an artificial intelligence trained for object recognition. However, the object recognition can also be carried out using classic algorithms. The object recognition can be preceded by image processing, such as an image transformation of the respective received image, in order to improve the object recognition. The received images are preferably rectified, so that vertical and horizontal lines that are actually present in the respective image also run vertically and horizontally. Such rectification can be carried out by the camera that captures the respective image and/or can be carried out subsequently by the parking assistance system.

The object determined in the images is the same object present in the area surrounding the vehicle, such as a parked vehicle. An identifier is assigned to the determined object, for example, the same identifier being assigned to the same object in different images. You can also say that the object is tracked across the images. This ensures that the triangulation refers to vertices of the same object.

The bounding box is, for example, a rectangle that is drawn in the respective image in such a way that the determined object is contained in the rectangle. The bounding box preferably delimits the determined object, which means that, for example, an edge of the object borders the bounding box.

The corner point of the determined object can also be referred to as the lateral edge or lateral boundary of the object. Each bounding box includes two vertical bounding lines. It should be noted that "vertical" specifically refers to an actual vertical direction, that is, parallel to a force of gravity. The vertical line may appear curved unless the underlying image is rectified. For example, the first vertical boundary line is such that it touches a front right corner of a vehicle (object), and the second vertical boundary line is such that it touches a rear left corner of the vehicle (object). Which corner of the vehicle (object) corresponds to a respective boundary line depends in particular on a perspective from which the image was captured, ie the relative arrangement of vehicle (object) and camera.

For example, the received images include a pixel matrix, with each pixel corresponding to a specific direction, starting from a front lens of the camera. An angle enclosed by the respective directions can therefore be determined for each two image points. A direction can be viewed as a straight line in space, with the two straight lines corresponding to the two directions spanning a plane in which the angle is measured. Two images of a traffic situation, which are captured from different positions, show objects of the traffic situation at different positions in the image, ie at different pixels in the pixel matrix. An angle can be determined on the basis of the two image points which show a respective object, as explained above. A triangulation can be carried out on the basis of this angle be performed if the distance between the recording positions is known. A distance from the object to the camera can thus be determined.

• Eintragen der ersten Aufnahmeposition und der zweiten Aufnahmeposition in eine zweidimensionale Karte der Umgebung des Fahrzeugs,
• Eintragen einer jeweiligen Linie, welche sich ausgehend von der jeweiligen Aufnahmeposition in die Richtung der dem bestimmten Eckpunkt entsprechenden vertikalen Linie erstreckt, und
• Ermitteln eines Winkels zwischen den beiden Linien.

The triangulation can be carried out graphically, for example. For example, for a given vertex of the object, this may involve the following steps:

• Entering the first recording position and the second recording position in a two-dimensional map of the area around the vehicle,
• entering a respective line which, starting from the respective recording position, extends in the direction of the vertical line corresponding to the determined vertex, and
• Finding an angle between the two lines.

Applying trigonometric principles and using the distance between the two recording positions as a scale, the respective distance of the corner point from the recording positions can be determined. This determines the relative position of the corner point.

• Ermitteln einer maßstabsgerechten zweidimensionalen Karte der Umgebung auf Basis der ermittelten relativen Position des Eckpunkts und der ersten Aufnahmeposition und der zweiten Aufnahmeposition.

According to one embodiment of the method, this comprises:

• Determining a true-to-scale two-dimensional map of the surroundings based on the determined relative position of the corner point and the first recording position and the second recording position.

The two-dimensional map is based, for example, on a stationary coordinate system provided by the parking assistance system. The coordinate system is re-initialized, for example, each time you start driving. The coordinate system may also refer to an externally provided coordinate system such as a world coordinate system of a satellite positioning system such as GPS or the like. In the two-dimensional map, each object has specific coordinates. If the vehicle of the parking assistance system moves, it changes its position in the coordinate system.

The true-to-scale, two-dimensional map preferably includes all of the determined objects with the correct position and extent, so that distances can be measured on the basis of the map.

When determining the two-dimensional map, for example, the determined corner points are entered into the map according to the determined relative position. The area between two corner points of an object can be identified as an object area, it being noted that the course of the object in the object area is not determined on the basis of the method described above. The profile in the object area can be “filled in” based on ultrasound measurements, for example, as explained below.

• Empfangen eines Ultraschall-Sensorsignals, welches indikativ für die Umgebung des Fahrzeugs ist,
• Ermitteln einer Position eines Objektpunkts, der ein Punkt eines Objekts ist, der ein abgestrahltes Ultraschall-Signal reflektiert, auf Basis des empfangenen Ultraschall-Sensorsignals, und
• Eintragen des ermittelten Objektpunkts in die ermittelte maßstabsgerechte zweidimensionale Karte der Umgebung zum Bereitstellen einer vervollständigten Karte.

According to a further embodiment of the method, this includes:

• Receiving an ultrasonic sensor signal, which is indicative of the surroundings of the vehicle,
• determining a position of an object point, which is a point of an object reflecting a radiated ultrasonic signal, based on the received ultrasonic sensor signal, and
• Entering the determined object point in the determined true-to-scale two-dimensional map of the environment to provide a completed map.

In this embodiment, information derived from image data and information derived from ultrasound measurements are thus combined in a common map. This can also be referred to as sensor fusion.

• Ermitteln eines Objektbereichs in der vervollständigten Karte auf Basis der bestimmten Begrenzungsrahmen, wobei der Objektbereich den Bereich zwischen zwei Eckpunkten des Objekts umfasst, und
• Ermitteln einer Objekt-Begrenzungslinie in der vervollständigten Karte auf Basis des ermittelten Objektbereichs, der Eckpunkte und von in dem Objektbereich liegenden Objektpunkten.

According to a further embodiment of the method, this includes:

• determining an object area in the completed map based on the determined bounding box, the object area comprising the area between two vertices of the object, and
• determining an object boundary line in the completed map based on the determined object area, the corner points and object points lying in the object area.

This geometric method makes it possible to reliably determine a course of the object. Knowledge or empirical values can be used here. For example, vehicles are known to have a substantially rectangular horizontal cross-section. Therefore, a vehicle is characterized sufficiently well by specifying the positions of four corner points (on the basis of symmetry considerations, even three corner points are sufficient) so that autonomous driving maneuvers can be carried out in the vicinity of the vehicle while avoiding a collision. Based on the triangulation, the positions of two vertices are known, and based on the ultrasonic measurements, the position of the intervening vertex is known.

• Ermitteln eines Parkplatzes auf Basis der ermittelten Objekt-Begrenzungslinie.

According to a further embodiment of the method, this includes:

• Determination of a parking space based on the determined object boundary line.

The parking space is determined, for example, in such a way that a predetermined safety distance from the object boundary line is maintained.

• Durchführen eines automatisierten Einparkmanövers auf Basis des ermittelten Parkplatzes.

According to a further embodiment of the method, this includes:

• Carrying out an automated parking maneuver based on the determined parking space.

According to a further embodiment of the method, the first and the second image of the surroundings are captured and provided by a number of on-board cameras.

For example, the first image is captured and provided by a front camera and the second image by a side camera. A relative position of the two cameras on the vehicle, in particular their distance from one another, is known, which is why their captured images can be related to one another, so that the previously explained triangulation of the vertical boundary line can be carried out. If a respective imaging scale and/or a resolution of the different cameras are different, this must be taken into account accordingly in the triangulation.

According to a further embodiment of the method, the first and the second image of the surroundings are captured and provided by the same on-board camera at different times, a driving condition sensor signal also being received, and the distance between the first recording position and the second recording position being based on the received Driving condition sensor signal is determined.

The driving state sensor signal includes, in particular, odometry data such as a wheel speed sensor signal, a wheel angle sensor signal and/or a steering angle sensor signal.

According to a further embodiment of the method, the object is determined in the received first and second image and the respective boundary frame is determined by an artificial intelligence trained thereon.

The artificial intelligence is designed, for example, as a neural network. An example of a structure of the neural network is as follows. The neural network can be divided into three layers or sections, an input layer, one or more hidden layers and an output layer. Each of the layers includes a plurality of individual neurons, it also being possible for different layers to have a different number of neurons. The amount of neurons in the input layer depends, for example, on the amount and format of the data to be processed by the neural network, which can also be referred to as the dimensionality of the data. For example, the amount of neurons in the output layer depends on the dimensionality of the output vector. A neural network that is set up for classifying objects has, for example, a number of neurons in the output layer that corresponds to the number of classes.

The input layer forms the input of the neural network, to which the input vector is fed. The neurons of the input layer accept the information contained in the input vector and pass it on weighted to the neurons of the first hidden layer. The neurons of the first hidden layer receive the weighted signals from the input layer and pass them on to the following hidden layer in a weighted manner. The signals are passed through the neural network from layer to layer in this way until the output layer is reached. The individual weights with which the neurons transmit signals encode in particular the "knowledge" of the neural network. By training the neural network, for example with a training data set and using a suitable training method, the weights are adjusted before the neural network goes into operation. The output vector is tapped off at the neurons of the output layer. The amount of hidden layers is basically unlimited, however the computational power needed to operate the neural network scales with the amount of hidden layers. The more layers a neural network includes, the more complex tasks the neural network can solve, but the training effort, i.e. the size of the training data set, increases accordingly. It is therefore advantageous to use only as many hidden layers as are necessary for the task at hand.

According to a second aspect, a computer program product is proposed which comprises instructions which, when the program is executed by a computer, cause the latter to execute the method according to the first aspect.

A computer program product, such as a computer program means, can be used, for example, as a storage medium such as a memory card, USB Stick, CD-ROM, DVD, or in the form of a downloadable file provided or supplied by a server in a network. This can be done, for example, in a wireless communication network by transferring a corresponding file with the computer program product or the computer program means.

The computer is designed in particular as a parking assistance system for a vehicle.

• eine Empfangseinheit zum Empfangen eines ersten Bildes einer Umgebung des Fahrzeugs, das von einer ersten Aufnahmeposition aus erfasst ist, und eines zweiten Bildes der Umgebung des Fahrzeugs, das von einer zweiten Aufnahmeposition aus erfasst ist,
• eine Ermittlungseinheit zum Ermitteln eines Objekts in dem empfangenen ersten Bild und in dem empfangenen zweiten Bild,
• eine Bestimmungseinheit zum Bestimmen eines jeweiligen, das ermittelte Objekt umfassenden Begrenzungsrahmens in dem ersten Bild und in dem zweiten Bild, wobei der jeweilige Begrenzungsrahmen eine jeweilige erste und eine jeweilige zweite vertikale Begrenzungslinie umfasst, die jeweils einen Eckpunkt des ermittelten Objekts in dem jeweiligen Bild kennzeichnet, und
• eine Triangulationseinheit zum Durchführen einer Triangulation zumindest eines Eckpunkts des ermittelten Objekts auf Basis eines Abstands von der ersten Aufnahmeposition zu der zweiten Aufnahmeposition und der jeweiligen vertikalen Begrenzungslinie, die den Eckpunkt in dem ersten Bild und in dem zweiten Bild kennzeichnet, zum Ermitteln einer relativen Position des Eckpunkts von der ersten Aufnahmeposition und der zweiten Aufnahmeposition.

According to a third aspect, a parking assistance system for a vehicle is proposed. The parking assistance system has:

• a receiving unit for receiving a first image of an area surrounding the vehicle, which is captured from a first recording position, and a second image of the area surrounding the vehicle, which is recorded from a second recording position,
• a determination unit for determining an object in the received first image and in the received second image,
• a determination unit for determining a respective bounding box comprising the determined object in the first image and in the second image, wherein the respective bounding box comprises a respective first and a respective second vertical boundary line, each of which characterizes a corner point of the determined object in the respective image, and
• a triangulation unit for carrying out a triangulation of at least one corner point of the determined object on the basis of a distance from the first recording position to the second recording position and the respective vertical boundary line, which characterizes the corner point in the first image and in the second image, to determine a relative position of the Vertex of the first recording position and the second recording position.

This parking assistance system has the same advantages as explained for the method according to the first aspect. The embodiments and features described for the proposed method apply accordingly to the proposed parking assistance system.

The respective unit of the parking assistance system can be implemented in terms of hardware and/or software. In the case of a hardware implementation, the respective unit can be embodied, for example, as a computer or as a microprocessor. In the case of a software implementation, the respective unit can be designed as a computer program product, as a function, as a routine, as an algorithm, as part of a program code or as an executable object. Furthermore, each of the units mentioned here can also be designed as part of a higher-level control system of the vehicle, such as a central electronic control device and/or an engine control unit (ECU: Engine Control Unit).]

According to a fourth aspect, a vehicle is proposed with at least one camera for capturing and outputting an image of the surroundings of the vehicle, and with a parking assistance system according to the third aspect.

The vehicle is, for example, a passenger car or a truck. The vehicle preferably includes a number of sensor units that are set up to detect the driving state of the vehicle and to detect an environment of the vehicle. Examples of such sensor units of the vehicle are image recording devices such as a camera, radar (radio detection and ranging) or also a lidar (engl. light detection and ranging), ultrasonic sensors, location sensors, wheel angle sensors and/or wheel speed sensors. The sensor units are each set up to output a sensor signal, for example to the parking assistance system, which can be set up to carry out partially autonomous or fully autonomous control of the vehicle on the basis of the received sensor signals.

Further possible implementations of the invention also include combinations of features or embodiments described above or below with regard to the exemplary embodiments that are not explicitly mentioned. The person skilled in the art will also add individual aspects as improvements or additions to the respective basic form of the invention.

• 1 zeigt ein schematisches Beispiel für ein Fahrzeug aus einer Vogelperspektive;
• 2 zeigt eine schematische Ansicht einer Verkehrssituation aus einer Vogelperspektive;
• 3 zeigt ein schematisches Beispiel dafür, wie auf Basis zweier Bilder einer Verkehrssituation eine Triangulation zum Ermitteln einer zweidimensionalen Karte durchführbar ist;
• 4 zeigt ein schematisches Beispiel für eine zweidimensionale Karte;
• 5 zeigt ein schematisches Beispiel für eine Sensorfusion;
• 6 zeigt ein schematisches Beispiel für das das Ermitteln einer Objekt-Begrenzungslinie und das Ermitteln eines Parkplatzes auf Basis einer zweidimensionalen Karte;
• 7 zeigt ein schematisches Blockdiagramm eines Beispiels für ein Parkassistenzsystem; und
• 8 zeigt ein schematisches Blockschaltbild eines Beispiels für ein Verfahren zum Betreiben eines Parkassistenzsystems.

Further advantageous refinements and aspects of the invention are the subject matter of the dependent claims and of the exemplary embodiments of the invention described below. The invention is explained in more detail below on the basis of preferred embodiments with reference to the enclosed figures.

• 1 shows a schematic example of a vehicle from a bird's eye view;
• 2 shows a schematic view of a traffic situation from a bird's eye view;
• 3 shows a schematic example of how a triangulation for determining a two-dimensional map can be carried out on the basis of two images of a traffic situation;
• 4 shows a schematic example of a two-dimensional map;
• 5 shows a schematic example of a sensor fusion;
• 6 shows a schematic example for determining an object boundary line and determining a parking space on the basis of a two-dimensional map;
• 7 shows a schematic block diagram of an example of a parking assistance system; and
• 8th shows a schematic block diagram of an example of a method for operating a parking assistance system.

Elements that are the same or have the same function have been provided with the same reference symbols in the figures, unless otherwise stated.

1 10 shows a schematic example of a vehicle 100 from a bird's eye view. The vehicle 100 is, for example, a car that is arranged in an environment 200 . Car 100 has a parking assistance system 110, which is embodied as a control unit, for example. In addition, a plurality of sensor units 120, 130 are arranged on the car 100, these being optical sensors 120 and ultrasonic sensors 130, for example. The optical sensors 120 include, for example, visual cameras, a radar and/or a lidar. The optical sensors 120 can each capture an image of a respective area from the environment 200 of the car 100 and output it as an optical sensor signal. The four optical sensors 120 can jointly form, in particular, an all-round view camera (surround-view camera), individual images from the various cameras 120 being combined to form an overall image which shows an all-round view around vehicle 100 . The ultrasonic sensors 130 are for detecting a distance to objects 230, 240 arranged in the environment 200 (see 2 ) and set up to output a corresponding sensor signal. Using the sensor signals detected by the sensors 120, 130, the parking assistance system 110 is able to control the car 100 semi-autonomously or fully autonomously, for example to carry out a parking process. Except the ones in the 1 illustrated optical sensors 120 and ultrasonic sensors 130 can be provided that the vehicle 100 has various other sensor units 120, 130. Examples of this are a wheel speed sensor, a wheel angle sensor, a steering angle sensor, an acceleration sensor, a position sensor, an antenna with a coupled receiver for receiving electromagnetically transmittable data signals, and the like.

The parking assistance system 110 is, for example, as shown in FIG 7 explained formed and is set up to the basis of 8th carry out the procedures explained. A concrete example of a corresponding method is based on the 2-6 explained.

2 shows a schematic view of a traffic situation from a bird's eye view. Two objects 230, 240, such as parked vehicles, are located next to vehicle 100, which is, for example, vehicle 100 of FIG 1 acts. There is a free parking space 270 between the two parked vehicles 230, 240.

Conventional parking aids, which are based only on ultrasound, for example, cannot reliably detect how deep the parking space 270 is and whether the vehicles 230, 240 are properly parked, as shown here, or whether they may be at an angle and protrude into the parking space 270. Based on the proposed method, which is based on this exemplary traffic situation and based on the 3 - 6 is explained, these problems can be overcome.

3 shows a schematic example of how on the basis of two images 210, 220 in FIG 2 illustrated traffic situation a triangulation can be carried out. The two images 210, 220 were taken from one on the vehicle 100 (see 1 or 2 ) arranged camera 130 (see 1 ) recorded, for example, a camera arranged in the area of the exterior mirror, which records an area to the side of vehicle 100 . The images 210, 220 were captured at different points in time, with the vehicle 100 having moved in the meantime. This means that the camera was in different positions when capturing a particular image. The images 210, 220 therefore have a different perspective, which is reflected in a different appearance of the objects 230, 240 depicted. In this example, the images 210, 220 are undistorted and correctly oriented, so that in reality vertical/horizontal lines run vertically/horizontally in the image and the bottom edge of the respective image 210, 220 is aligned according to the direction of gravity.

The objects 230, 240 that can be seen in the respective image 210, 220 are determined. This takes place, for example, by means of image-based object recognition, in which case artificial intelligence can be used.

A bounding box 231A, 241A, 231B, 241B is assigned to each determined object 230, 240. For example, all pixels associated with a respective object 230, 240, and then defines a rectangle whose sides are vertical and horizontal. A left side of the rectangle runs perpendicularly through the pixel of the object whose horizontal position has the smallest value of all pixels of the object (relative to an origin in the upper left corner of the image). This is equivalent to the pixel of the object that has the smallest distance from the left side of the image of all pixels of the object. The other sides of the rectangle can be determined accordingly. The respective bounding box 231A, 241A, 231B, 241B thus includes the image area of the respective image 210, 220 in which the respective object 230, 240 lies.

The two respective vertically running sides 232A, 233A, 242A, 243A, 232B, 233B, 242B, 243B each identify a specific corner point C1-1, C1-2, C2-1, C2-2 of the object 230, 240, as in Figures 210, 220 can be seen. These lines 232A, 233A, 242A, 243A, 232B, 233B, 242B, 243B are therefore suitable as features on the basis of which a triangulation can be carried out about the relative position of the vertices C1-1, C1-2, C2-1, C2 -2 to determine.

Carrying out the triangulation is shown schematically using the two-dimensional map MAP. The two-dimensional map MAP is determined as follows. First, the two recording positions 201, 202 from which the images 210, 220 were recorded are entered as a basis. The distance DIST between the two positions 201, 202 is known, for example, on the basis of an odometry. A respective line 232A, 233A, 242A, 243A, 232B, 233B, 242B, 243B corresponds to a specific horizontal viewing angle as seen from the camera. A horizontal view angle corresponds to a line V1, V2 in the two-dimensional map MAP. Now, to determine the relative position of a particular vertex C1-1, C1-2, C2-1, C2-2, in this example the front left vertex C2-2 of the second object 240, the respective line V1, V2 according to the from angles W1, W2 determined in the respective image 210, 220 are entered. Viewed graphically, the intersection of the lines V1, V2 gives the position of the vertex C2-2 of the object 240 in the two-dimensional map MAP. The scale of the two-dimensional map MAP results from the known distance DIST between the two recording positions 201, 202.

A position in the two-dimensional map MAP can thus be determined for the respective visible outer corner points C1-1, C1-2, C2-1, C2-2 of the objects 230, 240. It should be noted that this procedure loses accuracy with strongly rounded corner points.

An exemplary result of this method in the form of a true-to-scale two-dimensional map 250 is shown in FIG 4 shown. The four determined corner points C1-1, C1-2, C2-1, C2-2 of the objects 230, 240 are entered in the map 250, as well as the current position 202 of the camera and the outline of the vehicle 100. It can be seen that Based on this map 250, a parking maneuver into the space between the objects 230, 240 cannot yet be carried out reliably, since information about a boundary of the objects 230, 240 between the corner points C1-1, C1-2, C2-1, C2- 2 does not exist.

In order to overcome this problem, it is proposed to use ultrasonic sensor signals to determine object points P1-P7. The 5 12 shows a completed version of the two-dimensional map 260 based on that of FIG 4 based, but enriched with detections of object points P1 - P7. It should be noted that the object points P1-P7 do not necessarily have to be detected at the same time, but rather are detected, for example, in the course of the vehicle 100 driving past the objects 230, 240.

Also the one in the 5 The map 260 shown, which already contains information from different sensor systems, namely optical sensors and ultrasonic sensors, is not yet sufficient to reliably carry out a parking maneuver, since the object points P1 - P7 detected by means of ultrasound are a priori independent of one another and of the objects 230 , 240 are.

Based on 6 is explained by way of example how the object points P1-P7 can be assigned to a respective object 230, 240, so that an object boundary line L1, L2 can be determined, on the basis of which the parking lot 270 can in turn be determined. In order to assign the object points P1 - P7, a respective object area A1, A2 is defined, which is shown as an angle in this example. The respective object area A1, A2 extends between the respective corner points C1-1 and C1-2 as well as C2-1 and C2-2 of the objects 230, 240. Those object points P1 - P3 which lie in the object area A1 of the first object 230 are assigned to the first object 230, and those object points P4-P7 that lie in the object area A2 of the second object 240 are assigned to the second object 240. The respective boundary line L1, L2 is obtained by connecting the points C1-1, P1, P2, P3, C1-2 and C2-1, P4, P5, P6, P7, C2-2 assigned to an object with a line.

The parking space 270 can now be determined on the basis of the object boundary line L1, L2 and an automated parking maneuver into this parking space 270 can take place.

7 shows a schematic block diagram of an example of a parking assistance system 110 for a vehicle 100, for example the vehicle 100 of FIG 1 . Parking assistance system 110 includes a receiving unit 112 for receiving a first image 210 (see FIG 3 ) of an environment 200 (see 1 , 2 ) of the vehicle 100, which is transported from a first pick-up position 201 (see 3 ) is captured from and a second image 220 (see 3 ) of the surroundings 200 of the vehicle 100, which is viewed from a second recording position 202 (see 3 , 4 ) from is detected, a determination unit 114 for determining an object 230, 240 (see 2 , 3 ) in the received first image 210 and in the received second image 220, a determination unit 116 for determining a respective bounding box 231 A, 231 B, 241A, 241B encompassing the determined object 230, 240 (see 3 ) in the first image 210 and in the second image 220, the respective bounding box 231A, 231B, 241A, 241B having a respective first and a respective second vertical boundary line 232A, 233A, 242A, 243A, 232B, 233B, 242B, 243B ( please refer 3 ), each of which has a vertex C1-1, C1-2, C2-1, C2-2 (see 3 , 4 ) of the determined object 230, 240 in the respective image 210, 220, and a triangulation unit 118 for carrying out a triangulation of at least one corner point C1-1, C1-2, C2-1, C2-2 of the determined object 230, 240 on the basis a distance DIST (see 3 ) from the first recording position 201 to the second recording position 202 and the respective vertical boundary line 232A, 233A, 242A, 243A, 232B, 233B, 242B, 243B, which defines the corner point C1-1, C1-2, C2-1, C2-2 in the first image 210 and in the second image 220, for determining a relative position of the corner point C1-1, C1-2, C2-1, C2-2 from the first recording position 201 and the second recording position 202.

8th FIG. 1 shows a schematic block diagram of an example of a method for operating a parking assistance system 110 for a vehicle, for example the parking assistance system in FIG 7 and the vehicle of 1 . In a first step S1, a first image 210 (see 1 ) of an environment 200 (see 1 ) of the vehicle 100, which is transported from a first pick-up position 201 (see 3 ) is captured from and a second image 220 (see 3 ) of the surroundings 200 of the vehicle 100, which is viewed from a second recording position 202 (see 3 ) is detected from received. In a second step S2, an object 230, 240 (see 2 , 3 ) in the received first image 210 and in the received second image 220 is determined. In a third step S3, a respective bounding box 231A, 231B, 241A, 241B encompassing the determined object 230, 240 (see 3 ) in the first image 210 and in the second image 220, the respective bounding box 231A, 231B, 241A, 241B having a respective first and a respective second vertical boundary line 232A, 233A, 242A, 243A, 232B, 233B, 242B, 243B (please refer 3 ), each of which has a vertex C1-1, C1-2, C2-1, C2-2 (see 3 - 6 ) of the determined object 230, 240 in the respective image 210, 220, and in a fourth step S4 a triangulation of at least one corner point C1-1, C1-2, C2-1, C2-2 of the determined object 230, 240 is carried out Basis of a distance DIST (see 3 ) from the first recording position 201 to the second recording position 202 and the respective vertical boundary line 232A, 233A, 242A, 243A, 232B, 233B, 242B, 243B, which defines the corner point C1-1, C1-2, C2-1, C2-2 in the first image 210 and in the second image 220, in order to determine a relative position of the vertex C1-1, C1-2, C2-1, C2-2 from the first recording position 201 and the second recording position 202.

Although the present invention has been described using exemplary embodiments, it can be modified in many ways.

Reference List

100

vehicle

110

parking assistance system

112

receiving unit

114

investigation unit

116

unit of determination

118

triangulation unit

120

optical sensor

130

ultrasonic sensor

201

position

202

position

210

Picture

220

Picture

230

object

231A

bounding box

231B

bounding box

232A

vertical line

232B

vertical line

233A

vertical line

233B

vertical line

240

object

241A

bounding box

241B

bounding box

242A

vertical line

242B

vertical line

243A

vertical line

243B

vertical line

250

Map

260

Map

270

parking spot

A1

object area

A2

object area

C1-1

vertex

C1-2

vertex

C2-1

vertex

C2-2

vertex

DIST

Distance

L1

object boundary line

L2

object boundary line

MAP

Map

P1

object point

p2

object point

P3

object point

P4

object point

P5

object point

P6

object point

P7

object point

S1

process step

S2

process step

S3

process step

S4

process step

V1

line

v2

line

w1

angle

W2

angle

Claims (12)

Method for operating a parking assistance system (110) for a vehicle (100), the method comprising: Receiving (S1) a first image (210) of an area (200) of the vehicle (100), which is recorded from a first recording position (201), and a second image (220) of the area (200) of the vehicle (100) , which is detected from a second recording position (202), determining (S2) an object (230, 240) in the received first image (210) and in the received second image (220), Determining (S3) a respective bounding box (231A, 231B, 241A, 241B) encompassing the determined object (230, 240) in the first image and in the second image, the respective bounding box (231A, 231B, 241A, 241 B) a respective first and a respective second vertical boundary line (232A, 232B, 233A, 233B, 242A, 242B, 243A, 243B), each of which defines a corner point (C1-1, C1-2, C2-1, C2-2) of the identified object (230, 240) in the respective image (210, 220), and Carrying out a triangulation of at least one corner point (C1-1, C1-2, C2-1, C2-2) of the determined object (230, 240) based on a distance (DIST) from the first recording position (201) to the second recording position ( 202) and the respective vertical boundary line (232A, 232B, 233A, 233B, 242A, 242B, 243A, 243B) that defines the corner point (C1-1, C1-2, C2-1, C2-2) in the first image ( 210) and in the second image (220), for determining a relative position of the corner point (C1-1, C1-2, C2-1, C2-2) to the first recording position (201) and the second recording position (202) . procedure after claim 1 , characterized by : determining a true-to-scale two-dimensional map (MAP, 250) of the surroundings (200) on the basis of the determined relative position of the corner point (C1-1, C1-2, C2-1, C2-2) and the first recording position (201 ) and the second pickup position (202). procedure after claim 2 , characterized by : receiving an ultrasonic sensor signal which is indicative of the surroundings (200) of the vehicle (100), determining a position of an object point (P1 - P7) which is a point of the object (230, 240) which is a emitted ultrasonic signal is reflected on the basis of the received ultrasonic sensor signal, and entering the determined object point (P1 - P7) in the determined true-to-scale two-dimensional map (MAP, 250) of the environment (200) to provide a completed map (260). procedure after claim 3 , characterized by : determining an object area (A1, A2) for the determined object (230, 240) in the completed map (260) on the basis of the determined corner points (C1-1, C1-2, C2-1, C2-2) of the object (230, 240), the object area (A1, A2) covering the area between schen two vertices (C1-1, C1-2, C2-1, C2-2) of the object (230, 240), and determining an object boundary line (L1, L2) in the completed map (260) based on the determined object area (A1, A2), the corner points (C1-1, C1-2, C2-1, C2-2) and in the object area (A1, A2) lying object points (P1 - P7). procedure after claim 4 , characterized by : determining a parking space (270) on the basis of the determined object boundary line (L1, L2). procedure after claim 5 , characterized by : carrying out an automated parking maneuver based on the determined parking space (270). Method according to one of the preceding claims, characterized in that the first and the second image (210, 220) of the surroundings (200) are recorded and made available by a number of on-board cameras (120). Method according to one of the preceding claims, characterized in that the first and the second image (210, 220) of the surroundings (200) are captured and made available by the same on-board camera (120) at different times, a driving condition sensor signal also being received , and wherein the distance (DIST) of the first recording position (201) to the second recording position (202) is determined on the basis of the received driving condition sensor signal. Method according to one of the preceding claims, characterized in that the determination of the object (230, 240) in the received first and second image and / or the determination of the respective bounding box (231A, 231B, 241A, 241B) by an artificial intelligence trained thereon he follows. Computer program product, comprising instructions which, when the program is executed by a computer, cause the latter to carry out the method according to one of Claims 1 - 9 to execute. Parking assistance system (110) for a vehicle (100), with a receiving unit (112) for receiving a first image (210) of surroundings (200) of the vehicle (100), which is recorded from a first recording position (201), and a second image (220) of the surroundings (200) of the vehicle (100) captured from a second pickup position (202), a determination unit (114) for determining an object (230, 240) in the received first image (210) and in the received second image (220), a determination unit (116) for determining a respective bounding frame (231A, 231B, 241A, 241B) encompassing the determined object (230, 240) in the first image (210) and in the second image (220), the respective bounding frame ( 231A, 231B, 241A, 241B) comprises a respective first and a respective second vertical boundary line (232A, 232B, 233A, 233B, 242A, 242B, 243A, 243B) each having a vertex (C1-1, C1-2, C2-1, C2-2) of the determined object (230, 240) in the respective image (210, 220), and a triangulation unit (118) for carrying out a triangulation of at least one corner point (C1-1, C1-2, C2-1, C2-2) of the determined object (230, 240) on the basis of a distance (DIST) from the first recording position (201 ) to the second recording position (202) and the respective vertical boundary line (232A, 232B, 233A, 233B, 242A, 242B, 243A, 243B) which defines the corner point (C1-1, C1-2, C2-1, C2-2 ) in the first image (210) and in the second image (220), for determining a relative position of the corner point (C1-1, C1-2, C2-1, C2-2) to the first recording position (201) and the second receiving position (202). Vehicle (100) with at least one camera (120) for capturing and outputting an image (210, 220) of an environment (200) of the vehicle (100), and with a parking assistance system (110). claim 11 .

Images