DE102022112317A1 - Method for determining three-dimensional extent information of a target object, motor vehicle, computer program and electronically readable data carrier - Google Patents

Abstract

Method for determining three-dimensional expansion information of a target object (2, 12, 22), in particular a vehicle, from a two-dimensional camera image (1, 11, 21) showing the target object (2, 12, 22), which has a camera (39) which is intrinsically and extrinsically calibrated with respect to an ego object carrying the camera (39), in particular a motor vehicle (38), is recorded, with ground information relating to the three-dimensional position of the ground plane (30) under the target object (2, 12, 22) to the camera (39) and a rectangle (4) which is determined by evaluating the camera image (1, 11, 21) using an evaluation algorithm and which encloses the target object (2, 12, 22) in the camera image (1, 11, 21), 14, 24) describing the two-dimensional evaluation information, the three-dimensional expansion information is determined, whereby - as part of the evaluation information, the orientation of the target object (2, 12, 22) for at least one vertically extending side of the target object (2, 12, 22) on its lower side is also determined Orientation information indicating the edge in the camera image (1, 11, 21), in particular an orientation line (6, 26) or an orientation point (16) depending on the image content of the camera image (1, 11, 21), is determined, and - for at least one by means The calculation point (31, 33, 35, 36, 37) of the camera image (1, 11, 21) determined from the orientation information and the rectangle (4, 14, 24) determines the three-dimensional position of an assigned floor corner point of a which lies on the floor plane (30). Target object (2, 12, 22)

Description

The invention relates to a method for determining three-dimensional expansion information of a target object, in particular a vehicle, from a two-dimensional camera image showing the target object, which is recorded with a camera that is intrinsically and extrinsically calibrated with respect to an ego object carrying the camera, in particular a motor vehicle , wherein the three-dimensional expansion information is determined from ground information on the three-dimensional position of the ground plane under the target object to the camera and from two-dimensional evaluation information determined by evaluating the camera image using an evaluation algorithm and describing at least one rectangle enclosing the target object in the camera image. The invention also relates to a motor vehicle, a computer program and an electronically readable data carrier.

For a variety of applications based on sensor data, especially in the automotive sector, information about the position and extent of target objects in the detection range of the respective sensor should be derived from sensor data. A cheap variant of such a sensor is a camera that records two-dimensional camera images. However, since this only provides two-dimensional camera images, the subject of research and development is also methods for deriving three-dimensional expansion information regarding at least one target object from such two-dimensional camera images.

In particular, the detection of target objects in road traffic, for example the detection of other vehicles as well as their position, orientation and size, is an important area of application, especially with regard to applications that are intended to enable the at least partially automatic guidance of vehicles, in particular motor vehicles. If there is only a single, outward-facing camera in the motor vehicle, for example a front camera, current approaches can either localize the target objects in the two-dimensional camera image or in three-dimensional space. It has already been proposed to use machine learning methods for both approaches, for example trained functions such as neural networks. However, large amounts of data are required to achieve the required performance and reliability.

With regard to many driving functions, in particular driving functions that enable at least partially automatic guidance of a motor vehicle, methods and approaches known in the prior art do not provide sufficient information (2D target object recognition) or require a large number of precise annotations of target objects in three-dimensional space (3D object detection). Such three-dimensional annotations for training data are very complex and can often only be obtained using additional sensor technology, for example laser scanners. Two-dimensional annotations, in turn, are easier and faster to create. In this context, it was suggested to use annotated rectangles around the target objects or to assign pixels to target objects (instance segmentation). However, these 2D annotations initially do not provide any information about the position and orientation of the target object in three-dimensional space.

US 2020 / 0160033 A1 describes systems and methods for collecting three-dimensional representations from monocular two-dimensional images. A monocular 2D image is captured by a camera and processed to produce one or more property maps whose properties may include these properties or object labels.

Regions of interest that correspond to vehicles in the image are determined and a lifting function is applied to each region of interest to determine values such as height and width, camera distance and orientation. An eight-point box, i.e. a cuboid, is determined, which is a 3D representation of the vehicle contained in the region of interest. The 3D representations can be used, for example, for route planning, collision avoidance or as training data. Information about the subsoil, terrain, roads, surfaces and other properties can be included.

A method for estimating a relative position of an object in the environment of a vehicle is provided by EP 3 594 002 A1 described. Here, the relative position is determined on the basis of a two-dimensional camera image, wherein an object contour of the object in the camera image is first determined and at least one digital object template is determined which represents the object on the basis of the object contour. The at least one object template is projected from different positions onto an image plane of the camera image, with each forward-projected object template achieving a corresponding two-dimensional contour suggestion. The correct position should be determined by comparing the contour suggestions with the object contour of the object. The object contour can be a two-dimensional boundary box, i.e. a rectangle. The object template can be a three-dimensional boundary box, i.e. a cuboid, that represents the object. It is also conceivable that each object template represents a specific object type and size as well as spatial orientation, for example a vehicle, a pedestrian or a cyclist. The number of possible positions for the forward projections can be reduced by providing a ground plane.

US 2016/0140400 A1 discloses systems and methods for providing an advanced warning system (AWS) to a driver of a vehicle in which traffic scenes are recorded using a single camera video. The individual camera video is evaluated to generate a monocular SFM (Structure from Motion) and two-dimensional object detection in real time, from which a ground level is determined. A dense 3D estimation is performed using the monocular SFM and the two-dimensional object detection to produce a joint 3D object localization from the ground plane and the dense three-dimensional estimation. In particular, two-dimensional bounding boxes in the images can be determined from the monocular video, which are then tracked across multiple images to derive 3D information.

The invention is based on the object of providing an improved, in particular easy to implement and robust, determination of expansion information describing the three-dimensional position and extent of a target object from two-dimensional camera images.

This task is solved by a method, a motor vehicle, a computer program and an electronically readable data carrier according to the independent claims. Advantageous refinements result from the subclaims.

• - als Teil der Auswertungsinformation zusätzlich eine die Orientierung des Zielobjekts für wenigstens eine vertikal verlaufende Seite des Zielobjekts an deren unterem Rand in dem Kamerabild anzeigende Orientierungsinformation, insbesondere je nach Bildinhalt des Kamerabildes eine Orientierungslinie oder ein Orientierungspunkt, ermittelt wird, und
• - für wenigstens einen mittels der Orientierungsinformation und des Rechtecks ermittelten Rechenpunkt des Kamerabildes die dreidimensionale Position eines zugeordneten, auf der Bodenebene liegenden Bodeneckpunkts der Bodenfläche eines das Zielobjekt wenigstens teilweise umschließenden Quaders unter Verwendung eines Aufnahmegeometriemodells der Kamera ermittelt wird und daraus die Ausdehnungsinformation als zumindest die Bodenfläche des Quaders, insbesondere als der das Zielobjekt wenigstens teilweise umschließende Quader, rekonstruiert wird.

In a method of the type mentioned at the outset, the invention provides that

• - As part of the evaluation information, orientation information indicating the orientation of the target object for at least one vertically extending side of the target object at the lower edge in the camera image, in particular an orientation line or an orientation point depending on the image content of the camera image, is determined, and
• - for at least one computing point of the camera image determined by means of the orientation information and the rectangle, the three-dimensional position of an assigned floor corner point of the floor surface of a cuboid at least partially enclosing the target object, lying on the floor plane, is determined using a recording geometry model of the camera and from this the expansion information is determined as at least the floor surface of the cuboid, in particular as the cuboid that at least partially encloses the target object, is reconstructed.

The term "vertically running side" refers to an at least essentially vertical course of the corresponding side of the target object in three-dimensional space, which means, in particular with regard to the enclosing cuboid, the front, back and lateral sides, i.e. right and left sides. It is fundamentally assumed that the target object can be represented by a cuboid that at least partially, in particular completely, encloses it, which is the case, for example, in vehicles, in particular motor vehicles, which easily have a top, a bottom and vertical sides such as the front, back and right as well left side can be assigned, is expediently possible.

In other words, the cuboid that at least partially encloses the target object, which can be described in the corresponding extension information in particular by its eight corner points, i.e. four floor corner points and four roof corner points, can be viewed as a representation of the target object, which shows the three-dimensional position, the three-dimensional orientation and reproduces the three-dimensional extent of the target object with sufficient accuracy. Here, the rectangle and/or the cuboid are preferably determined in such a way that they enclose the target object as closely as possible, in particular as the smallest possible two-dimensional or three-dimensional bounding box. It should be noted that the preferred closest possible enclosing of the cuboid in three-dimensional space does not necessarily result in the closest possible enclosing of the projection in the camera image, but rather the rectangle can protrude particularly on the underside, for example at contact points on the floor level that are offset from the corners. This knowledge can be taken into account when determining the rectangle, as will be explained later. In addition, it is also conceivable to define the target object more narrowly in some applications, for example in the case of a motor vehicle, to exclude the exterior mirrors and therefore not to enclose them, since they only protrude locally in a very small area.

The evaluation information can therefore be understood as an annotation in the two-dimensional space of the camera image. Therefore, according to the The present invention proposes that, instead of working with annotations in three-dimensional space or only with rectangles in the camera image, the annotated rectangles of the evaluation information are supplemented with additional information that makes it possible to create three-dimensional representations, specifically floor areas or preferably entire cuboids, based on knowledge of a floor level Create target objects. It has been shown that, in particular, information regarding the orientation in the camera image, i.e. in two dimensions, is easy to determine and, as long as it relates to the lower edge of the target object, together with the rectangle it already provides essential information for determining ground corner points Calculation points that can already be derived through simple geometric considerations. At least in the case that two adjacent, vertically extending sides are completely visible in the camera image and the orientation information, in particular as an orientation line, has been determined for at least one of these sides, it is possible through simple geometric considerations solely from the evaluation information, in particular the rectangle and the orientation information, a recording geometry model of the camera, in particular a simple pinhole camera model, and the knowledge of the floor level on which the target object stands, to derive the complete floor area of the cuboid, described by four floor corner points, and if necessary to complete it into a complete cuboid with the help of the rectangle .

The reconstruction of the entire floor area or preferably the entire cuboid on the basis of the at least one floor corner point, which has already been determined based solely on the evaluation information, the floor plane and the recording geometry model, is preferably carried out solely on the basis of the evaluation information, the floor plane and the recording geometry model, but can at least at Image content that does not completely encompass two vertical sides of the target object in perspective, additional information, in particular dimensional information contained in or assigned to the evaluation information, is also taken into account. In particular, the floor area can first be completed by determining the remaining floor corner points, after which the upper boundary surface of the cuboid (hereinafter roof area) parallel to the floor surface can then be determined, if desired, using the floor corner points, the rectangle and the recording geometry model, if necessary with additional and /or alternative consideration of the dimensional information (e.g. if the target object is “cropped” in the camera image above).

The evaluation information can be generated cost-effectively and with little effort, and the derivation of the floor area, in particular the cuboid, as a three-dimensional representation of the target object is possible with little computational effort, i.e. can be implemented inexpensively, and also enables a robust implementation. The determination of the position, orientation and size of the target object in three-dimensional space, described by the floor area or the entire cuboid as extension information, is therefore made possible in a simple, low-effort and robust manner by considering orientation information in the camera image as a basis .

The camera, which can be, for example, a front camera on a motor vehicle as an ego object, provides a camera image of the captured scene containing at least one target object, for example from road traffic when viewing vehicles as target objects. The roll angle of the camera relative to the ground or to the ego object, in particular a motor vehicle, can preferably be at least essentially zero. This makes it particularly easy to identify the vertical in the two-dimensional camera image. Preferably, the camera images are rectified, as is generally known, so that a pinhole camera model can be used as the recording geometry model.

The camera is intrinsically and extrinsically calibrated. The intrinsic calibration of the camera enables the projection of points in the three-dimensional camera coordinate system into the “equalized” camera image using the recording geometry model. Extrinsic calibration enables the transformation of points in the camera coordinate system into the ego object coordinate system, in particular the vehicle coordinate system, and vice versa. In this case, additional sensors provided on the ego object, in particular in the motor vehicle, in particular distance sensors such as radar sensors and/or lidar sensors and/or laser scanners, can preferably also be calibrated, so that the transformation of points in the respective sensor coordinate system into the ego object coordinate system and vice versa is possible . The ego object geometry is known, so that a description of a (reference) ground plane can be given, especially when the ego object is stationary.

The ground level is preferably determined as accurately as possible so that it corresponds to the base area of the respective target object on which it stands. Within the scope of the present invention it can be provided that that the ground level under the target object is determined using the extrinsic calibration of the camera by assuming the same position as the ground level under the ego object, whereby, particularly in the case of possible pitching and/or rolling movements, position data from a position sensor and/or other sensor data are also taken into account in order to take into account changes in the position of the camera relative to the ground level on the ego object side. However, changes in the ground level from the ego object to target objects in its environment would not be taken into account or would be assumed to be small. Therefore, within the scope of the present invention, it is preferred to determine an individual ground level for each target object, which describes the ground on which the target object stands as precisely as possible. This is always an advantage when the entire ground in the area cannot be approximated accurately enough using a single level, for example in the case of undulating roads or branching ramps. In this case, it would no longer be practical to neglect the changes in the ground plane compared to the (reference) ground plane of the ego object. In this regard, an embodiment of the present invention can preferably provide that the ground plane under the target object, again using the extrinsic calibration of the camera, from sensor data of a further distance sensor that is also registered extrinsically with the ego object, in particular a laser scanner and/or radar sensor and/or lidar sensor , is determined. Using such distance sensors, the course of the ground around the ego object can be scanned using known methods and used accordingly, for example by using the scanned ground course around the target object to determine the ground level. Additionally or alternatively, to determine target object-specific ground levels, map data present in particular in the ego object, for example a navigation system of the motor vehicle, can be used, which in particular describes the topology in the surroundings of the ego object, in particular the motor vehicle. In this context, it has already been proposed for navigation environments, in particular parking environments, to provide digital map data in motor vehicles operated in them.

As already mentioned, a pinhole camera model is preferably used as the recording geometry model, which, with intrinsic calibration and corresponding equalization of the camera images, represents an extremely simple way of determining rays, which require little computational effort, along which features/points lie at a specific image point (pixel) in the camera image lay. In addition, straight lines in three-dimensional space are mapped onto straight lines in the image, which also means that the rectangle provides a more precise impression of the cuboid.

According to the invention, the evaluation information, which is determined by the evaluation algorithm, comprises at least one rectangle that encloses the projection of the cuboid surrounding the object as closely as possible into the camera image as accurately as possible. It should be noted at this point that the rectangle does not necessarily have to precisely enclose the two-dimensional, visible silhouette of the target object, but in exemplary embodiments it can also be chosen to be larger, especially in the area of the lower edge of the rectangle, in order, for example, to increase the stroke of vehicles as target objects to compensate for wheels that are offset along the longitudinal direction.

If the target object is partially obscured in the camera image, the evaluation algorithm can be used to estimate the actual dimensions of the rectangle. Such an estimate can also be made with regard to other partial information of the evaluation information if it is partially obscured.

Appropriately and with particular advantage, separation information describing the position of at least one vertically extending side of the target object visible in the camera image, in particular a vertically extending dividing line between different vertically extending sides of the target object in the camera image, is also determined as part of the evaluation information. The separation information therefore shows, on the one hand, how many vertically extending sides of the target object were detected in the camera image, but on the other hand it also shows in which area of the camera image, in particular the rectangle, these sides lie, so that the separation information is vertical when there are several visible ones running sides of the target object allows the rectangle to be divided as to where which of the vertically running sides of the target object can be found. It is particularly advantageous to determine a dividing line as separating information, which runs vertically in the camera image at least through the rectangle and divides it into areas in which one of the vertically extending sides of the target object can be found. In other words, a vertical, straight dividing line can be determined which, if there is more than one vertical side of the target object visible in the camera image, divides the rectangle into different parts or areas, each of which contains exactly one vertical side of the target object. If only one vertical side of the target object is visible, the dividing line is conveniently aligned with the right or left edge of the rectangle. With others In other words, if only one vertical side of the target object is visible in the camera image, the dividing line can be determined as a vertical edge of the rectangle as separating information.

In this context, it is also particularly advantageous if for at least one vertically extending side of the target object, in particular at least the at least one side to which the orientation information is assigned, a page classification into front, back or lateral side, in particular right side or left, is part of the evaluation information Page, is determined. In particular, for each vertical side of the target object described by the separation information and visible in the camera image, a side classification can take place as a front, back, right or left side, although it is also conceivable to combine the right side and left side as the lateral side. Together with the separation information, if there are several visible, vertical sides of the target object, it becomes clear which part of the rectangle shows which side and how it should be classified. It should be noted that, in principle, assigning one side or an area of the rectangle to a side classification is sufficient, since adjacent sides that run vertically can be identified from this, assuming that the target object has its bottom side on the ground level. For example, it can be assumed that a vehicle as a target object is standing with its wheels on the ground.

Basically, in the context of the present invention, the specific evaluation information also includes the orientation information, in particular either as an orientation line or as an orientation point, with an orientation line expediently resulting when two vertical sides of the target object are visible in perspective in the camera image and an orientation point is used when only one vertical side of the target object is visible in the camera image (the orientation line then degenerates, so to speak, into the orientation point). In other words, whenever a vertical side of the target object, in particular a lateral side, is perspectively visible in the camera image, the orientation line can be determined as an oblique line along the visible lower edge of the side of the target object, in particular the right or left side of the target object . In the case of vehicles as target objects, the orientation line can therefore run along the points of contact between the wheels and the ground. If only one vertical side is visible, especially only the back or front, the orientation line degenerates into a point. In other words, it can be provided that the orientation information is determined as an orientation line along the lower edge of one of the sides of the target object visible in the camera image, in particular a lateral side, in one Vehicle as a target object, in particular as an orientation line touching the lower edge of the vehicle's wheels. Furthermore, it can be provided that if only one vertically extending side of the target object is visible in the camera image, the orientation information is determined as an orientation point indicating an orientation extending into the image plane.

In an advantageous development of the present invention, a class of the target object can also be determined as part of the evaluation information, to which, for example, dimensional information describing at least one expected dimension of a target object of the class can be assigned. For example, the class of the target object describes a type of the target object. For example, the class information for vehicles as target objects, which can also be usefully determined independently of the separation information, can include at least an assignment to at least one two-wheeler class and/or at least one passenger car class and/or at least one truck class and/or at least one bus class. However, a more detailed division into classes based on the actual size and shape of the vehicles can also be carried out with particular advantage. For example, with regard to passenger cars, classes such as “passenger car, upper middle class, combination car”, “passenger car, small car, station wagon”, “passenger car, upper class, SUV” and the like can be considered. All of these classes or types can expediently be assigned dimensional information, which in particular describes expected dimensions (expected dimensions) of corresponding target objects, for example in the form of an expected width and/or an expected length and/or an expected height. The dimensional information can also contain intervals for permissible dimensions in the corresponding class of the target object, which will be discussed in more detail below.

If there is a page classification in the evaluation information, it can expediently be provided that pages of the cuboid in the extent information are assigned to corresponding pages of the target object. For example, if the cuboid is described by its vertices (floor vertices and roof vertices), the assignments in the side classification can be used to determine which of the ground vertices corresponds to the front/rear left/lower right corner of the target object. One Corresponding assignment can be obtained for roof vertices to the front/rear left/right upper corner points. This results in immediate assignments from the side of the cuboid to the side of the target object. This is useful, for example, when driving directions and/or travel options are to be estimated for vehicles as target objects, since the orientation can then be assigned to a corresponding orientation.

If the evaluation information comprehensively determines the separation information, a case distinction can be made in a simple manner as to exactly how the extent information is to be determined. In a first case, the separation information indicates that two vertical sides of the target object are visible in the rectangle. In other words, the dividing line separates the rectangle into two rectangles, each with a positive area. Then the complete floor area and thus the complete cuboid can be determined using only the rectangle, the orientation line of the orientation information, the floor plane and the recording geometry model, which is particularly advantageous since no external information or information from other time steps has to be included. In a second case, which, like the first case, relates to target objects completely contained in the camera image, only a vertical side of the target object is visible in the camera image, so that in the specific case of a dividing line, this corresponds to the left or right edge of the rectangle. In such a case, as will be explained, it is expedient to rely on the dimensional information already mentioned, which can also come at least in part from analyzes of at least one previous time step in which more vertical sides of the target object were visible.

• - ein erster Rechenpunkt als Schnittpunkt der Orientierungslinie mit der unteren Kante des Rechtecks in dem Kamerabild und ein zweiter Rechenpunkt als Schnittpunkt der Orientierungslinie mit der der Seite, auf die die Orientierungslinie bezogen ist, gemäß der Trenninformation zugeordneten vertikalen Kante des Rechtecks ermittelt werden,
• - mittels des Aufnahmegeometriemodells für den ersten und den zweiten Rechenpunkt Strahlen, auf denen die Rechenpunkte liegen, ermittelt werden und die jeweiligen Schnittpunkte der Strahlen mit der Bodenebene als erste und zweite Bodeneckpunkte im dreidimensionalen Raum bestimmt werden,
• - eine senkrecht zu einer den ersten und den zweiten Bodeneckpunkt verbindenden Verbindungsgeraden in der Bodenebene liegende, durch den ersten Bodeneckpunkt verlaufende Suchgerade sowie ein auf der Suchgerade liegender dritter Bodeneckpunkt, dessen aufgrund des Aufnahmegeometriemodells bestimmte Projektion in das Kamerabild auf der der Kante der Seite, auf die die Orientierungslinie bezogen ist, gegenüberliegenden vertikalen Kante des Rechtecks in dem Kamerabild liegt, im dreidimensionalen Raum bestimmt werden, und
• - ein vierter, die rechteckförmige Bodenfläche vervollständigender Bodeneckpunkt aus dem ersten, zweiten und dritten Bodeneckpunkt bestimmt wird, insbesondere durch Addition der dem dritten und dem zweiten Bodeneckpunkt zugehörigen Vektoren und Subtraktion des dem ersten Bodeneckpunkt zugehörigen Vektors.

In a concrete, advantageous embodiment of the present invention, it can therefore be provided that with two vertically extending sides of the target object visible in the camera image (case 1) and an orientation line related to one of the visible sides, in particular a lateral side, as orientation information for determining a surface covered by the target object on the ground

• - a first calculation point is determined as the intersection of the orientation line with the lower edge of the rectangle in the camera image and a second calculation point is determined as the intersection of the orientation line with the vertical edge of the rectangle assigned to the side to which the orientation line relates according to the separation information,
• - using the recording geometry model for the first and second calculation points, rays on which the calculation points lie are determined and the respective intersection points of the rays with the ground plane are determined as first and second ground corner points in three-dimensional space,
• - a search line lying perpendicular to a connecting line in the floor plane connecting the first and second floor corner points and running through the first floor corner point, as well as a third floor corner point lying on the search line, whose projection into the camera image on the edge of the page is determined based on the recording geometry model which is related to the orientation line, lies opposite vertical edge of the rectangle in the camera image, can be determined in three-dimensional space, and
• - a fourth floor corner point completing the rectangular floor area is determined from the first, second and third floor corner points, in particular by adding the vectors associated with the third and second floor corner points and subtracting the vector associated with the first floor corner point.

It should be noted at this point that if orientation lines have been determined for both sides, they should ideally intersect at the first calculation point. If necessary, this can also be brought about, for example, by averaging and subsequent parallel displacement of the orientation lines, weighted in particular with regard to the robustness of the determination of the orientation lines. It can also be provided that the evaluation algorithm already determines both orientation lines accordingly. If the orientation lines intersect at the first calculation point, a third calculation point can be determined analogous to the second calculation point, which directly supplies the third floor corner point analogous to the first and second floor corner points and with which the fourth floor corner point can be derived directly. The procedure described above with a search line can then be used to check plausibility. However, for the application of vehicles as target objects, it has been shown that for lateral sides, due to the fact that the contact points of the wheels can be determined, the orientation information can be determined much more robustly and easily than for back or front sides, which are also significantly shorter and do not touch the ground. so that it is then preferred to determine an orientation line as orientation information, which is preferably related to a lateral side.

This case of only one orientation line used for the orientation information will now be explained in more detail.

First, intersection points of the orientation line with edges of the rectangle are determined, whereby the intersection of the orientation line with the lower edge of the rectangle forms a first calculation point, the intersection of the orientation line with the side edge of the rectangle, which is intersected by the orientation line (and thus also the vertical one , the side of the target object visible in the camera image to which it is assigned) forms a second calculation point. According to the recording geometry model, in particular in the preferred example of the pinhole camera model, all points in the three-dimensional camera coordinate system that are projected onto a point in the camera image lie on a ray. This can be calculated using the recording geometry model based on the intrinsic camera calibration. This results in a first ray for the first calculation point and a second ray for the second calculation point. The intersections of these rays with the ground plane in the three-dimensional camera coordinate system can now be calculated and form the first ground vertex (intersection of the first ray and the ground plane) and the second ground vertex (intersection of the second ray and the ground plane). Now there is a clear search line that runs in the ground plane and through the first ground corner point and forms a right angle with the line that connects the first ground corner point and the second ground corner point. On the search line, in turn, there is a clear third ground corner point, whose projection into the camera image lies on the line formed by extending the side edge of the rectangle, which is not intersected by the orientation line. If there are no intersections of the first and second rays with the ground plane, this can be corrected, in particular by raising the horizon.

It should be noted at this point that if the third ground corner point is determined in such a way that its projection into the camera image lies on an extension of the corresponding edge outside the rectangle, the ground plane and/or the orientation line and/or the rectangle and/or or the first and/or the second calculation point can be corrected in such a way that the projection comes to lie on the edge. Here, for example, the information can be used as to the direction in which the projection lies outside the rectangle, and/or correction algorithms using optimization methods can be used in order to achieve a plausible result through minimal possible adjustments.

If the third floor corner point lies on the side edge of the rectangle that is not intersected by the orientation line, it is assumed to be the correctly determined third floor corner point. After calculating the third floor corner point, the fourth floor corner point is determined accordingly, especially if the vectors to the corresponding floor corner points are designated as V1, V2, V3 and V4, using V4=V3 + V2 - V1.

At this point it is useful to check whether the projection of the fourth ground corner point in the camera image lies in the rectangle. If this is not the case, the floor plane and/or the orientation line and/or the rectangle and/or the first and/or second calculation point can also be adjusted in such a way that the projection comes to lie within the rectangle. In summary, with regard to this plausibility check it can be said that in the case of a projection on an extension of the corresponding edge outside the rectangle having a certain third floor corner point and/or in the case of a projection outside the rectangle having a certain fourth floor corner point, the floor plane and/or the orientation line and/or the rectangle and/or the first and/or the second calculation point are corrected in such a way that the respective projections come to lie on the edge or within the rectangle. In both cases as well as in the combined case, as already mentioned, information about the location outside the rectangle as well as optimization methods to make the smallest possible changes can be used. In this case, the evaluation information, i.e. the two-dimensional annotation, can preferably be assumed to be more accurate and the primary aim is to correct the ground level, which may be inaccurate, especially for distant target objects, for example when determining using lower-resolution LiDAR.

The floor corner points describe a rectangular floor area that lies completely in the floor plane. So they describe the estimated floor area that the target object covers.

If a complete cuboid is to be obtained, its height must still be determined. For this purpose, it can be provided that a roof area of the cuboid assigned to the floor area is determined by the set of roof checkpoints resulting from the addition of a common height value multiplied by the normal unit vector of the floor plane to the floor corner points, their projection into the camera image at the maximum height value for at least one roof checkpoint or all roof checkpoints lie within the rectangle is determined. This is again called the vector ren of the floor vertices with V1, V2, V3 and V4 and if one sets T1, T2, T3 and T4 as vectors of the roof vertices, h also denotes the height and N the normal unit vector, i.e. the vector orthogonal to the floor surface, which points upwards and has the length 1, it can be said that the height h can be chosen as large as possible and so that the projections of all points Tk = S1k + hx V with k = 1, 2, 3, 4 in the camera image are all contained in the rectangle . The rectangular roof area created for this height consisting of T1, T2, T3 and T4 forms the upper end of the cuboid. For reasons of plausibility, it is advisable to require that all projections of the roof vertices in the camera image should lie within the rectangle.

It should be noted at this point that due to the extrinsic calibration of the camera, the position of the floor corner points and the roof corner points and thus of the cuboid are known not only in the camera coordinate system, but also in the ego object coordinate system, in particular the motor vehicle coordinate system.

In the second case mentioned above, additional dimensional information is expediently taken into account in order to be able to estimate as reliably as possible a dimension that is not visible from the camera image, for example length or width. For this purpose, it can be provided that, as part of the evaluation information, a class of the target object, to which dimensional information describing at least one expected dimension of a target object in the class is assigned, is determined, as already mentioned. Additionally or alternatively, dimension information describing at least one expected dimension of the target object can be assigned to the target object from a previous determination of the extent information. For example, if extension information regarding the target object was already determined in the past, in particular on the basis of a camera image recorded in the past, for which the first case existed, i.e. two vertically extending sides of the target object were visible in the camera image, it was also used at this point in time Reliability determines the width and / or length of the target object and can be provided as dimensional information, possibly also from other camera images for which a determination according to the first case is possible, for example constantly updated and refined by averaging. In particular, for cases in which no such dimensional information is available from a previous time step, external dimensional information, so to speak, can also be used, which can expediently be selected based on the class of the target object, which was determined as part of the evaluation information. In this case in particular, it is particularly advantageous to use the most precise possible division of the target object into different classes based on their size and/or shape. In the case of vehicles as target objects, examples have already been given above, so that, for example, smaller expected dimensions can be assumed for a small car than for an SUV. If the expansion information is to be used for a safety-relevant vehicle function, it may also be expedient to specifically set the expected dimension in a maximum range of conceivable values.

• - ein erster Rechenpunkt als unterer linker Eckpunkt des Rechtecks in dem Kamerabild und ein zweiter Rechenpunkt als unterer rechter Eckpunkt des Rechtecks in dem Kamerabild ermittelt werden,
• - mittels des Aufnahmegeometriemodells für den ersten und den zweiten Rechenpunkt Strahlen, auf denen die Rechenpunkte liegen, ermittelt werden und die jeweiligen Schnittpunkte der Strahlen mit der Bodenebene als erste und zweite Bodeneckpunkte im dreidimensionalen Raum bestimmt werden, und
• - zwei senkrecht zu einer den ersten und den zweiten Bodeneckpunkt verbindenden Verbindungsgeraden in der Bodenebene liegende Suchgeraden durch die jeweiligen Bodeneckpunkte und dritte und vierte Bodeneckpunkte entlang der jeweiligen Suchgerade in einer Richtung weg von der Kamera bei Aufnahme des Kamerabildes in einem gemäß der Dimensionsinformation ermittelten Abstand im dreidimensionalen Raum bestimmt werden.

Then in the second case already mentioned above, i.e. with only one vertical side of the target object visible in the camera image, indicated by the separation information, it can be provided to determine a floor area covered by the target object on the ground

• - a first calculation point is determined as the lower left corner point of the rectangle in the camera image and a second calculation point is determined as the lower right corner point of the rectangle in the camera image,
• - using the recording geometry model for the first and second calculation points, rays on which the calculation points lie are determined and the respective intersection points of the rays with the ground plane are determined as first and second ground corner points in three-dimensional space, and
• - two search lines lying perpendicular to a connecting line in the floor plane connecting the first and second floor corner points through the respective floor corner points and third and fourth floor corner points along the respective search line in a direction away from the camera when recording the camera image at a distance determined according to the dimensional information three-dimensional space can be determined.

If the entire cuboid is to be determined again, a roof area of the cuboid assigned to the floor area can be determined by the set of roof vertices resulting from the addition of a common height value multiplied by the normal unit vector of the floor plane to the floor corner points, their projection into the camera image at the maximum height value for at least one Roof checkpoint, in particular all roof checkpoints, that lie within the rectangle can be determined.

In this second case, the lower right and left corners of the rectangle in the camera image are used as calculation points after they correspond to ground corner points. As in the first case, the corresponding floor corners can be used points in the three-dimensional camera coordinate system can be determined by intersecting the corresponding rays with the ground plane. If there are no intersections of the first and second rays with the ground plane, this can be corrected, in particular by raising the horizon. However, two search lines are now used, which are perpendicular to the connection between the first and second ground corner points and run through them. In other words, they can also be understood as search beams that run away from the camera at the time the camera image is recorded and begin at the first or second ground corner point. A distance can now be derived from the dimensional information, for example based on a result obtained in at least one previous time step or based on a corresponding expected dimension. For example, if only the front or back of the target object can be seen, the distance can be understood as a reference length of the target object. If only a lateral side, for example the left side or the right side of the target object, can be seen, the distance can be understood as a reference width . In this context, page classification as part of the evaluation information proves to be particularly useful, since the visible side of the target object is then classified and it is therefore immediately known which dimension is missing.

In other words, it can be provided that if a page classification is present as part of the evaluation information, the expected dimension in the direction perpendicular to the vertically running, classified page visible in the camera image is selected from the dimensional information as a distance. As already mentioned, assigned dimensional information can be used based on a class of the target object; However, if a length or width of the target object that has already been observed at other times is known as the expected extent, this should be used preferentially. The third and fourth ground corner points are now determined by walking along the search line away from the camera at the time the camera image is taken by the distance, i.e. the reference length or the reference width.

In this context, it should also be noted that error cases are fundamentally conceivable in which at least one of the search lines initially runs out of the rectangle in its projection into the camera image. It can then expediently be provided that the ground plane and/or the rectangle are corrected in such a way that the projections of the search line extend at least up to the distance within the rectangle in the camera image. If, in other words, the projection of a search beam into the camera image immediately runs out of the rectangle, a correction step is carried out and the ground plane and / or the rectangle are corrected so that after recalculation of the first and second ground corner points and the search line, the projections of the search line starting from the first or second floor corner point, initially run within the rectangle.

In this way, the floor area of the cuboid is again determined, so that if the entire cuboid is to be determined as expansion information, the roof area can also be determined accordingly, as described in the first case.

In particular, if the target object is not completely contained in the camera image, other cases are conceivable in addition to the first case and the second case, but these can also be dealt with within the scope of the present invention. In such special cases, the target object whose enveloping cuboid is to be determined is not completely visible in the camera image because it is cut off by the edge of the image. However, in many such special cases, at least a very good estimate can still be achieved, at least with the addition of the dimensional information. In other words, if the target object is not completely recorded in the camera image, in particular if there is only one determinable ground corner point, missing information for reconstructing the ground surface or the cuboid can be taken from the dimensional information.

In these special cases, depending on visibility, only partial information from the camera image is used, for example only two or three edges of the rectangle or only one of the first or second ground corner point. Depending on availability, parts of the cuboid you are looking for can then be calculated using the methods described for the first and second cases. Ground corner points that cannot be determined from the camera image can be derived using the dimensional information, i.e. using expected dimensions observed at another time and/or inferred from a class of the target object. For example, if only the second calculation point and part of the orientation line are visible in the camera image, the second ground corner point can be determined. From the orientation line, a ray can be calculated that runs in the ground plane away from the second ground corner point. You walk along this search beam along the expected dimension to get to the first ground corner point.

In exemplary embodiments of the present invention, provision can also be made to use the dimensional information for a plausibility check. For example, it can be provided that the dimensional information additionally includes an interval of permissible dimensions for the respective class of the target object, with certain dimensions of the target object from the camera image being compared with the interval in a plausibility step and, if not plausible, a new determination and/or correction of the extent information and /or its basis and/or the extension information is discarded. Based on the class of the target object, which can already be determined as part of the evaluation information, the obtained extensions, i.e. dimensions, of the cuboid can be further checked for plausibility in the extension information and corrected if necessary. For this purpose, reference intervals for the length, width and, if necessary, the height of target objects of the class can be used for the respective class of the target object. If the dimensions of the floor area or the specific cuboid lie outside these intervals, a correction can be made. It can be provided, for example, that the dimensions are corrected to this low value when the lowest value of the interval is undershot and to be corrected to this highest value when the highest value of the interval is exceeded. The corrections regarding length and width are preferably made symmetrically. However, particularly in the case of strong deviations, it is also conceivable to discard extent information for this time step and use a camera image from the next time step in order to attempt a new determination. For example, the extent information from a previous time step can then continue to be used. Of course, in principle a plausibility check over time steps is also conceivable within the scope of the present invention.

In an expedient development of the present invention, it can be provided that a trained function, in particular a neural network, is used as the evaluation algorithm, which was trained in particular with manually annotated camera images as training data. In the context of the present invention, it has been shown that for the at least desired evaluation information (rectangle, orientation information), but also with the addition of optional, particularly preferred further evaluation information such as the separation information, the page classification and the class of the target object, a robust, fast and manageable one A trained function of artificial intelligence that requires a lot of training data, for example a convolutional neural network (CNN), can be used. Machine learning techniques can therefore be used to train the evaluation algorithm and to derive the evaluation information quickly and with little effort from the image data of the camera image, in particular exclusively from the camera image. Here, for example, manual annotations can be used in order to be able to provide corresponding training data, but it is also conceivable to use at least partially automatically determined annotations to generate the training data. For example, even without artificial intelligence, image processing algorithms are often able to calculate rectangles and orientation lines that closely surround target objects. With the help of such training data, an artificial intelligence function is trained through machine learning, which recognizes relevant target objects in the camera image and outputs exactly the above-mentioned evaluation information for all detected target objects, for example for vehicles.

In this context, it should be noted that with regard to target objects that are not completely contained in the camera image, it is also conceivable to design the evaluation algorithm in such a way that it can estimate the missing information, at least in some cases. It can thus be provided that for at least a portion of target objects that are not completely captured in the camera image, the evaluation algorithm is designed, in particular based on appropriate training, to estimate the complete rectangle protruding from the camera image. In particular, an evaluation algorithm is conceivable that first determines a class of the target object and then uses the dimensional information assigned to the class to determine the evaluation information for the expected dimensions. It is of course also conceivable that the evaluation algorithm first identifies the target object as a target object that has already been processed at a previous point in time, so that the dimensional information from an earlier time step can then be used accordingly.

In addition to the method, the present invention also relates to a motor vehicle, having a control device designed to carry out the method according to the invention and the camera. In this case, vehicles are preferably used as target objects. The camera can be, for example, the front camera of the motor vehicle. All statements regarding the method according to the invention can be transferred analogously to the motor vehicle according to the invention, so that the advantages already mentioned can also be achieved with this.

The control device in particular has at least one memory means and at least one processor, wherein the processor can provide functional units for carrying out the method according to the invention. The control device can in particular include an interface for receiving the camera image recorded by the camera, an evaluation unit for carrying out the evaluation algorithm, a ground level determination unit for determining the ground level/ground information and an extent information determination unit for determining the extent information.

The extent information obtained can fundamentally and generally be used in a variety of ways, including in other applications, for example to generate training data for other artificial intelligence algorithms. However, the expansion information can be used particularly expediently when operating the motor vehicle, in particular for vehicle functions. An expedient development of the motor vehicle according to the invention provides that the or another control device is designed to use the extension information when executing at least one vehicle function, in particular a vehicle guidance function comprising at least partially automatic guidance of the motor vehicle.

In particular, if information from distance sensors is to be used to determine the ground level for the target object, it is also conceivable that the motor vehicle also has the at least one distance sensor.

A computer program according to the invention can be loaded directly into a storage means of a computing device, for example a control unit of a motor vehicle, and has program means in order to carry out the steps of a method according to the invention when the computer program is executed on the computing device. The computer program can be stored on an electronically readable data carrier according to the invention, which therefore includes control information stored thereon, which includes at least one computer program according to the invention and, when the data carrier is used in a computing device, is designed to carry out a method according to the invention. The data carrier can be, for example, a non-transient data carrier, for example a CD-ROM.

• 1 einen Ablaufplan eines Ausführungsbeispiels des erfindungsgemäßen Verfahrens,
• 2 ein erstes Kamerabild mit Annotationen,
• 3 ein zweites Kamerabild mit Annotationen,
• 4 ein drittes Kamerabild mit Annotationen, und
• 5 eine Prinzipskizze eines erfindungsgemäßen Kraftfahrzeugs.

Further advantages and details of the present invention emerge from the exemplary embodiments described below and from the drawing. Show:

• 1 a flowchart of an exemplary embodiment of the method according to the invention,
• 2 a first camera image with annotations,
• 3 a second camera image with annotations,
• 4 a third camera image with annotations, and
• 5 a schematic sketch of a motor vehicle according to the invention.

1 shows a basic flowchart of exemplary embodiments of the method according to the invention. In the following, exemplary embodiments are explained in which a motor vehicle has a camera as the ego object, for example a front camera. However, especially when it comes to generating 3D annotations as training data for an artificial intelligence algorithm, other ego objects are also conceivable, for example tripods or mobile units with cameras. The camera is intrinsically and extrinsically calibrated, with the camera in particular also rectifying recorded two-dimensional camera images, so that rays can be calculated using a pinhole camera model as a recording geometry model, on which visible features, for example points, must lie in certain image points (pixels) of the camera image . Due to the external calibration of the camera, the three-dimensional camera coordinate system is registered with the three-dimensional ego object coordinate system, in the example motor vehicle coordinate system, which means that a conversion of corresponding three-dimensional positions and directions is possible. In the motor vehicle coordinate system, the position of the ground plane on which the motor vehicle travels is known, in particular due to the known position of the wheels.

The camera now delivers two-dimensional camera images of a traffic situation in which a large number of objects are visible. For certain target objects, here vehicles of different types and classes, the method described here is intended to determine extent information that describes the position, orientation and extent of the target objects, i.e. vehicles. In the exemplary embodiments described here, the extent is described by a cuboid surrounding the corresponding vehicle as a target object, and the position and orientation are described by the position and/or orientation of the cuboid. The cuboid is therefore a three-dimensional bounding box.

In a step S1, a two-dimensional camera image is recorded, in particular controlled by the computing device executing the method. In the case of a motor vehicle as ego object, the computing device can be a control unit of the motor vehicle.

In a step S2, the camera image is transferred to an evaluation algorithm, which outputs evaluation information in the form of two-dimensional annotations for the target objects that can be detected in the camera image, here vehicles. The evaluation algorithm is a trained function of artificial intelligence that was trained using appropriate training data (machine learning). The training data can include manually and/or automatically annotated two-dimensional camera images. However, exemplary embodiments are also conceivable in which at least some of the evaluation information is determined “conventionally”, for example by appropriate image processing algorithms without machine learning.

In this case, the evaluation information comprises five components. First, a rectangle is determined for each target object in the camera image, which encloses the target object in the camera image and thus also includes the projection of the cuboid that precisely surrounds the real, three-dimensional target object in the camera image. The rectangle can, but does not necessarily have to, precisely enclose the two-dimensional visible silhouette of the target object, but can be larger, especially on the underside. This is particularly true in the case of vehicles as target objects discussed here, since the wheels, which define the lowest areas, are arranged offset from the front and back.

It should be noted at this point that since the roll angle of the camera relative to the ground or motor vehicle is at least essentially zero, the vertical direction in the camera images as well as “top” and “bottom” are clearly defined and recognizable.

In the present case, the evaluation algorithm is also trained to estimate actual missing dimensions when the target object is partially obscured in the camera image when the rectangle and other evaluation information are annotated. Furthermore, in embodiments it can also be provided that, in the case of target objects that are not completely visible in the camera image, i.e. that are cut off by the edge of the image, missing parts outside the image area of the camera image can be supplemented, in particular using advance information, in particular dimensional information of the target object, so that the rectangle protrudes from the camera image . However, the examples discussed specifically below do not assume this.

In addition to the rectangle, a vertical, straight dividing line is also determined as separation information by the evaluation algorithm in step S2, which divides the rectangle B in the case of two visible, vertically extending sides of the target object into two partial rectangles, each of which is exactly one of these vertically extending surfaces of the target object, i.e. a lateral side (right or left side), front or rear of the motor vehicle. If only one of these vertical sides of the target object can be seen in the camera image, the dividing line can also correspond to the right or left vertical edge of the rectangle.

As evaluation information, a side classification is also determined for each vertically running side of the target object that is visible in the camera image and can be assigned in particular based on the separation information. This means that for each of these visible, vertical pages it is known whether it is the front, the back, the left side or the right side. It should be noted that the assignment of a side class to a visible, vertical side of the target object inevitably results in other sides, here assuming that the vehicle as the target object stands with its wheels on the ground.

The evaluation algorithm also determines orientation information in the two-dimensional camera image as evaluation information, namely an orientation line in the case of two perspectively visible, vertically extending sides of the target object, and in the case of only one visible, vertically extending side an orientation point to which the orientation line has degenerated. In such a (second) case, the orientation information for the actual derivation of the cuboid is only taken into account indirectly.

Finally, the evaluation algorithm also determines a class of the target object as evaluation information. Since the target objects are vehicles, the class describes in each case whether it is a two-wheeler/motorcycle, a passenger car, a truck, a bus, a tractor or the like. However, in this case there is an even more detailed division of the classes according to the actual size and shape of the vehicles. For example, for cars, a distinction can be made as to whether it is a small car, lower middle class, upper middle class or luxury class and a type of passenger car can also be determined, for example combination power cars, combination sedans, SUVs, ... The same procedure can of course also be used for trucks, buses, two-wheelers, construction vehicles, tractors and the like.

Each class of target objects, for example “cars, small cars, station wagons”, is assigned dimensional information which, on the one hand, includes expected dimensions for target objects of the class, for example an expected reference height, an expected reference width and an expected reference length, but on the other hand also contains intervals , in which the corresponding dimensions usually lie, in particular intervals with minimum and maximum values for height, length and width. It should be noted at this point that dimensional information, but then for certain target objects, can also be assigned to them from previous time steps or from previous points in time. Especially in the case of a motor vehicle, it can be assumed that camera images are recorded with the camera regularly, especially in time steps, and evaluated accordingly. In addition, it is known and common to track target objects between these time steps. If the expansion information, in particular the cuboid, is determined for a target object in a time step for this time step, its dimensions already describe expected dimensions for subsequent time steps, in particular an expected height, an expected width and an expected length. This can remain assigned as dimensional information to the target object if it is tracked and, in particular, can be statistically combined over time in order to obtain increasingly reliable and refined values for the expected dimensions. When it comes to the question of whether dimensional information assigned to the target object from previous time steps or dimensional information assigned to the class of the target object should be used, if the dimensional information from the previous time steps is sufficiently reliable, this is preferable, especially if it comes solely from the Camera image, i.e. without any external further assumption regarding the target object, was determined.

Examples of camera images with annotations provided by the evaluation information are in the 2 until 4 shown. This shows 2 schematically a camera image 1 in which a motor vehicle on a road 3 can be seen as the target object 2. In addition, the rectangle 4, the dividing line 5 determined as separating information and the orientation line 6 determined as orientation information are shown as evaluation information. The orientation line 6 was determined as the lateral side for the right side 7 of the target object 2 that can be seen in the right area of the rectangle 4 divided by the dividing line 5, since a particularly clear, robust detection is possible there due to the contact points of the wheels 8 of the target object 2. The vertical side of the target object 2 that can be seen on the left side is the back side 9. The side classification therefore indicates “back side” for the left area of rectangle 4 and “right side” for the right area of rectangle 4.

As can be seen, in the present example, the rectangle 4 was extended downwards so that the dividing line 5 and the orientation line 6 intersect at the lower edge 10 of the rectangle 4. This is intentional because the wheels 8 also form the clearly lowest point of the target object 2, which is higher at its rear, so that in this way this variation is at least partially covered. This is an example of the fact that the rectangle 4 does not necessarily have to precisely enclose the two-dimensional, visible silhouette of the target object 2, but can be larger if the circumstances make this appear sensible.

It should also be noted that in 2 the class of the target object was determined to be “passenger car, upper middle class, combination vehicle”.

In the example of 3 Another camera image 11 is shown, in which a target object 12, here again a motor vehicle, can be seen, which is located on a surface 13. The evaluation information in turn includes the rectangle 14 and the dividing line 15, which here, since only a vertical side of the target object 12 is visible, coincides with the right edge of the rectangle 14 and is highlighted by dashed lines. Accordingly, in this case only one orientation point 16 is shown as orientation information, which indicates that the orientation of the lateral side, in this case the right side 17 of the target object 12, extends into the image plane. The wheels 18 are also more difficult to recognize because the rims are not visible. In the present case, the page classification contains the information that the front 19 can be seen in the rectangle 14. Furthermore, since the dividing line 15 corresponds to the right edge of the rectangle 14, it can contain the information that the right side 17 would follow there. If the dividing line 15 were, which would also be possible, on the opposite, left edge of the rectangle 14, the left side 20 of the target object 12 would follow accordingly.

The class of target object 12 was determined here as “car, small car, combination sedan”.

4 finally shows a third example of a camera image 21, which in turn is a driving trained target object 22 shows on a road 23. It can be seen that the rear end of the target object 22 is cut off by the edge of the image. This means that the evaluation information here was incomplete and only partially determined. In fact, only two edges of the rectangle 24 could be determined. There is no separation information in this sense, since the right side 27 of the target object 22 cannot be completely seen, see section edge 25. However, the orientation line 26 could be easily determined as orientation information due to the visibility of the wheels 28. The side classification contains at least the information that the partially visible, vertical side of the target object 22 is the right side 27. The fact that the front 29 would be on the left and the (not visible) back on the right is immediately apparent from the fact that the target object 22 with its wheels 28 is on the ground.

The class of target object 22 was recognized as “car, luxury class, SUV”.

Returning to 1 In a step S3, the ground level on which the target object 2, 12, 22 is located is determined, described by corresponding ground information. In the 2 until 4 For the sake of simplicity, the floor level is indicated throughout with the reference number 30. There are several ways to determine them.

On the one hand, it is conceivable to determine the floor plane 30 so that it corresponds to the reference floor surface of the ego object, here your own motor vehicle. This reference ground plane is also known in the three-dimensional camera coordinate system due to external calibration. However, this assumption is particularly less suitable when the entire surrounding ground cannot be approximated accurately enough by a single plane, for example in the presence of ramps.

Therefore, in another variant, provision can also be made for determining a ground level 30 that specifically relates to the target object 2, 12, 22, using sensor data from a distance sensor of the ego object, which is also calibrated extrinsically to the ego object, and/or map data, for example map data from a navigation system of the own motor vehicle as an ego object. Such a distance sensor that scans the ground can be, for example, a radar sensor, a lidar sensor and/or a laser scanner.

After the evaluation information and the floor level 30 have been determined, in a step S4, see again 1 , which determines the extent information. For this purpose, two cases are first distinguished using the separation information when the target object 2, 12 can be seen completely in the camera image 1, 11 (cf. 2 and 3 ). In a first case, cf. 2 , are two vertical sides of the target object 2, in the example the 2 the right side 7 and the back 9 can be seen in perspective in the camera image 1, so that an orientation line 6 could also be determined accordingly. In a second case, however, there is only one vertical side, in the example the 3 the front 19 can be seen in the camera image 11, so that the orientation information was determined as a landmark 16.

If the separation information shows the first case in which the separation line 5 (cf. 2 ) the rectangle 4 is divided into two partial rectangles, each with a positive area, the intersection of the orientation line 6 with the lower edge 10 of the rectangle 4 is first determined as the first calculation point 31, the intersection of the orientation line 6 with the corresponding lateral edge 32 of the rectangle 4 as the second Calculation point 33. The orientation line 6 has an intersection with the right edge 32, since this edge delimits the partial rectangle that contains the right side 7 for which the orientation line 6 was determined.

According to the pinhole camera model, which is used here as the recording geometry model, all points in the three-dimensional camera coordinate system that are projected onto a point in the camera image 1 lie on a ray. This can be calculated using intrinsic camera calibration. In the present case, this happens for the calculation points 31 and 32. The intersection points of these rays with the ground plane 30 in the three-dimensional camera coordinate system are now calculated and form a first ground corner point of the searched cuboid for the first calculation point 31 and a second ground corner point of the searched cuboid for the second calculation point 33. The ground corner points are connected by a connecting line. A clear search line is now determined which runs in the floor plane 30 and through the first floor corner point and forms a right angle with the connecting line.

On the search line, in turn, there is a clear third ground corner point, the projection of which lies in the camera image 1 on the straight line that results from the extension of the left side edge 34 of the rectangle 4, which is not intersected by the orientation line 6. If the projection of the third floor corner point determined in this way is not on the left edge 34, but outside the rectangle 4, a correction of the floor plane 30, the rectangle 4 and/or the calculation points 31 and/or 33 must be made in such a way that after recalculation of the first , second and third floor corners point the projection of the third floor corner point in the camera image 1 comes to lie on the edge 34.

If the third floor corner point, more precisely its three-dimensional position in the camera coordinate system, has been determined in this way, the fourth floor corner point completing the floor corner points of the floor surface of the cuboid can also be easily determined by forming the sum of the vectors for the third floor corner point and the second floor corner point and from this the vector is subtracted to the first ground corner point.

In this regard, it is also checked here whether the projection of the fourth floor corner point lies in rectangle 4. If this is not the case, a correction can be made to the floor plane 30, the rectangle 4, the first calculation point 31 and/or the second calculation point 33, so that after recalculation of the floor corner points, the third and fourth floor corner points meet the described projection requirements.

There is now a rectangular floor surface of the cuboid, described by the floor corner points, which lies completely in the floor plane 30. To obtain the complete cuboid, the height of the cuboid is now determined, which is chosen as large as possible and in such a way that the projections of the set of all roof points, which result from the floor corner points by shifting the height in the direction of the upward normal unit vector of the floor plane 30 , in which camera image 1 is all contained in rectangle 4. The roof surface formed from the corresponding set of roof vertices forms the upper end of the cuboid.

In the second case, cf. 3 , the dividing line 15 corresponds to the left or right edge of the rectangle 14. In this case, the lower left corner of the rectangle 14 is selected as the first calculation point 35 and the lower right corner of the rectangle 14 is selected as the second calculation point 36, which here also corresponds to the orientation point 16. Analogous to the first, using 2 In the case discussed, the corresponding rays and the intersection points with the ground plane 30, i.e. the first ground corner point and the second ground corner point, can in turn be determined using the recording geometry model. Two search lines are then determined, which run through the respective first and second floor corner points in the floor plane 30 perpendicular to the straight line connecting the first and second floor corner points.

If the projection of one of these search lines from the viewer, i.e. the position of the camera at the time the camera image 11 was taken, immediately runs out of the rectangle 14, starting from the first calculation point 35 or the second calculation point 36, a correction step is carried out and the floor plane 30 and / or the rectangle 14 are corrected so that after recalculation of the first and second ground corner points and the search line, the projections of the search line starting from the first calculation point 35 or second calculation point 36 initially run within the rectangle 14.

As mentioned at the beginning, dimensional information with expected dimensions is now known either based on the class of the target object 12 according to the evaluation information or from previous time steps. Since in the present case only the front 19 of the target object 12 can be seen in the camera image 11, the reference length or expected length perpendicular to it is selected as the expected dimension in the dimensional information as a distance that must be walked along the search line away from the viewer in order to reach the third and to find the fourth ground corner point with this assumed reference length. Dimensional information is preferably used, which is determined from an earlier determination of the extent information when two vertically extending sides of the target object 12 were seen in the camera image, as already described.

If only the right or left side of the target object 12, i.e. a lateral side, can be seen in a camera image 11, the reference width is derived from the dimensional information and used.

The roof vertex points for the roof area are determined as in the first case.

Through the side classification, in order to complete the extent information, it is now also possible to assign sides of the target object 2, 12, 22 to the corresponding sides of the cuboid, in particular to determine which of the ground corner points corresponds to the front/rear left/lower right corner point of the cuboid and accordingly for the roof checkpoints.

It should be noted that if the dimensional information assigned to the class of the target object 2, 12, 22 also contains the intervals described, a further plausibility check of the dimensions of the cuboid obtained and, if necessary, a correction can be carried out. To do this, the dimensions of the specific cuboid are compared with the respective reference intervals for the length, width and height of target objects for the class and, if dimensions are outside the intervals, appropriate measures are taken, for example the dimensions, in particular symmetrical ric, set to the minimum or maximum value of the interval or, in the case of larger deviations, the result is even rejected.

Even in the special case of 3 , in which the target object 22 cannot be completely seen on the camera image 21, can be done in step S4 1 An extent information can be determined after partial information can be used in such cases, which can then be supplemented by suitable dimensional information. That's how it is 3 only the second computing point 37 is visible in the camera image 21; after the lower edge of the rectangle 24 is missing. Nevertheless, the corresponding second ground corner point can again be determined for the second calculation point 37. Using the orientation line 26, a search beam can be calculated which extends away from the ground plane 30 from the second ground corner point. You can now walk along this search beam for a distance that corresponds to a reference length from the dimensional information in order to determine the first ground corner point. You can then continue as in the second case, with the distance along the search line then corresponding to a reference width.

Returning to 1 In step S5, the specific extent information can be used, which describes the position, orientation and extent of the corresponding target object 2, 12, 22. While such specific extent information can be used, for example, as training data in the sense of a three-dimensional annotation for trained functions, the extent information can preferably also be used in a vehicle function of one's own motor vehicle as an ego object.

5 shows schematically a motor vehicle 38 according to the invention in the form of a schematic diagram. The motor vehicle 38 has the camera 39, which in the present case is a front camera and is directed towards the front of the motor vehicle 38 and which supplies the camera images. This is therefore connected to a control device 40, which is designed to carry out the method according to the invention. The control device or another control device can also be designed to carry out a vehicle function that, as described in step S5, uses the extent information, in particular in an at least partially automatic vehicle guidance function. Further components of the motor vehicle 38 can include a navigation system 41, which can provide digital map data for determining the ground level 30, as well as distance sensors 42, which can also provide information in this regard.

The control device 40 comprises, in addition to at least one storage means for carrying out the method according to the invention, in particular an interface and/or control unit for controlling the camera 39 or for receiving the two-dimensional camera image 1, 11, 21 (step S1), an evaluation unit for applying the evaluation algorithm to the Camera image 1, 11, 21 for determining the evaluation information according to step S2, a ground level determination unit for determining the ground information describing the ground level 30, as described in step S3, and an extent information determination unit for determining the extent information according to step S4.

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• EP 3594002 A1 [0007]

Claims (18)

Method for determining three-dimensional expansion information of a target object (2, 12, 22), in particular a vehicle, from a two-dimensional camera image (1, 11, 21) showing the target object (2, 12, 22), which has a camera (39) which is intrinsically and extrinsically calibrated with respect to an ego object carrying the camera (39), in particular a motor vehicle (38), is recorded, with ground information relating to the three-dimensional position of the ground plane (30) under the target object (2, 12, 22) to the camera (39) and a rectangle (4) which is determined by evaluating the camera image (1, 11, 21) using an evaluation algorithm and which encloses the target object (2, 12, 22) in the camera image (1, 11, 21), 14, 24) describing the two-dimensional evaluation information, the three-dimensional expansion information is determined, characterized in that - as part of the evaluation information, an additional orientation of the target object (2, 12, 22) for at least one vertically extending side of the target object (2, 12, 22) at the lower edge of which in the camera image (1, 11, 21) orientation information, in particular depending on the image content of the camera image (1, 11, 21), an orientation line (6, 26) or an orientation point (16), is determined, and - for at least one computing point (31, 33, 35, 36, 37) of the camera image (1, 11, 21), determined by means of the orientation information and the rectangle (4, 14, 24), the three-dimensional position of an assigned one lying on the ground level (30). The ground corner point of the ground surface of a cuboid at least partially enclosing the target object (2, 12, 22) is determined using a recording geometry model of the camera (39) and from this the expansion information is determined as at least the ground surface of the cuboid covered by the target object (2, 12, 22), especially when the cuboid is reconstructed. Procedure according to Claim 1 , characterized in that the ground level (30) under the target object (2, 12, 22) using the extrinsic calibration of the camera (39) - by assuming the same position as the ground level under the ego object or - from sensor data from another, as well is determined extrinsically with the ego object registered distance sensor (42), in particular a laser scanner and / or radar sensor, and / or using map data. Procedure according to Claim 1 or 2 , characterized in that a pinhole camera model is used as the recording geometry model. Method according to one of the preceding claims, characterized in that - for at least one vertically extending side of the target object (2, 12, 22), in particular at least the at least one side to which the orientation information is assigned, a page classification in front ( 19, 29), back side (9) or lateral side, in particular right side (7, 17) or left side (20, 27), is determined and/or - that the orientation information is present in more than one in the camera image (1, 11, 21) visible, vertical side of the target object (2, 12, 22) as an orientation line (6, 26) along the lower edge of one of the sides visible in perspective in the camera image (1, 11, 21), in particular a lateral side, of the target object ( 2, 12, 22), in the case of a vehicle as a target object (2, 12, 22), in particular as an orientation line (6, 26) touching the lower edge of wheels (8, 18, 28) of the vehicle, and/or - that with only one vertical side of the target object (2, 12, 22) visible in the camera image (1, 11, 21), the orientation information is determined as an orientation point (16) indicating an orientation extending into the image plane. Method according to one of the preceding claims, characterized in that the evaluation information is separating information describing the position of at least one vertically extending side of the target object (2, 12, 22) that is visible in the camera image (1, 11, 21), in particular a vertically extending separating line (5, 15) between different vertically extending sides of the target object (2, 12, 22) in the camera image (1, 11, 21) is comprehensively determined. Procedure according to Claim 5 , characterized in that with two vertically extending sides of the target object (2, 12, 22) visible in the camera image (1, 11, 21) displaying separation information and an orientation line (6, 26) related to one of the visible sides, in particular a lateral side ) as orientation information for determining the floor area covered by the target object (2, 12, 22) on the floor level (30) - a first calculation point (31) as the intersection of the orientation line (6, 26) with the lower edge (10) of the rectangle ( 4, 14, 24) in the camera image (1, 11, 21) and a second calculation point (33) as the intersection of the orientation line (6, 26) with that of the side to which the orientation line (6, 26) is related, according to vertical edge (32) of the rectangle (4, 14, 24) assigned to the separation information are determined, - determined by means of the recording geometry model for the first and second computing points (31, 33) rays on which the computing points (31, 33) lie and the respective intersection points of the rays with the floor plane (30) are determined as first and second floor corner points in three-dimensional space, - a search line lying perpendicular to a connecting line in the floor plane (30) connecting the first and second floor corner points and running through the first floor corner point and a search line on the Third ground corner point lying in a straight line, the projection of which can be determined based on the recording geometry model into the camera image (1, 11, 21) on the vertical edge (34) opposite the edge (32) of the side to which the orientation line (6, 26) is related Rectangle (4, 14, 24) in the camera image (1, 11, 21) is determined in three-dimensional space, and - a fourth floor corner point completing the rectangular floor surface is determined from the first, second and third floor corner points, in particular by addition the vectors associated with the third and second ground corner points and subtraction of the vector associated with the first ground corner point. Procedure according to Claim 6 , characterized in that in a projection on an extension of the corresponding edge (34) outside the rectangle (4, 14, 24) having a certain third bottom corner point and / or in a projection outside the rectangle (4, 14, 24) having a certain fourth Ground corner point, the ground plane (30) and / or the orientation line (6, 26) and / or the rectangle (4, 14, 24) and / or the first and / or second calculation point (31, 33) are corrected in such a way that the respective Projections come to lie on the edge (34) or within the rectangle (4, 14, 24). Procedure according to one of the Claims 5 until 7 , characterized in that as part of the evaluation information, a class of the target object (2, 12, 22), to which dimensional information describing at least one expected dimension of a target object (2, 12, 22) of the class is assigned, is determined and / or the target object ( 2, 12, 22) a dimension information describing at least one expected dimension of the target object (2, 12, 22) is assigned from a previous determination of the extent information. Procedure according to Claim 8 , characterized in that with only one vertical side of the target object (2, 12, 22) visible in the camera image (1, 11, 21) displaying separation information for determining the distance through the target object (2, 12, 22) on the ground level ( 30) covered floor area - a first calculation point (35) as the lower left corner point of the rectangle (4, 14, 24) in the camera image (1, 11, 21) and a second calculation point (36) as the lower right corner point of the rectangle (4, 14, 24) are determined in the camera image (1, 11, 21), - rays on which the computing points (35, 36) lie are determined by means of the recording geometry model for the first and second computing points (35, 36) and the respective intersection points of the rays with the floor plane (30) are determined as first and second floor corner points in three-dimensional space, and - two search lines lying perpendicular to a connecting line in the floor plane (30) connecting the first and second floor corner points through the respective floor corner points and third and fourth ground corner points are determined along the respective search line in a direction away from the camera (39) when the camera image (1, 11, 21) is recorded at a distance in three-dimensional space determined according to the dimensional information. Procedure according to Claim 9 , characterized in that when the search line initially runs out of the rectangle (4, 14, 24) directly in its projection into the camera image (1, 11, 21), the ground plane (30) and / or the rectangle (4, 14, 24) be corrected in such a way that the projections of the search line extend at least up to the distance within the rectangle (4, 14, 24) in the camera image (1, 11, 21). Procedure according to Claim 9 or 10 , characterized in that if a page classification is present as part of the evaluation information, the expected dimension in the direction perpendicular to the vertically running, classified page visible in the camera image (1, 11, 21) is selected from the dimensional information as a distance. Procedure according to one of the Claims 8 until 11 , characterized in that if the target object (2, 12, 22) is not completely recorded in the camera image (1, 11, 21), in particular if there is only one determinable ground corner point, missing information for reconstructing the ground surface is taken from the dimensional information. Method according to one of the preceding claims, characterized in that a roof area of the cuboid assigned to the floor area is determined by the set of roof vertices resulting from the addition of a common height value multiplied by the normal unit vector of the floor plane (30) to the floor corners, whose projections into the camera image (1 , 11, 21) is determined at the maximum height value for at least one roof checkpoint, in particular all roof checkpoints, within the rectangle (4, 14, 24). Method according to one of the preceding claims, characterized in that a trained function, in particular a neural network, is used as the evaluation algorithm, which was trained in particular with manually annotated camera images as training data. Method according to one of the preceding claims, characterized in that for at least some of the target objects (2, 12, 22) that are not completely recorded in the camera image (1, 11, 21), the evaluation algorithm, in particular based on appropriate training, is used to estimate the complete The rectangle (4, 14, 24) protruding from the camera image (1, 11, 21) is formed. Motor vehicle (38), comprising a control device (40) designed to carry out a method according to one of the preceding claims and the camera (39). Computer program that contains the steps of a method according to one of the Claims 1 until 15 performs when it is executed on a computing device. Electronically readable data carrier on which a computer program can be written Claim 17 is stored.

Images