DE102018202244A1 - Method for imaging the environment of a vehicle - Google Patents

Abstract

A method (300) for imaging the environment of a vehicle (100) comprising the steps of: detecting (302) an image sequence by means of a monocamera; Splitting (303) at least one image of the image sequence into columns; Determining (304, 304-a to 304-f) segment data in at least one column; Mapping (305) the environment based on the segment data; and outputting (306) at least one signal representing the imaged environment. The gist of the invention is that the step of determining (304) the segment data in at least one column comprises a substep of the mapping (304-e) of pixels of the at least one column to a predetermined segment type.

Description

The present invention relates to a method for imaging the surroundings of a vehicle, a computer program for carrying out the method and a device for imaging the surroundings of a vehicle.

State of the art

The imaging of the surroundings of a vehicle by means of a camera system is used in driver assistance system, for example, to identify parking spaces, trigger emergency braking function or to navigate partially autonomously driving or autonomously driving vehicles through the environment. To image the environment, at least one image captured by means of the camera system can be segmented into segments, so-called stixels, in columns. A segment describes a small area in the environment.

From the DE 10 2009 009 047 A1 , the DE 10 2011 111 440 A1 , the DE 10 2012 000 459 A1 and Schneider, L. et al., Semantic Stixels: Depth is not Enough. Intelligent Vehicles Symposium (IV), 2016 IEEE. IEEE, 2016, pages 110-117 Methods are known in which each captured by a stereo camera images are segmented column by column into stixels. Here, the formation of the stixels is based on a range estimate for each pixel, which is calculated from the two images of the stereo camera.

The DE 10 2016 106 293 A1 discloses a method and system for detecting an object, comprising the steps of: receiving, by a processor, image data from a single camera, the image data representing an image of a scene; Determining, by the processor, stixel data from the image data; Detecting by the processor of an object based on the stixel data; and selectively generating by the processor a warning signal based on the detected object. The DE 10 2016 122 190 A1 discloses a similar method in which radar data is further received by a radar system and wherein the image data and radar data are processed by the processor using a deep learning method. With both methods it is possible to identify the first object in each column.

Disclosure of the invention

The present invention is based on a method for imaging the surroundings of a vehicle. The method comprises the steps of capturing an image sequence using a monocamera, splitting at least one image of the image sequence into columns, determining segment data in at least one column, mapping the environment based on the segment data, and outputting at least one of the imaged environment representative signal.

According to the invention, the step of determining the segment data in at least one column has a partial step of assigning pixels of the at least one column to a predetermined segment type.

The advantage of the invention is that differences in the pixels within a column can be taken into account. Differences of the pixels with regard to their distance to the image plane, movement or semantic class within a column can be taken into account. These differences can be considered in the assignment to a segment, whereby a segment is assigned to a given segment type. This makes it possible to identify multiple objects in a column. The environment of the vehicle can be imaged with higher accuracy. It is sufficient to capture an image sequence using a single camera. In particular, it is sufficient to capture an image sequence by means of a monocamera. This makes the process more cost effective than processes where multiple cameras are needed. The method disclosed here is less expensive than a method in which a stereo camera is necessary.

In an advantageous embodiment of the invention, it is provided that a predetermined segment type is a potentially movable object. A potentially movable object may be an object that is movable. A potentially moving object can be an object that can move. A potentially moving object may move during the capture of the image sequence. A potentially mobile object may linger at one point in an image or in multiple images of the image sequence during the capture of the image sequence. A potentially moving object may be positioned at one location in an image or in multiple images of the image sequence during capture of the image sequence. A potentially movable object may be a vehicle such as a car, a truck, a motorcycle, a bicycle or a scooter. A potentially movable object may for example also be a stroller, a handcart or the like. For example, a potentially mobile object may also be a person. For example, a potentially mobile object may also be an animal.

Another given segment type can be a static object. A static object can be an object that is modified Conditions remain largely unchanged. A static object may be positioned at a location in an image or in multiple images of the image sequence during capture of the image sequence. Parts of a static object may move during the capture of the image sequence, with the static object as a whole remaining positioned at one point in one image or in multiple images of the image sequence during acquisition of the image sequence. Parts of a static object may move during the acquisition of the image sequence, with at least one fixed point of the static object remaining positioned during the acquisition of the image sequence at a location in one or more images of the image sequence. A static element may be, for example, a building, a mast, a traffic sign, a lighting system, a part of the vegetation or the like.

Another given segment type can be sky. Another given segment type may be ground. A segment type can be defined by its geometry. A segment type can be defined by its orientation. For the segment type sky, the geometry may be such that a sky segment is infinitely far away. For the segment type of floor, the geometry may be such that a floor segment is considered lying. For the segment type static object, the geometry may be such that the orientation of the segment of the static object is defined perpendicular to the ground. For the segment type potentially moving object, the geometry may be such that the orientation of the segment of the potentially mobile object is defined perpendicular to the ground.

The advantage of this embodiment is that the accuracy of imaging the environment of the vehicle can be increased. In particular, objects can be clearly distinguished from one another by the distinction between static and potentially movable objects.

In a further advantageous embodiment of the invention, it is provided that the step of determining the segment data in at least one column has a further sub-step of forming at least one segment in a column. In this case, the formation of a segment takes place as a function of the segment types assigned to the respective pixels and a respective distance of the pixels to the image plane of the one image and to a variable characterizing the respective movement of a pixel. The advantage of this embodiment is that the environment of the vehicle can be imaged with greater accuracy. Multiple objects in one column can be identified with high accuracy. Due to the dependence on a respective distance of the pixels to the image plane, a deep structure of the environment can be detected. Thus, for example, at least two segments may be formed in a column, with, for example, a segment associated with a first potentially dynamic object having a first distance of the pixels to the image plane, and an at least second segment being associated with a second potentially dynamic object having a second distance of the pixels Image level is assigned. The size characterizing the particular motion of a pixel can also greatly enhance the imaging of the environment. In this case, for example, two different speeds of two vehicles can be taken into account.

In a further advantageous embodiment of the invention, it is provided that the step of determining the segment data in at least one column comprises a further sub-step of determining an expected optical flow of at least one pixel of the at least one column by means of a homography. In particular, the step of determining the segment data in at least one column comprises a further sub-step of determining an expected optical flow of at least one segment of the at least one column by means of homography. As an optical flow, a movement of a pixel between two images can be understood. A homography can be understood as the image of the segment level between two images. Homography can be understood as the projection of the segmental plane between two images. Alternatively to the explicit description by means of homography, a heuristic determination of the expected optical flow can also be carried out. The advantage of this embodiment is that it allows a distinction of segments based on the expected optical flow. The assignment of pixels to a given segment type, such as sky, ground, static object or potentially movable object can be made possible. The assignment of pixels to segments with different distances to the image plane can be made possible. The assignment of pixels to segments with different movement can be made possible.

In a further advantageous embodiment of the invention it is provided that the expected optical flux is determined taking into account at least one predetermined assumption. A given assumption may be the geometry of the segment type. A given assumption may be the motion of the segment type. A given assumption may be the distance of the segment to the image plane. A given assumption may be the movement of the segment. The advantage of this embodiment is that the Number of parameters for determining the expected flow is reduced. The computational effort of the described method can be reduced. For example, it is possible to avoid unnecessarily determining movement for a static object. The accuracy of the determination of the segment data can be increased.

In a further advantageous embodiment of the invention, it is provided that the determination of the expected optical flow takes place taking into account at least one signal representing a movement of the camera. The advantage of this embodiment is that the expected optical flux can be determined with higher accuracy.

In a further advantageous embodiment of the invention, it is provided that the step of determining the segment data in at least one column has a further sub-step of comparing the expected optical flow of a pixel with an optical flow of the same pixel determined on the basis of the acquired image sequences. The optical flux of a pixel determined on the basis of the acquired image sequences can here be understood as a variable characterizing the respective movement of a pixel. The advantage of this embodiment is that the assignment of pixels to a given segment type can be done with high accuracy. The assignment of pixels to segments with different distances to the image plane can be done with high accuracy. The assignment of pixels to segments with different motion can be done with high accuracy.

In a further advantageous embodiment of the invention, it is provided that the assignment of pixels of the at least one column to a predetermined segment type by means of a minimization of an energy term happens. The minimization of the energy term can be done via dynamic programming. The advantage of this embodiment is that the computational effort of the described method can be reduced.

In a further advantageous embodiment of the invention, it is provided that the assignment of pixels of the at least one column to the segment type of a potentially movable object is dependent on an assignment of each pixel of the at least one column to a predetermined semantic class. For example, a given semantic class can be understood as a class of objects of similar properties. For example, a given semantic class can be assigned a sky. For example, road, sidewalk or terrain may be assigned to another predetermined semantic class. For example, vehicles or persons can be assigned to another predetermined semantic class. For example, buildings, masts, signs or vegetation can be assigned to another predetermined semantic class. The advantage of this embodiment is in particular that the distinction between static and potentially movable objects can be supported. The mapping of the environment can be done with greater accuracy.

In a further advantageous embodiment of the invention, it is provided that the step of determining the segment data in at least one column has a further sub-step of determining the distance of the pixels of potentially movable objects to the image plane by means of a vehicle environment model. Under a vehicle environment model, the knowledge of typical structures of street scenes can be understood. For example, it is assumed that potentially moving objects such as vehicles or pedestrians in street scenes are typically on the ground. The advantage of this embodiment is that the distinction of segments of potentially moving objects with different distances to the image plane is made possible even if the expected optical flow of these segments is the same.

The invention is further based on a computer program which is set up to carry out all steps of the described method.

The invention is further based on a machine-readable storage medium on which the computer program is stored.

The invention is further based on a device for imaging the surroundings of a vehicle. The device has at least one monocamera for capturing an image sequence, at least one module for splitting at least one image of the image sequence into columns, at least one module for determining segment data in at least one column, at least one module for imaging the environment based on the segment data. Data and at least one module for outputting at least one signal representing the imaged environment. The module for determining segment data in at least one column is designed to assign pixels of the at least one column to a predetermined segment type.

In an advantageous embodiment of the invention, the module for determining segment data in at least one column is further adapted to determine an expected optical flow of at least one pixel of the at least one column by means of a homography. In particular, the module for determining the segment Data in at least one column designed to determine an expected optical flow of at least one segment of the at least one column by means of a homography.

list of figures

• 1 ein Fahrzeug mit einem Ausführungsbeispiel einer Vorrichtung zur Abbildung der Umgebung;
• 2 ein Ausführungsbeispiel eines Steuergeräts auf der ein Verfahren zur Abbildung der Umgebung eines Fahrzeugs ablaufen kann;
• 3 ein Ausführungsbeispiel eines Verfahrens zur Abbildung der Umgebung eines Fahrzeugs.

Hereinafter, embodiments of the present invention will be explained in more detail with reference to the accompanying drawings. Like reference numerals in the figures indicate the same or equivalent elements. Show it:

• 1 a vehicle with an embodiment of a device for mapping the environment;
• 2 an embodiment of a control device on which a process for mapping the environment of a vehicle can run;
• 3 An embodiment of a method for imaging the environment of a vehicle.

1 shows an example of a vehicle 100 with an embodiment of a device 110 for imaging the environment of the vehicle 100 , The device 110 assigns the mono camera 101 on. The mono camera 101 can be a picture sequence of the environment of the vehicle 100 to capture. The device 110 also has at least one control unit 102 on which the procedure for imaging the environment of a vehicle is running. From the mono camera 101 becomes at least one signal representing the image data of the image sequence 105 to the control unit 102 transmitted. The control unit 102 can be designed to at least one signal representing the image data of the image sequence 105 to recieve. The control unit 102 may be configured to split at least one image of the image sequence into columns. The control unit 102 may be further configured to determine segment data in at least one column. The control unit 102 may be further adapted to the environment of the vehicle 100 based on the segment data. The control unit 102 can also be designed to at least one signal representing the imaged environment 106 . 107 issue. A signal representing the imaged environment may be in the form of a warning signal, for example 106 to a warning module 104 of the vehicle 100 be issued. A warning module 104 For example, it may be configured to be an occupant of the vehicle 100 in front of one in the vicinity of the vehicle 100 to warn the detected object. A signal representing the imaged environment may be in the form of a control signal, for example 107 to a control module 103 of the vehicle 100 be issued. A control module 103 For example, it may be configured to the vehicle 100 driving. Furthermore, the vehicle 100 at least one sensor 108 for detecting a movement of the monocamera 101 respectively. From the sensor 108 can at least one signal representing the movement of the mono camera 109 to the control unit 102 be transmitted. The control unit 102 can be designed to at least one signal representing the movement of the monocamera 109 to recieve.

2 shows an embodiment of a control device 102 on which a process for mapping the environment of a vehicle can proceed. As just described, the controller 102 a part of in 1 described device 110 his. How out 2 it can be seen, the control unit 102 have several modules. In addition, in the control unit 102 given data 207 be deposited. For example, in the control unit 102 at least one predetermined segment type 207 - 1 be deposited. It can be in the control unit 102 at least one given assumption 207 - 2 be deposited to determine an expected optical flow. It can be in the control unit 102 at least one given semantic class 207 - 3 be deposited. It can be in the control unit 102 at least one vehicle environment model 207 - 4 be deposited.

The control unit 102 can be a module 201 for splitting at least one image of the image sequence into columns. The module 201 may be designed to be a signal representing the image data of the image sequence 105 to recieve. The control unit 102 can continue a module 202 for determining segment data in at least one column. The module 201 can be designed to be a signal 209 representing a columnized image to the module 202 to convey. The module 202 may be configured to signal 209 to recieve. The module 201 and the module 202 may be formed as a common control unit. The module 201 and the module 202 can work together as the module 208 be designed to determine segment data in at least one image of an image sequence. The control unit 102 can continue a module 205 for determining an optical flow of at least one pixel of an image of a captured image sequence. The module 205 may be designed to be a signal representing the image data of the image sequence 105 to recieve. The module 205 can be designed to be a signal representing the determined optical flow of at least one pixel of an image 210 to the module 202 to convey. The module 202 may be configured to signal 210 to recieve. The control unit 102 can continue a module 206 for assigning at least one pixel of at least one image of the image sequence to a predetermined semantic class. The module 202 For example, it may be designed to have at least one predetermined semantic class 207 - 3 representing signal 212 - 3 to recieve. The module 206 may be designed to be a signal representing the image data of the image sequence 105 to recieve. The module 206 For example, it can be designed to have a signal representing the assignment of at least one pixel of at least one image of the image sequence to a given semantic class 211 to the module 202 to convey. The module 202 may be configured to signal 211 to recieve. The module 202 may be further adapted to given data 207 to recieve. The module 202 For example, it may be configured to receive a signal representing a predetermined data 212 to recieve.

The module 202 Furthermore, it can be designed to send a signal representing the segment data determined in at least one column to a module 203 to convey the picture of the environment. The module 203 can also be part of the controller 102 his. The module 203 For example, you can merge the multi-column segment data into one image. Such an image can map the surroundings of the vehicle. The module 203 may be adapted to a signal representing the imaged environment to a module 204 for outputting the at least one signal representing the imaged environment 214 to convey. The module 204 may be the signal representing the imaged environment 214 for example in the form of a warning signal 106 output. The module 204 may be the signal representing the imaged environment 214 for example in the form of a control signal 107 output.

mit $$s_i=\left(v_i^b,v_i^t,m_i,p_i,t_i,c_i\right)$$

3 shows an embodiment of a method 300 for imaging the environment of a vehicle. The procedure 300 For example, on a device 110 as they are in 1 shown is expire. The procedure 300 starts in step 301 , In step 302 it comes to capturing an image sequence of the environment of the vehicle. In step 303 it comes to the division of at least one image of the image sequence in columns. The columns can be a fixed width w _s respectively. The width w _s can be the same for each of the columns. In step 304 it comes to the determination of segment data in at least one of the columns. The determination of segment data thus becomes a one-dimensional optimization problem. This optimization problem can be solved independently for each column. The determination of segment data can be done as follows: $$s=\left\{s_i|1\le i\le N\le H\right\}$$

With $$s_i=\left(v_i^b.v_i^t.m_i.p_i.t_i.c_i\right)$$

s

= Menge der Segmente in dieser Spalte

s_i

= Segment

i

= Index des Segments

N

= Anzahl der Segmente in dieser Spalte

h

= Anzahl der Bildpunkte entlang der Spalte

= untere Bildkoordinate des Segments i

= obere Bildkoordinate des Segments i

m_i

= Segment-Typ des Segments i

p_i

= inverse Entfernung des Segments i zur Bildebene

t_i

= Bewegung des Segments i

c_i

= semantische Klasse des Segments i.

Herein mean:

s

= Amount of segments in this column

s _i

= Segment

i

= Index of the segment

N

= Number of segments in this column

H

= Number of pixels along the column

= lower image coordinate of the segment i

= upper image coordinate of the segment i

m _i

= Segment type of the segment i

p _i

= inverse distance of the segment i to the image plane

t _i

= Movement of the segment i

c _i

= semantic class of segment i.

The determination of segment data in at least one of the columns can be formulated as an energy minimization problem: $$\begin{array}{l}\widehat{s}=\underset{s}{\text{min}}e\left(s.f.l\right)\hfill \\ =\underset{s}{\text{min}}\left(\mathrm{\Psi }\left(s\right)+\mathrm{\Phi }\left(s.f.l\right)\right)\hfill \end{array}$$

E =

Energie

ŝ =

ermittelte Menge der Segmente in dieser Spalte

Ψ(s) =

Fahrzeugumfeldmodell

Φ(s,f, l) =

Datenwahrscheinlichkeit

f =

optischer Fluss

I =

Zuordnung eines Bildpunktes zu einer vorgegebenen semantischen Klasse.

Herein mean:

E =

energy

ŝ =

determined amount of segments in this column

Ψ (s) =

Vehicle environment model

Φ (s, f, l) =

data probability

f =

optical flow

I =

Assignment of a pixel to a given semantic class.

The vehicle environment model Ψ (s) describes the knowledge of typical structures of street scenes. The term Ψ (s) leads to the preferred assignment of a pixel to a given segment type based on the geometry of the segment type. For example, an assignment based on the geometry that is most likely for a typical structure of a street scene is preferred. For example, if a segment in a column is assigned to the segment type ground, then with a certain probability, a next segment may be the segment type object either the segment type static object or the segment type potentially movable object. This is based on the knowledge of typical structures of street scenes, according to which objects usually stand on the ground. For example, the vehicle environment model may result in more accurate mappings in the following case: If a previous segment is associated with the segment type static object or the segment type potentially mobile object, and the next segment may also be the segment type static object or the segment type potentially moving object, it can be assumed with a certain probability that the lower image coordinate of the next segment does not represent the lower edge of the next object. It can be assumed that in this case one object covers another.

The vehicle environment model may be deposited as described above in the apparatus in which the method is carried out. In the partial step 304-a of the step 304 it comes to determining a respective distance of the pixels to the image plane by means of a vehicle environment model. In the partial step 304-a In particular, it is necessary to determine a respective distance of the pixels for dynamic objects by means of a vehicle environment model. This makes it possible to determine a deep structure.

In partial step 304-b it comes to the determination of a variable characterizing the respective movement of a pixel. A variable characterizing the respective movement of a pixel can be an optical flow of a pixel determined on the basis of the acquired image sequence. The determination of a variable characterizing the respective movement of a pixel can be based on the movement of the monocamera.

mit $$H_i=K\left(R_{cam}-p_i{t}_{cam,i}{n}_i^T\right)K^{-1}$$

In partial step 304-c it comes to the determination of an expected optical flow f _v at least one pixel of the at least one column by means of a homography. In partial step 304-c In particular, it is possible to determine an expected optical flow f _{v of} at least one segment of the at least one column by means of homography. Since a segment is considered a planar part, the expected optical flow f _v for an assumption of optical flow can be described as follows: $$\widehat{f_v}=H_i{x}_v-x_v$$

With $$H_i=K\left(R_{cam}-p_i{t}_{cam.i}{n}_i^T\right)K^{-1}$$

H_i

= Homographie

x_v

= Bildpunkt

K

= intrinsische Kameramatrix

R_cam

= Rotationsmatrix der Kamerabewegung

t_cam,i

= Translationsvektor der Kamerabewegung

= Normalenvektor.

Herein mean:

H _i

= Homography

x _v

= Pixel

K

= intrinsic camera matrix

R _cam

= Rotation matrix of the camera movement

t _{cam, i}

= Translation vector of the camera movement

= Normal vector.

ist definiert durch die Geometrie eines Segment-Typs. Der Normalenvektor $n_i^T$

kann liegend sein. Der Normalenvektor $n_i^T$

kann stehend sein. Hierbei wird angenommen, dass ein Segment mittig der Monokamera zugewandt ist. Für den Segment-Typ statisches Objekt sind die Rotationsmatrix der Kamera R_cam und der Translationsvektor der Kamera t_cam,i aufgrund der Kamerabewegung vorgegeben. Für den Segment-Typ Himmel ist die inverse Entfernung des Segments zur Bildebene p_i = 0. Hierdurch vereinfacht sich für diesen Segment-Typ die Homographie zu H_i = K R_cam K^-1. Im Gegensatz dazu muss für den Segment-Typ potentiell bewegliches Objekt auch eine Translationsbewegung t_i des Segments selbst berücksichtigt werden. Der erwartete optische Fluss ist mittels einer Homographie beschreibbar. Der Translationsvektor der Kamera t_cam,i
kann hierbei als relative Translation zwischen der Kamera und der Annahme bezüglich eines Segments s_i angesehen werden.The normal vector $n_i^T$

is defined by the geometry of a segment type. The normal vector $n_i^T$

can be lying down. The normal vector $n_i^T$

can be upright Here it is assumed that a segment is centered facing the monocamera. For the segment type static object are the rotation matrix of the camera R _cam and the translation vector of the camera t _{cam, i} dictated by the camera movement. For the segment type sky, the inverse distance of the segment is to the image plane p _i = 0. This simplifies the homography for this segment type H _i = KR _cam K ^-1 . In contrast, for the segment type potentially moving object also has a translational motion t _i of the segment itself. The expected optical flux can be described by means of homography. The translation vector of the camera t _{cam, i}
can here as relative translation between the camera and the assumption with respect to a segment s _i be considered.

Furthermore, a coincidence of an assumption regarding the optical flow can be judged. This is what happens in the partial step 304-d of the step 304 to compare the expected optical flux of the segment s _i a pixel with an optical flow of the same pixel determined on the basis of the acquired image sequence. Here, in particular, the sub-steps 304-b determined a characterizing the movement of a pixel size. The following energy term can be used for the comparison: $${\mathrm{\Phi }}_F\left(s_i.f_{\nu }.\nu \right)=\text{min}\left({\alpha }_F.\text{log}\left(\text{det}{C}_{\nu }\right)+\frac{r_{i.\nu }^T{C}_{\nu }^{-1}{r}_{i.\nu }}{2}\right)$$

Φ_F(s_i, f_v, v)

= Energie basierend auf dem ermittelten optischen Fluss f_v bei vorgegebenem Segment s_i in der Reihe v der Spalte

α_F

= Konstante zur Beschränkung der des Energieanteils

C_v

= Vertrauensbereich des ermittelten optischen Flusses

= transponierter Vektor der komponentenweise Differenz zwischen erwartetem und ermittelten optischen Fluss

r_i,v

= Vektor der komponentenweise Differenz zwischen erwartetem und ermittelten optischen Fluss.

Herein mean:

Φ _F (s _i , f _v , v)

= Energy based on the detected optical flux f _v at a given segment s _i in line v the column

α _F

= Constant for limiting the energy component

C _v

= Confidence interval of the detected optical flux

= transposed vector of the component-wise difference between expected and determined optical flux

r _{i, v}

= Vector of the component-wise difference between expected and determined optical flux.

und die obere Bildkoordinate eines Segments $v_i^t$

optimiert werden. Für die anderen Variablen werden vorgegebene Annahmen getroffen. Es können Annahmen bezüglich der inverse Entfernung des Segments zur Bildebene p_i getroffen werden. Es können Annahmen bezüglich der Bewegung t_i eines Segments getroffen werden. Es können Annahmen bezüglich der semantischen Klasse c_i getroffen werden. Beispielsweise kann die wahrscheinlichste Entfernung des Segmentes basierend auf dem Fahrzeugumfeldmodell angenommen werden. Beispielsweise kann eine Entfernung und Bewegung angenommen werden, bei der mindestens für einen Bildpunkt der erwartete optische Fluss dem ermittelten optischen Fluss entspricht. Beispielsweise kann eine semantische Klasse angenommen werden, bei der mindestens ein Bildpunkt der im Modul 206 zugeordneten Klasse entspricht.The determination of segment data in at least one column can be done as a solution to the energy minimization problem in formula (2) by means of dynamic programming, for example by means of a Viterbi algorithm. In order to reduce the computational effort, it is possible to proceed in such a way that the segment type m _i , the lower image coordinate of a segment $v_i^b$

and the upper image coordinate of a segment $v_i^t$

be optimized. For the other variables, given assumptions are made. There may be assumptions regarding the inverse distance of the segment to the image plane p _i to be hit. There may be assumptions about the movement t _i of a segment. There may be assumptions about the semantic class c _i to be hit. For example, the most probable distance of the segment may be assumed based on the vehicle environment model. For example, a distance and movement can be assumed in which the expected optical flux corresponds to the determined optical flux for at least one pixel. For example, a semantic class can be assumed in which at least one pixel in the module 206 corresponding class.

Im on the substep 304-d following sub-step 304-e it comes to the assignment of pixels of at least one column to a predetermined segment type. The assignment of pixels of the at least one column to a given segment type can be done by minimizing the energy term from formula (4). In particular, this is the sub-step 304-a taken into account respective distance of the pixels to the image plane.

In the partial step 304-f it comes to the formation of at least one segment. In this case, adjoining pixels, in the sub-step 304-e have been assigned to the same segment type.

In step 305 it comes to mapping the environment of the vehicle based on the segment data. For example, the multi-column segment data may be merged into one image. In step 306 it comes to the output of at least one signal representing the imaged environment. The procedure 300 ends in step 307 ,

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• DE 102009009047 A1 [0003]
• DE 102011111440 A1 [0003]
• DE 102012000459 A1 [0003]
• DE 102016106293 A1 [0004]
• DE 102016122190 A1 [0004]

Cited non-patent literature

• Schneider, L. et al., Semantic Stixels: Depth is not Enough. Intelligent Vehicles Symposium (IV), 2016 IEEE. IEEE, 2016, pages 110-117 [0003]

Claims (12)

A method (300) for imaging the environment of a vehicle (100), comprising the steps of: - detecting (302) an image sequence by means of a monocamera; Splitting (303) at least one image of the image sequence into columns; - determining (304, 304-a to 304-f) segment data in at least one column; - mapping (305) the environment based on the segment data; - output (306) at least one signal representing the imaged environment; characterized in that - the step of determining (304) the segment data in at least one column comprises a substep of the mapping (304-e) of pixels of the at least one column to a predetermined segment type. Method (300) according to Claim 1 , characterized in that a predetermined segment type is a potentially movable object. Method (300) according to Claim 1 or 2 wherein the step of determining (304) the segment data in at least one column comprises a further substep of forming (304-f) at least one segment in a column, and wherein the formation (304-f) of a segment depends on the assigned to respective pixels associated with segment types and a respective distance of the pixels to the image plane of the one image and a characterizing the respective movement of a pixel size. Method (300) according to one of Claims 1 to 3 wherein the step of determining (304) the segment data in at least one column comprises a further substep of determining (304-c) an expected optical flow of at least one pixel of the at least one column by means of homography. Method (300) according to Claim 4 wherein the expected optical flux is determined taking into account at least one predetermined assumption. Method (300) according to Claim 4 or 5 wherein the step of determining (304) the segment data in at least one column comprises a further substep of comparing (304-d) the expected optical flow of a pixel with an optical flux of the same pixel determined from the acquired image sequence. Method (300) according to one of Claims 2 to 6 wherein the mapping (304-e) of pixels of the at least one column to a given segment type is done by means of a minimization of an energy term. Method (300) according to one of Claims 2 to 7 , characterized in that the mapping (304-e) of pixels of the at least one column to the segment type of a potentially mobile object is dependent on an association of each pixel of the at least one column with a predetermined semantic class. Method (300) according to one of Claims 3 to 8th wherein the step of determining (304) the segment data in at least one column comprises a further substep of determining (304-a) the removal of the pixels of potentially moving objects to the image plane by means of a vehicle environment model. Computer program adapted to perform all steps of the method (300) according to one of Claims 1 to 9 perform. Device (110) for imaging the surroundings of a vehicle (100), comprising at least: - a monocamera (101) for detecting an image sequence; - at least one module (201) for splitting at least one image of the image sequence into columns; - at least one module (202) for determining segment data in at least one column; - at least one module (203) for mapping the environment based on the segment data; at least one module (204) for outputting at least one signal representing the imaged environment, characterized in that - the module (202) for determining segment data in at least one column is designed to image pixels of the at least one column into a predetermined segment Assign type. Device (110) according to Claim 11 wherein the module (202) for determining segment data in at least one column is further adapted to determine an expected optical flow of at least one pixel of the at least one column by means of a homography.

Images