DE102019219756A1 - Method for generating artificial roadway features, computer program, computer-readable data carrier and device for generating artificial roadway features - Google Patents

Abstract

A method (20) for generating artificial roadway features, comprising: • obtaining (21) image information from a camera (2, 2a); • inputting (22) the image information obtained into a neural network (34), wherein the neural network (34) generates the artificial roadway features based on the image information obtained; and • supplementing (23) real roadway features represented in the image information obtained with the generated artificial roadway features.

Description

The invention relates to a method for generating artificial roadway features, a computer program, a computer-readable data carrier and a device for generating artificial roadway features.

Today's driver assistance systems in vehicles can rely on image information from the vehicle environment in order to be able to perform their function safely and reliably. This information can e.g. B. provided by one or more cameras in or on the vehicle. For example, driver assistance systems for lane or blind spot monitoring, which can output a warning to the driver in certain driving situations, are known. Other systems, such as an emergency braking system, can also automatically initiate a braking process based on the image information.

In autonomous vehicles in particular, it can be important to precisely record the course of the road in an area in front of the vehicle in order to be able to plan a safe route precisely. The autonomous vehicle can analyze a large number of different sensor or data sources, in particular cameras, in order to model the course of the road.

From the German Offenlegungsschrift DE 10 2011 109 569 A1 For example, a method for lane recognition is known. The method records at least one image of a vehicle environment with a camera, recognizes a driving corridor from the at least one image, and recognizes a structure taking into account the determined driving corridor in order to determine the course of at least one edge of the road from the at least one image. The method is based on the idea that the road edge can generally be detected by means of a camera and the lane can be estimated from the position of the road edge or the position of the road edge is at least taken into account when recognizing the lane.

The German Offenlegungsschrift DE 10 2014 223 259 A1 describes a method for estimating the course of the lane of a roadway from environment data recorded by at least one environment sensor in front of a vehicle. First, the width of the lane in front of the vehicle is determined from the environment data and at least one hypothesis is created about the number of lanes. On the basis of the hypothesis about the number of lanes, a probability model for the position of the middle of the lane is determined. The position of the middle of the lane is determined from the probability model with the greatest probability value.

From the German Offenlegungsschrift DE 10 2016 211 730 A1 a method for predicting the course of a roadway in the surroundings of a vehicle is known.

Driving situations can be problematic in which lane markings, traffic signs or, in general, the course of the lane cannot be recognized well or only partially in the image information. This can be the case, for example, when other road users such as a large truck cover the corresponding markings or lane boundaries. Such situations cannot be taken into account by the solutions from the prior art documents mentioned above.

The object of the present invention is therefore to provide a method for generating artificial roadway features, a computer program, a computer-readable data carrier and a device for generating artificial roadway features, which at least partially overcome the above-mentioned disadvantages.

This object is achieved by the method according to the invention according to claim 1, the computer program according to claim 8, the computer-readable data carrier according to claim 9 and the device according to claim 10.

• • Erhalten von Bild-Informationen von einer Kamera;
• • Eingeben der erhaltenen Bild-Informationen in ein neuronales Netzwerk, wobei das neuronale Netzwerk basierend auf den erhaltenen Bild-Informationen die künstlichen Fahrbahn-Merkmale erzeugt; und
• • Ergänzen von in den erhaltenen Bild-Informationen repräsentierten realen Fahrbahn-Merkmalen mit den erzeugten künstlichen Fahrbahn-Merkmalen.

According to a first aspect, the invention provides a method for generating artificial roadway features, comprising:

• • Obtaining image information from a camera;
• Inputting the image information obtained into a neural network, the neural network generating the artificial roadway features based on the image information obtained; and
• • Supplementing real roadway features represented in the image information obtained with the generated artificial roadway features.

According to a second aspect, the invention provides a computer program comprising instructions which, when the program is executed by a computer, cause the computer to execute the method according to the first aspect.

According to a third aspect, the invention provides a computer-readable data carrier on which the computer program according to the second aspect is stored.

According to a fourth aspect, the invention provides a device for generating artificial roadway features which is set up to carry out the method according to the first aspect.

Further advantageous embodiments of the invention emerge from the subclaims and the following description of preferred exemplary embodiments of the present invention.

• • Erhalten von Bild-Informationen von einer Kamera;
• • Eingeben der erhaltenen Bild-Informationen in ein neuronales Netzwerk, wobei das neuronale Netzwerk basierend auf den erhaltenen Bild-Informationen die künstlichen Fahrbahn-Merkmale erzeugt; und
• • Ergänzen von in den erhaltenen Bild-Informationen repräsentierten realen Fahrbahn-Merkmalen mit den erzeugten künstlichen Fahrbahn-Merkmalen.

A method according to the invention for generating artificial roadway features comprises:

• • Obtaining image information from a camera;
• Inputting the image information obtained into a neural network, the neural network generating the artificial roadway features based on the image information obtained; and
• • Supplementing real roadway features represented in the image information obtained with the generated artificial roadway features.

As mentioned at the beginning, when driving a motor vehicle, the view of the road ahead and the lane markings by other road users, e.g. B. be partially or heavily covered by a truck. Road markings can also be covered in driving situations on, for example, leafy or snow-covered roads. Consequently, in many everyday traffic situations, the field of view of a camera in or on the vehicle on the course of the road can be reduced by other objects. As a result, the camera can only process a section of the image information. B. road markings are estimated. This can be problematic in particular in autonomous vehicles, since there an exact modeling of the course of the road based on road markings is important for a safe and reliable functioning of the vehicle and otherwise, for example, autonomous driving can be restricted.

Consequently, in some exemplary embodiments, the method is used in particular in the motor vehicle environment, without the invention being restricted to this case.

The method can in principle be carried out on a computer, in particular an on-board computer or a control device in a motor vehicle. For this purpose, the computer has electronic components such as, for example, one or more CPUs (from “Central Processing Unit”), one or more GPUs (from “Graphical Processing Unit”), one or more ASICs (from “Application Specific Integrated Circuit”), electronic ones Storage elements or the like, which are typically used for the calculations necessary in the method. In exemplary embodiments in the motor vehicle environment, the method can in particular be carried out on a control unit or on-board computer or the like, or can be appropriately distributed to several control units, processors, GPUs, ASICs or the like in order to carry out the method described herein.

It was recognized that with the application of e.g. B. generating opposing networks (also “GAN” in the following from “Generative Adversial Networks”) from the field of artificial intelligence. B. an artificial camera image can be generated, which shows the further course of the road, which was not previously available in the image information from the camera. In some exemplary embodiments, this can be used to support driver assistance systems in that the artificial camera image is made available to the driver assistance systems.

Neural networks, in particular GANs, are known in principle to the person skilled in the art. A GAN can e.g. B. designed and / or trained in such a way that it generates artificial image information. Artificial image information such. B. represent people, lane markings or vehicles or any object classes, which real image information (i.e. e.g. camera images of real existing objects etc.) representing the same object classes can be very similar. Whereby similar can mean that the artificial image information from an image analysis method (which, for example, can also be based on a neural network) cannot be differentiated from real image information.

Based on neural networks, in particular GANs, in some exemplary embodiments these can therefore be designed and / or trained in such a way that the method for generating artificial roadway features can be used, in particular in the motor vehicle environment, without the invention being restricted to this case should.

In this case, artificial lane features in some exemplary embodiments can correspond to artificial image information representing lane markings, traffic signs and the like, that is to say elements or features of a lane on which z. B. is a motor vehicle.

In some exemplary embodiments, image information can correspond to a recorded camera image. In other exemplary embodiments, image information can be represented by image Analytical methods extracted features, such as B. points, lines, objects and / or outlines of objects, color gradients, etc .. In general, the image information can be represented by digital and / or analog values and / or signals.

Thus, in some exemplary embodiments, artificial roadway features can correspond to a generated camera image in which roadway markings, traffic signs and the like can be represented. In other exemplary embodiments, the artificial lane features can be points, lines, objects and / or outlines of objects, and the like, which lane markings, traffic signs and the like can represent.

In some exemplary embodiments, image information is first received from a camera in order to generate artificial roadway features. In some exemplary embodiments, the camera can be mounted in or on a motor vehicle and record the surroundings or a part of the surroundings of the motor vehicle, without the invention being restricted to this case. In such exemplary embodiments, real lane features can be represented in the image information from the camera, i. H. Elements or features of a roadway on which the motor vehicle is located. These can in particular be road boundaries, markings and the like, which can in principle be suitable for determining and / or modeling a road course in an area around the motor vehicle.

The image information obtained is input into a neural network, the neural network generating the artificial roadway features based on the image information obtained.

The neural network can be a GAN, a convolutional neural network (also CNN in the following from “convolutional neural network”), and it can generally be a single-layer or multi-layer neural network or the like. In some exemplary embodiments, the neural network can be characterized by the weights between the individual layers and / or by the weights of the individual neurons. In some exemplary embodiments, the weights of the neural network are loaded from a memory in order to carry out the calculations for generating the artificial roadway features based on the image information obtained, which, as mentioned above, is carried out on one or more CPUs, one or more GPUs, or a corresponding distribution can be carried out on these. The memory can be a read-only memory or a volatile memory. In other exemplary embodiments, the neural network is implemented by one (or more) ASIC.

In some exemplary embodiments, the neural network can record and process the image information obtained as input values in order to have generated the artificial roadway features after the calculations in the individual layers of the neural network. Artificial can mean in some exemplary embodiments that the lane features can actually exist, but are not represented in the image information obtained (because, as mentioned above, they are, for example, by an object such as a vehicle, pedestrian, foliage, Snow, water, etc. are covered). In some exemplary embodiments, the generated artificial roadway features also have the real roadway features represented in the image information obtained and additional roadway features which, based on the received image information or the real roadway features, have a high probability of the roadway or correspond to the further course of the road. In other exemplary embodiments, the generated artificial roadway features only have the additional roadway features.

In some exemplary embodiments, the generated artificial roadway features therefore supplement the real roadway features represented in the image information obtained.

In some exemplary embodiments, adding artificial lane features to the real lane features can be, for example, adding lane markings that were previously covered (that is, they were not contained in the image information), traffic signs and the like. In other embodiments, a lane boundary or the like, for. B. be extended according to their likely course.

As mentioned above, for example, the field of view of a camera in a motor vehicle on the course of the road in some exemplary embodiments can be covered by other objects such as a truck, or the object can (partially) cover a lane feature that is, for example, for a or several assistance systems are required. Based on neural networks, in particular methods such as GANs, it may be possible in some exemplary embodiments to train an algorithm or a neural network which, for example, contains a hidden image of the camera, i.e. the image information output by the camera Motor vehicle receives as input data and would reproduce an image as an output in which z. B. the further course of the road is shown without an obscuring object. In some exemplary embodiments, this image could be used as a further input for a The following algorithm can be used in order to be able to estimate or model the roadway and the course of the roadway, even if the input image from the camera only allows this to a limited extent. This is advantageous, in particular in the case of autonomous vehicles, since it allows the roadway and the course of the roadway to be modeled more precisely. In addition, various driver assistance systems can use the artificial roadway features, as a result of which the robustness of the systems is increased in problematic situations and the safety during operation of the vehicle is thus improved.

As mentioned above, in some exemplary embodiments the generated artificial roadway features have roadway markings which can be suitable for modeling the roadway and the course of the roadway. The lane markings can be, for example, guidelines (or broken lines), lane or lane boundaries (or solid line), or generally longitudinal markings (acceleration lanes, yellow markings in construction sites, cycle lanes, etc.), area markings (slash gates or restricted areas , Radfahrerfurt), border markings, cross markings (stop lines, zebra crossings, etc.), parking area markings, arrows, and the like. In some exemplary embodiments, the road markings can have delineator posts or the like. In some exemplary embodiments, roadway features include features which characterize a course of the roadway, a boundary or other characteristics of it.

As mentioned above, the generated artificial roadway features have traffic signs in some exemplary embodiments. The traffic signs can be, for example, signs or signs for speed limits, right of way, stop signs, motorway signs, warning signs (e.g. before sharp bends, deer crossing, etc.), and the like.

In some exemplary embodiments, the (obtained) image information is represented by image data. In such exemplary embodiments, image data can be or have analog and / or digital values and / or signals that can represent a recorded camera image. These can, for example, be or have the digitized signals from individual pixels of a camera and they can also be present in any data format (e.g. as raw data format, JPEG, SVG, etc.).

In some exemplary embodiments, the image information is represented by image features. In such exemplary embodiments, the image data recorded with a camera can, for example, be preprocessed in the camera, with various image features being extracted from the image data. As mentioned above, these can be, for example, points, lines, objects and / or outlines of objects, color gradients, etc., which represent lane markings, traffic signs, road users, people and the like.

In some exemplary embodiments, the image information is represented by a combination of image data and image features.

In some exemplary embodiments, the neural network has a generating opposing network. GANs are known in principle and can consist of two neural networks: a generator and a discriminator. In some exemplary embodiments, the generator can be designed and / or trained in such a way that it generates artificial roadway features based on real roadway features (input values) represented in the image information obtained or supplements the real roadway features that the discriminator does not can distinguish from real roadway features. In such exemplary embodiments, the discriminator is designed and / or trained in such a way that it could fundamentally recognize differences between artificial and real roadway features. Therefore, the generated artificial road features can subsequently be used for e.g. B. the modeling of the course of a roadway in an area around a motor vehicle can be used and various driver assistance systems are made available.

Therefore, in some exemplary embodiments, the neural network is trained with training image information. A trained neural network can differ from an untrained neural network by the values of the weights, as is generally known. The training image information can, as mentioned above, e.g. B. be image data or image features or contain image data and / or image features (generally represent). The training image information preferably exhibits a high degree of variation in different situations. In exemplary embodiments that relate to an application of the method in a motor vehicle, for example, a lane and the course of the lane in the training image information with various disturbances or obscurations (leaves, snow, people, other vehicles that have different positions on the lane , etc.) of individual sections are available in order to train the neural network. The training image information preferably has situations that show the roadway and the course of the roadway with and without obscuring.

In some exemplary embodiments, the training image information can be obtained from camera images that were recorded in real situations. In other exemplary embodiments, the training image information has been generated by a computer. In further exemplary embodiments, the training image information is a combination of real and computer-generated image information.

In some exemplary embodiments, the neural network can have been trained with the training image information on a computer, a server, a cloud or the like, these having electronic components such as a CPU, GPU, ASICs or the like, which are typically used for training a neural network can be used.

Some exemplary embodiments relate to a computer program, comprising instructions which, when the program is executed by a computer, cause the computer to execute the method described herein.

For this purpose, as mentioned above, the computer has electronic components such as, for example, one or more CPUs, one or more GPUs, one or more ASICs, electronic storage elements or the like, which are typically used for the calculations required in the method described herein.

Some exemplary embodiments relate to a computer-readable data carrier on which the aforementioned computer program is stored.

The computer-readable data carrier can be implemented by a digital storage medium such as a magnetic storage element (e.g. a magnetic hard disk or the like), a digital semiconductor memory (e.g. flash memory or the like), a combination of both, an optical data carrier, etc. In some exemplary embodiments, the computer-readable data carrier can also be implemented by a multiplicity of such storage elements on which the computer program is appropriately distributed.

Some exemplary embodiments relate to a device for generating artificial roadway features which is set up to carry out the method described herein.

The device can in principle be implemented by a computer as described above in order to carry out the method described herein.

• 1 schematisch ein erstes Ausführungsbeispiel eines Verfahrens zur Erzeugung von künstlichen Fahrbahn-Merkmalen in einem Kraftfahrzeug zeigt;
• 2 in einem Ablaufdiagramm ein zweites Ausführungsbeispiel eines Verfahrens zur Erzeugung von künstlichen Fahrbahn-Merkmalen zeigt; und
• 3 in einem Blockdiagramm eine Vorrichtung zur Erzeugung von künstlichen Fahrbahn-Merkmalen zeigt.

Embodiments of the invention will now be described by way of example and with reference to the accompanying drawing, in which:

• 1 schematically shows a first embodiment of a method for generating artificial roadway features in a motor vehicle;
• 2 shows in a flowchart a second exemplary embodiment of a method for generating artificial roadway features; and
• 3 shows in a block diagram a device for generating artificial roadway features.

The first exemplary embodiment of the method for generating artificial roadway features is shown in FIG 1 schematically for an application in a motor vehicle 1 shown.

The car 1 instructs a camera 2 and a device 3 for the generation of artificial roadway features. The device 3 In this exemplary embodiment, an on-board computer is used to generate artificial roadway features.

The car 1 drives on a roadway which is a roadway course 4th having. In front of the motor vehicle 1 drives a second motor vehicle 5 on the same lane. The course of the road 4th has a sharp left turn and has several lane markings 6th and a road sign 7th which indicates the sharp left turn. The camera 2 is in the motor vehicle 1 mounts and draws image information from an area in front of the motor vehicle 1 on, the area being through the camera's field of view 2 is given (illustrated by the dashed lines). In this exemplary embodiment, the image information is represented by image data. The camera's field of view 2 on the course of the road 4th who have favourited road markings 6th and the traffic sign 7th is by the second motor vehicle 5 restricted, ie the area of the roadway 4th between the dotted lines is through the second motor vehicle 5 covered. As a result, only the second motor vehicle is in the image information 5 and part of the road markings 6th that represents not for the camera 2 is covered.

The device 3 the image information is received from the camera to generate artificial roadway features 2 via a CAN bus system of the motor vehicle. Real lane features are represented in the image information obtained, which in this exemplary embodiment are part of the lane markings 6th are not in the by the second motor vehicle 5 hidden area are localized. The image information obtained is stored in a neural network (here a GAN) in the device 3 for the generation of artificial roadway features, which generates artificial roadway features based on the image information obtained. In this exemplary embodiment, the artificial roadway features are those previously created by the second motor vehicle 5 covered road markings 6a and the concealed traffic sign 7a . In this exemplary embodiment, the artificial lane features are represented by image data in which the “real” lane features (the previously uncovered part of the lane markings 6th ), the covered lane markings 6a and the concealed traffic sign 7a are represented.

The second embodiment of the method 20th for the generation of artificial roadway features is described with reference to the flowchart in 2 and the block diagram in 3 described.

The device 3 for the generation of artificial roadway features (corresponds to the from 1 ) in 3 has a CAN interface 30th , a CPU 31 , a GPU 32 , a computer-readable medium 33 (here a semiconductor memory) and a neural network 34 on and is in the motor vehicle 1 out 1 built-in. The computer-readable medium 33 contains a computer program and the weights of the neural network 34 , which is the neural network 34 characterize. The neural network 34 is shown separately for illustration and corresponds in this exemplary embodiment to the stored weights to which the CPU and GPU have access via a bus system (illustrated here by connecting lines). The camera 2a records image information (the camera 2a corresponds to the off 1 , whereby in this exemplary embodiment the image information is represented by a combination of image data and image features) and transmits this to the device 3 for generating artificial roadway features via a CAN bus system or the CAN interface 30th .

At 21, corresponding image information is received from the camera 2a receive.

The computer program from the computer-readable data carrier 33 is loaded, contains commands that the device 3 to generate artificial roadway features based on the data from the camera 2a image information obtained to generate artificial roadway features. The CPU performs the necessary calculations 31 and the GPU 32 from which for this on the neural network 34 access.

At 22, the image information obtained is therefore entered in the neural network 34 entered, the neural network 34 the artificial roadway features are generated based on the image information obtained, as set out herein.

At 23, the real roadway features represented in the image information obtained are supplemented with the generated artificial roadway features, as set out herein.

At 24, the artificial lane features generated include lane markings, as set forth herein.

At 25, the generated artificial roadway features include traffic signs, as set out herein.

At 26, the image information is represented by image data as set forth herein.

At 27, the image information is represented by image features, as set out herein.

At 28, the neural network points 34 a generating adversary network as set forth herein.

At 29 is the neural network 34 trained with training image information as set forth herein.

The generated artificial roadway features are transmitted via the CAN interface 30th a driver assistance system 40 made available. In this exemplary embodiment, the driver assistance system is 40 a system for lane monitoring.

In the process 20th thus the image information from the camera 2a receive. From this, artificial roadway features are generated that correspond to a new image (image data and image features) without being covered. This image is then exactly like the image information from the camera 2a in a processing process (to the driver assistance system 40 ) in order to recognize lane markings. These are then used to model the surroundings of the vehicle despite being covered.

List of reference symbols

1

Motor vehicle

2, 2a

camera

3

contraption

4th

Lane course

5

Second motor vehicle

6, 6a

Road markings

7, 7a

Traffic signs

20th

Procedure

21

Obtaining image information from a camera

22nd

Inputting the image information obtained into a neural network, the neural network generating the artificial roadway features based on the image information obtained

23

Supplementing real roadway features represented in the image information obtained with the generated artificial roadway features

24

Showing lane markings

25th

Showing traffic signs

26th

Representing the image information with image data

27

Representation of the image information by image features

28

Having a generating opposing network

29

Training the neural network with training image information

30th

CAN interface

31

CPU

32

GPU

33

Computer readable data carrier

34

Neural network

40

Driver assistance system

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• DE 102011109569 A1 [0004]
• DE 102014223259 A1 [0005]
• DE 102016211730 A1 [0006]

Claims (10)

A method (20) for generating artificial roadway features, comprising: • Obtaining (21) image information from a camera (2, 2a); • inputting (22) the image information obtained into a neural network (34), the neural network (34) generating the artificial roadway features based on the image information obtained; and • Supplement (23) of real roadway features represented in the image information obtained with the generated artificial roadway features. Method (20) according to Claim 1 , the generated artificial roadway features having roadway markings (6, 6a) (24). Method (20) according to Claim 1 or 2 , the generated artificial roadway features having traffic signs (7, 7a) (25). Method (20) according to one of the Claims 1 to 3 wherein the image information is represented by image data (26). Method (20) according to one of the Claims 1 to 3 , the image information being represented by image features (27). Method (20) according to one of the preceding claims, wherein the neural network (34) has a generating opposing network (28). Method (20) according to one of the preceding claims, wherein the neural network (34) is trained (29) with training image information. Computer program, comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method (20) according to one of the preceding claims. Computer-readable data carrier (33) on which the computer program after Claim 8 is stored. Device (3) for generating artificial roadway features, which is set up to carry out the method (20) according to one of the Claims 1 to 7th to execute.

Images