DE102021110069A1 - Method, assistance device and motor vehicle for the semantic segmentation of a digital image and method for providing a correction model for the semantic segmentation - Google Patents

Abstract

The invention relates to a method for semantically segmenting a digital image (14). An error-prone provisional semantic segmentation (22) is generated from the image (14) by a correspondingly trained segmentation model (20). This is supplied to a correction model (26) which is trained to correct such erroneous semantic segmentations (22). The correction model (26) automatically generates a corrected semantic segmentation (34) of the image (14) by adapting at least one classification determined by the segmentation model (20) as part of the preliminary semantic segmentation (22). The invention also relates to a method (10) for providing such a correction model (26). The invention further relates to an assistance device (12, 42) for correspondingly generating corrected semantic segmentations (34) and to a motor vehicle (36) equipped with such an assistance device (12, 42).

Description

The present invention relates to a method for the semantic segmentation of a digital image and a method for providing a correction model used therein. The invention further relates to an assistance device set up for such a semantic segmentation of digital images and a motor vehicle equipped therewith.

Digital image capture and processing can be used in a variety of technical fields and applications. The challenge here is to use the corresponding images as profitably as possible with purely technical means. In addition, there are now increasing efforts to use so-called computer vision, with images being automatically segmented semantically. Such a semantic segmentation assigns a meaning or object class to individual image areas or pixels of the respective image and thus in a certain sense enables a machine understanding of a scene represented in the respective image, a reaction to recognized objects and/or the like. The problem here, however, is that such methods have not yet been 100% accurate or reliable, meaning that they make errors in semantic segmentation, for example. Depending on the application, this can represent a potential security risk, reduce user comfort and/or reduce practicable application options.

As an application of an image segmentation is in the DE 11 2019 000 122 T5 a real-time detection of lanes and boundaries by autonomous vehicles is described. In the method provided there, sensor data, which are representative of an image of a field of view of the sensor, are received from a sensor of a vehicle and applied to a neural network. This computes a segmentation mask that is representative of portions of the image that correspond to the track markers of a bin surface in the field of view. A lane marker type is assigned to each of the lane markers based on the segmentation mask. Also based on the segmentation mask, curve fitting is then performed on the respective lane markers to generate lane boundaries representative of each of the lane marker types. Corresponding data is then sent to a component of the vehicle for use by the vehicle in navigating the driving surface. This is intended to achieve improved accuracy and reliability compared to conventional systems, which may be limited to achieving a minimum accuracy of ideal roads and weather conditions.

However, the semantic segmentation of image data can be very computationally intensive and complex, which can be particularly problematic for real-time applications. Some conventional approaches to reducing the computing effort can be at the expense of accuracy, reliability and robustness, which is ultimately undesirable in safety-critical applications such as vehicle technology. Against this background, the DE 11 2019 000 048 T5 proposes a method for determining an accessible free space for autonomous vehicles. In this case, sensor data generated by a sensor of the vehicle in a physical environment is recorded and applied to a neural network. A first stream of the neural network computes a set of boundary points, represented by the sensor data and corresponding to a boundary dividing a navigable space in the physical environment from a non-navigable space in the physical environment. Further there, locations in the physical environment are determined which correspond to the set of boundary points. A second stream of the neural network calculates class designations for boundary points of the group of boundary points. Finally, an operation to control the vehicle through the physical environment is performed depending at least in part on the locations and class designations. This is to enable the use of an efficient and accurate machine learning model. It also eliminates the need to classify each pixel of an image separately, as traditional segmentation approaches require, thereby reducing the need for extensive post-processing of the neural network output.

The object of the present invention is to enable improved semantic segmentation of images.

According to the invention, this object is achieved by the subject matter of the independent patent claims. Possible refinements and developments of the present invention are disclosed in the dependent patent claims, in the description and in the figures.

The method according to the invention serves and can therefore be used for the semantic segmentation of digital images, ie corresponding image data. In a method step of the method according to the invention, the image to be segmented is used as input, ie as input data supplied to a predetermined or provided trained segmentation model. This segmentation model is trained for the semantic segmentation of image data. For example, a conventional model for semantic image segmentation can be used as this segmentation model.

In a further method step of the method according to the invention, a provisional semantic segmentation of the respective image is generated by the segmentation model by processing the respective image, ie the input supplied to it. Since the segmentation models available today, as already indicated above, do not achieve 100% accuracy in the semantic segmentation, this provisional semantic segmentation is likely to be or typically error-prone. Preliminary semantic segmentation can also be referred to as initial segmentation prediction. The provisional semantic segmentation forms an output, i.e. initial data, of the segmentation model. This output can, for example, be output via a corresponding interface and/or stored in a data memory or buffer.

In a further method step of the method according to the invention, the provisional semantic segmentation of the respective image is supplied as input to a predetermined or provided trained correction model. This correction model is another model that is trained to correct error-prone semantic segmentations. This training of the correction model is explained in more detail elsewhere.

Incorrect semantic segmentations of images are, in the present sense, semantic segmentations that assign a specific class or classification to the individual image areas or pixels of the respective image, or corresponding maps of such area-by-area or pixel-by-pixel classifications, in which at least one area or pixel is incorrect, i.e. not an objective truth (English: ground truth) accordingly, is classified.

The correction model and the segmentation model can be trainable or trained models, algorithms or machine learning devices. A specific implementation may be selected or determined, for example, depending on a particular use case, available computational or execution resources, and/or the like.

In a further method step of the method according to the invention, a corrected semantic segmentation of the respective image is automatically generated or output as an output of the correction model by the correction model by processing the provisional semantic segmentation, ie the input supplied to it. The correction model can adapt, ie change, at least one classification determined by the segmentation model as part of the provisional semantic segmentation of the respective image. It has been shown that in this way a significant proportion of the segmentation errors made by the segmentation model and contained in the provisional semantic segmentation can be corrected by the correction model. This means that the correction model can therefore replace corresponding incorrect classifications of the segmentation model with the respective true, ie objectively correct, classification.

The corrected semantic segmentation can thus have at least fewer segmentation errors overall than the provisional semantic segmentation. With the present invention, a semantic segmentation of the respective image can be generated overall in the form of the corrected semantic segmentation, which can have improved accuracy compared to conventional methods or approaches. If, for example, a conventional segmentation model is used that has an accuracy, i.e. a rate of correct classifications, of about 90%, the additional application of the correction model provided according to the invention can, for example, achieve an overall accuracy of the corrected semantic segmentation of, for example, about 95% - depending from the concrete application scenario.

The correspondingly trained correction model has effectively learned to model and reverse errors made or introduced by the segmentation model, i.e. to correct them. In other words, the correction model can be trained to reconstruct the correct classifications from the classification predictions of the segmentation model, ie from the provisional semantic segmentation.

Conceptually, this can be understood as denoising the respective image. In this sense, segmentation errors made by the segmentation model can be understood as noise that was introduced into the preliminary semantic segmentation by the segmentation model. The correction model is then trained in this mental image to eliminate this noise to remove and thereby improve the classification accuracy, i.e. the accuracy of the semantic segmentation.

In a vehicle or traffic engineering application, the segmentation models available today often misclassify pedestrians, for example, due to their diverse, very variable appearances and contours. Based on such a pedestrian that is only partially correctly classified or a corresponding partial area of the provisional semantic segmentation classified as a pedestrian, the correction model can then fill in erroneous, i.e. misclassified areas that, contrary to the provisional semantic segmentation, actually belong to the pedestrian, with a pedestrian classification and thus reconstruct the entire area corresponding to the pedestrian with correct classification as a pedestrian.

It should be noted that there are already smoke extraction algorithms in classical image processing, for example for removing Gaussian noise. The errors of the segmentation model - which can be interpreted here as noise - i.e. the correspondingly error-prone provisional semantic segmentation are in comparison to this much more complex, depending on the segmentation model used and thus overall hardly practicable exactly and reliably modelable or predictable. Accordingly, correcting the erroneous preliminary semantic segmentation using, for example, a simple classical denoising algorithm or filter is not practical or possible with the accuracy or quality achievable by the present invention.

The present invention is based on the knowledge that segmentation errors made by current segmentation models are often not completely random, but rather follow a certain structure. This structure can be learned or has been learned by the correction model, but not readily modeled by a conventional or manually created correction algorithm.

Another advantage of the present invention is that the correction model, in particular only or completely, works in the semantic space, i.e. at the level of the semantic segmentations, since it accepts the preliminary semantic segmentation of the segmentation model as input and from this the corrected semantic segmentation of the respective image generated. The correction model therefore does not directly use the respective underlying image itself. As a result, the method according to the invention requires little effort and is efficient, for example in comparison to another or conventional multi-stage method for image processing, which uses the respective original image again for a second stage, i.e works at least partially in the pictorial space in several stages.

In one possible embodiment of the present invention, a respective deep artificial neural network is used at least as part of the segmentation model and/or at least as part of the correction model. In other words, a respective multi-layer artificial neural network is used, which has at least one middle, ie hidden, layer. It has been shown that deep artificial neural networks of this type represent an effective way of realizing or implementing the segmentation model or the correction model. For example, a convolutional neural network (CNN) can be used here.

In a further possible embodiment of the present invention, the segmentation model and the correction model are trained or set up to generate a pixel-precise semantic segmentation as the respective output in relation to the respective image. In other words, the segmentation model creates the provisional semantic segmentation and the correction model creates the corrected semantic segmentation with pixel accuracy. The provisional semantic segmentation and the corrected semantic segmentation can thus be generated with the same resolution that the respective underlying image also has. An individual classification can thus be assigned to each pixel of the respective image. In this way, the semantic segmentation can be particularly precise and detailed, which enables a correspondingly precise and detailed reaction based on the corrected semantic segmentation, for example for an assistance system of a motor vehicle. The pixel-precise semantic segmentation makes it possible, for example, to react particularly quickly to an object gradually entering a field of view mapped by the image, since such an object can possibly already be detected if it only occupies a single pixel of the current image.

In a further possible embodiment of the present invention, the segmentation model is trained for the semantic segmentation of images that represent traffic scenes. The image supplied as input to the segmentation model is then such an image that a traffic depicting a scene. In other words, the method according to the invention is used in particular in traffic or vehicle technology. This represents an application purpose or area in which automated image processing is of particular importance in view of the current efforts to develop automated vehicles and which can benefit in particular from an improved accuracy of the automatic semantic segmentation. For example, the present invention can achieve improved safety and robustness in the operation of corresponding assistance systems or at least partially automated motor vehicles equipped therewith, which use automatic semantic segmentation of images.

In a further possible embodiment of the present invention, the correction model is or will be trained introspectively in relation to the respective segmentation model provided. The correction model is or will be trained using actual error-prone semantic segmentations generated by this or a corresponding segmentation model. In other words, the correction model is or will be matched or adapted to the respective segmentation model used in the method according to the invention. As a result, a particularly high level of accuracy and reliability of the correction model can be achieved. This is the case because the correction model automatically and implicitly learns individual properties or behaviors of the respective segmentation model. This is particularly useful since different segmentation models, for example trained using different training data and/or having different internal structures or sizes, can make different errors when processing, ie when semantically segmenting the same image. These different errors, behaviors or properties of the segmentation model do not have to be determined or defined manually by the introspective training of the correction model proposed here and explicitly introduced or taken into account when training the correction model. The correction model can therefore not only have improved accuracy here, but can also be generated, ie trained, with particularly little effort.

In a further possible embodiment of the present invention, when processing the preliminary semantic segmentation, the correction model uses a representation or version of the preliminary that is reduced in comparison to the preliminary semantic segmentation internally at least at one point, in particular in at least one layer semantic segmentation. In other words, a correction model is used here that has an internal structure that results in a size or resolution of the respective input, in this case the respective preliminary semantic segmentation, being reduced in at least one processing step. As a result, the correction model can be achieved or forced to only use the most important or most relevant features of the provisional semantic segmentation for generating the corrected semantic segmentation as error-free as possible. In other words, a focussing of the correction model on areas or parts of the respective provisional semantic segmentation that are—at least presumably—incorrect can be achieved or enforced. In this way, a subsequent computing effort carried out or applied by the correction model can be focused or limited to precisely these faulty areas or parts or features. This can improve the efficiency of the correction model or the method according to the invention, since the processing of the reduced representation of the preliminary semantic segmentation requires less computing effort or resources than, for example, a continuous 1-to-1 representation or mapping of the preliminary semantic segmentation at each point or in each layer of the correction model, while at the same time the accuracy of the correction model can be maintained or even improved by focusing the correction model on the error-prone areas or parts of the preliminary semantic segmentation.

In a further possible embodiment of the present invention, an autoencoder with an input-side encoder part and an output-side decoder part is used as the correction model. A location or layer of the correction model can be arranged between this encoder part and decoder part, in which, when processing the respective preliminary semantic segmentation as described elsewhere, there is a reduced representation of the respective preliminary semantic segmentation, i.e. it is used or processed by the correction model. In this sense, this point or layer can therefore form a bottleneck in a processing pipeline of the correction model. In this case, the encoder part can be a reducing part which, starting from the provisional semantic segmentation, generates its reduced representation. The decoder part, on the other hand, can work in an expanding manner, that is to say starting from this reduced representation, generate the corrected semantic segmentation which is enlarged in comparison thereto, in particular with the resolution of the provisional semantic segmentation. The use of such an autoencoder as a correction model, ie the use of a correction model with such an autoencoder structure, has turned out to be particularly effective and efficient compared to other possible structures or types of correction models. A particularly accurate and efficient semantic segmentation of images can thus be achieved or implemented in a particularly simple manner by the embodiment of the present invention proposed here.

Another aspect of the present invention is a method for providing or generating a correction model for semantic segmentation. In this way, in particular, the correction model mentioned or used in connection with the method according to the invention for the semantic segmentation of a digital image can be generated or provided. In the method described here, erroneous semantic segmentations of images and associated, in particular pixel-precise, true labels or annotations that indicate a respective correct classification, for example of at least one image area or one pixel, of the respective image are generated or provided as training data. The semantic segmentations used or provided as part of this training data can be or will be generated in particular by a segmentation model trained for the semantic segmentation of digital image data. This can in particular be the segmentation model mentioned in connection with the method according to the invention for the semantic segmentation of a digital image, as a result of which the introspective training of the correction model mentioned elsewhere is made possible.

Additionally or alternatively, synthetic semantic segmentations, ie semantic segmentations generated in some other way, possibly without a segmentation model explicitly trained for this purpose, can also be used or provided as part of the training data.

In a further method step, an initial model with a predetermined structure is provided. This starting model can, for example, be or include a trained or pre-trained model, in particular a deep artificial neural network. In particular, as also described elsewhere, the starting model can have an autoencoder structure.

In a further method step, the initial model is iteratively trained using the training data provided to reconstruct the associated true labels or annotations from the erroneous semantic segmentations. A conventional error feedback method or the like can be used for this purpose, for example.

In a further method step, after this training of the initial model has been completed, the initial model trained in this way is made available as the correction model. Optionally, this correction model can then be combined with the or a segmentation model in order to create or provide a corresponding segmentation device.

In the method proposed here, the initial model, in particular a model set up for deep learning or based on it, is trained to carry out a correct or corrected semantic segmentation of a respective image based on a respective error-prone semantic segmentation, i.e. a partially incorrect segmentation prediction of the segmentation model. This is achieved by feeding the error-prone semantic segmentations as input to the initial model during training and providing or pre-determining the corresponding correct semantic segmentations, which do not have the errors of the segmentation model contained in the error-prone semantic segmentations, as target output.

As already explained, the correction of the erroneous semantic segmentations, which is learned by the initial model, can be understood as denoising the erroneous semantic segmentations. In contrast to known smoke extraction algorithms or models, noise or a noise signal is not manually added to predetermined noise-free output data in order to generate the training data. Rather, the segmentation errors of the segmentation model made by the segmentation model and thus automatically contained in the error-prone semantic segmentations are used or interpreted as noise or noise signal. In this sense, the segmentation model that generates the erroneous semantic segmentations functions as an error function that introduces errors in the form of erroneous classifications into imaginary noise-free, i.e. correct, semantic segmentations. The starting model or the correction model then learns—at least effectively or conceptually—the corresponding inverse error function. By applying them to the respective error-prone semantic segmentation, the errors contained therein can be removed in order to actually generate the - originally only thought of - correct or corrected semantic segmentation as an output.

A further aspect of the present invention is an assistance device which is provided in particular for a motor vehicle, ie can be used in a motor vehicle, for example for the semantic segmentation of images which each represent or depict a traffic scene in the vicinity of the respective motor vehicle.

The assistance device according to the invention has an input interface for capturing digital images, a trained segmentation model coupled thereto for automatic semantic segmentation of the captured images and a correction model downstream of the segmentation model for automatically generating a corrected semantic segmentation of the images from a respective output, i.e. a respective preliminary semantic segmentation , of the segmentation model.

The assistance device according to the invention can also have an output or output interface for outputting the corrected semantic segmentations that have been generated and/or a computer-readable data memory for storing the corrected semantic segmentations.

The assistance device according to the invention can in particular be set up to automatically execute or carry out the method according to the invention for the semantic segmentation of a digital image. For this purpose, the assistance device can include, for example, a computer-readable data memory in which a corresponding operating or computer program that represents the method steps of the method according to the invention, that is to say encoded or implemented, is stored. This computer program can then include the segmentation model and the correction model, for example, or be set up to access these models. In the latter case in particular, these models can also be stored, for example, in the or a computer-readable data memory of the assistance device according to the invention.

The assistance device according to the invention can also have a processor device, for example a microchip, microprocessor or microcontroller, by means of which the computer program can be executed in order to carry out the corresponding method according to the invention or to cause it to be carried out.

Although the method according to the invention and the assistance device according to the invention can be usefully applied or used in particular in the area of traffic or vehicle technology, the present invention can also be used in other technical areas or areas of use.

A further aspect of the present invention is a motor vehicle which has a camera for recording images, each of which represents a traffic scene in a respective area surrounding the motor vehicle, and an assistance device according to the invention coupled thereto. In other words, the motor vehicle according to the invention is set up to carry out the method according to the invention for the semantic segmentation of a digital image. The motor vehicle according to the invention can therefore in particular be the motor vehicle mentioned in connection with the method according to the invention and/or in connection with the assistance device according to the invention. Accordingly, the motor vehicle according to the invention can have some or all of the properties and/or features mentioned in these contexts.

Further features of the invention can result from the claims, the figures and the description of the figures. The features and feature combinations mentioned above in the description and the features and feature combinations shown below in the description of the figures and/or in the figures alone can be used not only in the combination specified in each case, but also in other combinations or on their own, without going beyond the scope of the invention to leave.

• 1 einen beispielhaften schematischen Ablaufplan für ein Verfahren zum Bereitstellen eines Korrekturmodells für die semantische Segmentierung von Bildern;
• 2 eine schematische Darstellung zur Veranschaulichung eines Verfahrens und einer Einrichtung für eine verbesserte semantische Segmentierung unter Verwendung des Korrekturmodells; und
• 3 eine schematische Darstellung eines Kraftfahrzeugs, das für eine verbesserte semantische Segmentierung von Umgebungsbildern eingerichtet ist.

The drawing shows in:

• 1 an exemplary schematic flowchart for a method for providing a correction model for the semantic segmentation of images;
• 2 a schematic representation to illustrate a method and a device for an improved semantic segmentation using the correction model; and
• 3 a schematic representation of a motor vehicle that is set up for improved semantic segmentation of images of the surroundings.

In the figures, identical and functionally identical elements are provided with the same reference symbols.

1 FIG. 1 shows an exemplary schematic flowchart 10 for a method for generating or providing a model or a device for semantically segmenting image data. In it, in one process step S1 generates or provides erroneous semantic segmentations of digital image data. These can be generated, for example, using a model trained for the semantic segmentation of image data. Such a model can be provided in a forum that has already been trained, for example by using a model available from the prior art. Likewise, as part of method step S1, this model can first itself be generated or trained.

In a method step S2, the provided error-prone semantic segmentations are provided with true labels (ground truth labels), ie annotated, in particular pixel-by-pixel or pixel-by-pixel. In this way, training data is generated and made available.

In a method step S3, a trainable starting model is provided and trained to generate corrected semantic segmentations from error-prone semantic segmentations, ie corrected for the errors contained therein. This can be achieved in an iterative training process, which is indicated here by a corresponding loop-shaped program or flow path.

After completion of the training, a resulting trained correction model 26 (see 2 ) provided. The correction model 26 can also be used here with the or a model for the semantic segmentation, hereinafter referred to as the segmentation model 20 (see 2 ) can be combined to create or provide a corresponding overall model or facility for the semantic segmentation of image data.

2 shows a schematic representation to illustrate such a segmentation device 12 and its mode of operation. Here, the segmentation device 12 acquires a digital image 14 as input or input data. By way of example, this image 14 shows a pedestrian 16 in the foreground, a cyclist 18 in the middle ground as well as surrounding and background areas.

The segmentation device 12 includes the segmentation model 20, to which the image 14 is supplied as input. The segmentation model 20 can be or include a deep artificial neural network, for example, which is indicated schematically here. By processing the image 14, the segmentation model 20 automatically generates an associated preliminary semantic segmentation 22 of the image 14 from this. However, since the segmentation model 20 - unavoidably - cannot always work correctly one hundred percent, the preliminary semantic segmentation 22 contains some segmentation errors 24, of which here some are marked as examples. The segmentation errors 24 therefore indicate areas or pixels of the provisional semantic segmentation 22 which are objectively incorrectly classified by the segmentation model 20 or represent or relate to such areas or pixels. For example, although the segmentation model 20 has correctly classified some areas of the pedestrian 16 shown in the image 14, other areas that actually also belong to the pedestrian 16 within a perimeter or an outline of the pedestrian 16 or an area corresponding to the pedestrian 16, however, differently and thus incorrectly classified. Likewise, the segmentation model 20 has at least partially misclassified the cyclist 18 here.

The segmentation device 12 also includes the correction model 26. This can likewise be embodied or implemented as a deep artificial neural network or include one. In particular, the correction model 26 here has an autoencoder structure with an encoder part 28 on the input side, a downstream bottleneck 30 and a decoder part 32 on the output side. The correction model 26 has, as shown in FIG 1 described, an error behavior of the segmentation model 20, ie a structure of the segmentation error 24 made by the segmentation model 20 and its correction learned.

The provisional semantic segmentation 22 generated by the segmentation model 20, including the segmentation errors 24, is fed here to the correction model 26 as input. The correction model 26 processes the preliminary semantic segmentation 22, a reduced representation of the preliminary semantic segmentation being generated in the bottleneck 30 as an intermediate step or intermediate result. The correction model 26 corrects the segmentation errors 24 according to its training and thus generates a corrected semantic segmentation 34 as output. The correction model 26 therefore generates a prediction based on the preliminary semantic segmentation in the form of the corrected semantic segmentation 34, which is a true, i.e. actual or objective prediction corresponds to the correct semantic segmentation of the image 14 or at least comes closer to it than the preliminary semantic segmentation 22.

The segmentation device 12 can then output or provide the corrected semantic segmentation 34 generated in this way as an output, ie as the end result of the semantic segmentation of the respective image 14, for example for Use by other systems or facilities.

Such a semantic segmentation of images 14 can be used for fully or partially automated vehicles, for example. For this shows 3 a schematic representation of a detail of a motor vehicle 36. The motor vehicle 36 can, for example, support a vehicle occupant 38 in driving the motor vehicle 36 and/or can drive automated at least at times. For this purpose, the motor vehicle 36 has an environment sensor system for recording or detecting a respective surrounding traffic situation or environment of the motor vehicle 36 . In this case, this environment sensor system includes, in particular, an external camera 40 for optically recording the environment, i.e. for recording corresponding image data, for example image 14. Motor vehicle 36 also has an assistance system 42, which is set up based on images 14 recorded by means of external camera 40 to support the vehicle occupant 38 and/or to guide the motor vehicle 36 automatically at least at times.

For this purpose, the assistance system 42 has an input interface 44 via which the respective image 14 can be captured. During operation, the external camera 40 can continuously record images 14 or a corresponding video stream, which can be processed analogously as described here for the image 14 .

The assistance system 42 also includes a processor 46 and a computer-readable data memory 48. The segmentation model 20 and the correction model 26, for example, are stored in the latter. In other words, assistance system 42 includes or implements segmentation device 12.

Assistance system 42 also has an output interface 50 here. The assistance system 42 can, for example, output the corrected semantic segmentation 34 to other devices or systems of the motor vehicle 36 via this output interface 50 . Additionally or alternatively, the assistance system 42 can generate at least one control or information signal based on the corrected semantic segmentation 34 and output it via the output interface 50, for example to the vehicle occupants 38 and/or an operating device 52 of the motor vehicle 36, for example for controlling or driving the motor vehicle 36 or to support the vehicle occupant 38.

Overall, the examples described show how reverse error modeling can be realized and implemented for improved semantic segmentation of image data.

reference list

10

schedule

12

segmentation facility

14

picture

16

pedestrian

18

cyclist

20

segmentation model

22

preliminary semantic segmentation

24

segmentation error

26

correction model

28

encoder part

30

bottleneck

32

decoder part

34

corrected semantic segmentation

36

motor vehicle

38

vehicle occupant

40

outdoor camera

42

assistance system

44

input interface

46

processor

48

data storage

50

output interface

52

facility

S1 - S4

process steps

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• DE 112019000122 T5 [0003]
• DE 112019000048 T5 [0004]

Claims (10)

A method for semantically segmenting a digital image (14), wherein - the image (14) is supplied as input to a segmentation model (20) which is trained for the semantic segmentation of image data (14), - a provisional semantic segmentation (22) of the image (14) is automatically generated by the segmentation model (20) by processing the image (14), - the provisional semantic segmentation (22) is supplied as input to a correction model (26) which is trained to correct them using error-prone semantic segmentations (22), and - automatically a corrected semantic segmentation (34) of the image by the correction model (26) by processing the preliminary semantic segmentation (22) by adapting at least one classification (24) determined by the segmentation model (20) as part of the preliminary semantic segmentation (22). (14) is generated. procedure after claim 1 , characterized in that a respective deep artificial neural network is used at least as part of the segmentation model (20) and/or the correction model (26). Method according to one of the preceding claims, characterized in that the segmentation model (20) and the correction model (26) are trained to generate a pixel-precise semantic segmentation (22, 34) based on the respective image (14) as the respective output. Method according to one of the preceding claims, characterized in that the segmentation model (20) is trained for the semantic segmentation of images (14) representing traffic scenes and the image (14) fed to the segmentation model (20) as input represents a traffic scene. Method according to one of the preceding claims, characterized in that the correction model (26) is trained introspectively in relation to the respective provided segmentation model (20) using actual error-prone semantic segmentations (22) generated by this or a corresponding segmentation model (20). Method according to one of the preceding claims, characterized in that the correction model (26) when processing the preliminary semantic segmentation (22) internally at at least one point (30), in particular in at least one layer (30), compared to the preliminary semantic Segmentation (22) reduced representation of the preliminary semantic segmentation (22) used. Method according to one of the preceding claims, characterized in that an autoencoder with an input-side encoder part (28) and an output-side decoder part (32) is used as the correction model (26). Method (10) for providing a correction model (26) in which - as training data, in particular by a segmentation model (20) trained for the semantic segmentation of digital images (14), error-prone semantic segmentations (22) of images (14) and associated, in particular pixel-precise, true labels that indicate a respective correct classification, to be provided, - an initial model with a predetermined structure is provided, - the initial model is trained iteratively using the training data to reconstruct the associated true label (34) from the erroneous semantic segmentations (22), and - the initial model trained in this way is made available as the correction model (26). Assistance device (12, 42), in particular for a motor vehicle (36), having an input interface (44) for capturing digital images (14), a trained segmentation model (20) for automatic semantic segmentation of the captured images (14) and a segmentation model (20) downstream correction model (26) for automatically generating a corrected semantic segmentation (34) of the images (14) from a respective output (22) of the segmentation model (20). Motor vehicle (36), having a camera (40) for recording images (14) representing traffic scenes in a respective environment of the motor vehicle (36), and an assistance device (12, 42) coupled thereto claim 9 .

Images