DE102021133203A1 - Segmentation device and method for generating a corrected image segmentation and method for producing a segmentation device - Google Patents

Abstract

The invention relates to a method and a segmentation device (2) for generating a semantic segmentation (12) of an image and a method for producing a corresponding segmentation device (2). To generate the semantic segmentation (12), a compressed image (1) generated from original image data by compression is recorded. The compressed image (1) is then processed, regardless of its compression level, using a predetermined trained segmentation model (4) in order to generate a preliminary semantic segmentation (5). Based on this, a corrected semantic segmentation (12) is then generated using a predetermined correction model (9, 10), which is trained to correct segmentation errors (7) due to compression artifacts.

Description

The present invention relates to a method for generating a semantic segmentation of an image and a corresponding segmentation device. The invention further relates to a method for producing a segmentation device for generating a semantic segmentation of images.

Today, image data is used in a variety of forms for a wide variety of applications. In order to provide a basis for further automated processes, a semantic segmentation of the image data is often necessary. The problem, however, is that recording, temporarily storing, transmitting and processing the image data can involve a significant data processing effort and an associated resource effort, but only limited resources are available at at least one point in a corresponding data processing chain. This can be, for example, limited computing power and/or storage capacity in a mobile image-recording device or a vehicle or the like, a limited bandwidth of a wireless data connection, for example for transmitting an image to a server or another recipient, and the like. Accordingly, technically possible applications can often not be used or used or only with limited quality. For example, the quality of image data is often limited or reduced in order to thereby also reduce a corresponding amount of data to be processed and transported. However, this can lead to artefacts or errors in further processing of corresponding image data and thus ultimately, depending on the application, impair the accuracy, reliability or even certainty of results or decisions based thereon. Since data processing resources, such as computing hardware and transmission bandwidth, cannot easily be increased in many cases, there is a need for solutions to the problems mentioned that do not require a complex, costly and often impractical improvement or increase in the limiting resources.

For example, US 2021/0 073 949 A1 describes a method for improving the quality of digital images. Therein, an original image is segmented into a segmented image having a plurality of segments by means of an artificial neural network. A curve fit is then applied to the segmented image to determine border artifacts. The determined boundary artifacts are removed from the segmented image to generate a smoothed outline image. An enhanced image is then generated from the original image and the smoothed outline image.

Motor vehicles or assistance systems for motor vehicles represent an important application for the use of image data for different functions or purposes. At the same time, this is an area in which typically only limited resources are available for data processing and data transmission, which are not easily improved or increased can become. In this technical field, US 2021/0 063 198 A1 describes map generation and localization for autonomous driving applications. In this case, several method steps are carried out for at least two frames from a large number of frames corresponding to a journey of a vehicle. These include generating sensor data from a sensor of the vehicle, calculating outputs indicative of locations of landmarks based at least in part thereon using a neural network, and converting the locations to three-dimensional world coordinate locations relative to an origin of the vehicle. Furthermore, the three-dimensional world coordinate locations are encoded in order to generate encoded data corresponding to the respective frame. This encoded data is then sent to a server to cause it to generate a map that includes the landmarks.

The object of the present invention is to enable improved semantic segmentation in applications with limited data processing resources.

According to the invention, this object is achieved by the subject matter of the independent patent claims. Possible refinements and developments of the present invention are disclosed in the dependent patent claims, in the description and in the figure.

The method according to the invention serves and can therefore be used to generate a semantic segmentation of images. Such a respective image can be, for example, a single image recorded by a camera or a frame from a video sequence. The method according to the invention can thus be used, for example, in order ultimately to generate a semantic segmentation for an environment or traffic scene recorded with a camera of a motor vehicle. However, the method according to the invention can also be used for any images or types of images in a variety of applications or purposes.

In a method step of the method according to the invention, a compressed image generated from original image data by compression is recorded. In the context of the present invention, such a detection can mean, for example, that the compressed image is received via a data or input interface or a data connection or is retrieved or read from a data memory, for example by a data processing or segmentation device which is used for, in particular automatic or semi-automatic, running the method is set up. The original image data can be or include, for example, the recorded image mentioned, corresponding raw data or, for example, preprocessed data generated or derived from the raw data or the recorded image. In particular, the compressed image can be compressed using a classic or conventional method, such as JPEG compression. Depending on the application or, for example, for different images or original image data, different compressions or compression levels, ie, for example, different bit-per-pixel values, can be used.

The compressed image can be generated, for example, by the respective segmentation device, which is also set up to generate the final semantic segmentation. In this case, the segmentation device can, for example, capture and compress the original image data and store the compressed image thus generated, for example, in an intermediate memory. Capturing the compressed image can then mean or include reading it out or retrieving it from the buffer. Likewise, the compressed image can be generated, for example, by the camera used to record the image or a data processing or pre-processing device connected to it, which can then send the respective compressed image to the segmentation device, for example - be it directly or, for example, via one or more intermediate stations, such as a server or a backend of the like.

In a further method step of the method according to the invention, the respective recorded compressed image is processed independently of its compression level using a predetermined trained segmentation model in order to generate a preliminary semantic segmentation. In other words, a single trained segmentation model can be provided and used, for example, to process images compressed in different compression stages and to generate a respective provisional semantic segmentation for them. The respective compressed image thus serves here as an input for the segmentation model, while its output at least includes the respective preliminary semantic segmentation.

The segmentation model can be or include, for example, an in particular deep, in particular convolutional, artificial neural network. The segmentation model can be trained on the basis of training data, which can include a large number of training images, in particular pixel-by-pixel or pixel-precise, semantically annotated. In this case, in particular, images compressed with or in different compression stages can be used. Ultimately, however, an at least essentially arbitrary segmentation model that is trained to generate semantic segmentations from images or image data can be used here.

The output generated by the segmentation model is referred to here as preliminary semantic segmentation, since it is usually subject to errors, in particular due to the compression of the compressed image serving as input. Since compression methods are often used for image data in order to reduce their size, ie data volume, the compression can in particular be associated with a loss of information and compression artifacts can arise which make correct semantic segmentation more difficult or can lead to segmentation errors.

In a further method step of the method according to the invention, a corrected semantic segmentation is generated based on the provisional semantic segmentation, which can also be referred to here as erroneous or erroneous semantic segmentation, using at least one predetermined correction model. The provisional semantic segmentation can therefore be used directly as input for the correction model or initially processed intermediately or further, for example re-encoded or combined with further data in order to generate an input for the correction model. The predefined correction model is trained to correct segmentation errors that have arisen due to compression artifacts. The corrected semantic segmentation can reproduce or at least approximate a correct semantic segmentation of the underlying captured image or the original image data. The processing of the preliminary semantic segmentation by means of the correction model can effectively be understood as post-processing, ie post-processing of the preliminary semantic segmentation generated by the segmentation model become. The provisional semantic segmentation can be updated pixel-by-pixel or pixel-by-pixel and ultimately output or provided as a corrected or final semantic segmentation, for example stored in a specified or data memory or sent to a further data processing device.

The predefined correction model can be or include, for example, a trained, in particular deep, in particular convolutional, artificial neural network. The correction model can be trained with compressed training image data, which can be annotated, ie labeled, with the, in particular pixel-precise, correct semantic segmentation of the respective underlying uncompressed image as a target variable.

The segmentation model used to initially semantically segment each compressed image can be thought of as an implicit error function that assigns incorrect values or classifications to some areas or pixels of each correct semantic segmentation or semantic map of the original uncompressed image data from which each compressed image was generated . This correct semantic segmentation of the original image data represents the target variable to be determined and is also referred to as ground truth in technical terms.

During its training, the correction model can effectively learn or have learned to apply or to approximate a corresponding inverse error function, ie the inverse function of the error function implicitly applied by the segmentation model. In this case, different inverse error functions can be learned by the correction model for different compression stages, or a respective correction model can be or has been trained for a number of different compression stages, for example. By using the actual incorrect predictions, i.e. the preliminary semantic segmentations generated by the segmentation model, as degraded or error-prone semantic segmentations and as input for the correction model, no corresponding degradation or error function has to be determined, estimated or designed manually.

To train the at least one correction model, error-prone semantic segmentations generated by the respective segmentation model can be used as training input and the ground truth annotations, i.e. the respective correct semantic segmentations of the underlying uncompressed original image data, can be used as target variables. The correction model is then trained to reconstruct the correct semantic segmentations from the erroneous semantic segmentations.

The present invention makes it possible to correct segmentation errors in the segmentation model, which are ultimately caused by the compression of the original image data. A single segmentation model can thus be used for a wide range of different compression levels.

The training of one or more correction models can be significantly less complex than the training of complete individual segmentation models for each of the different compression levels. Despite the use of at least two different models proposed here, namely the segmentation model and the at least one correction model, an ultimately improved segmentation result can be achieved with comparatively little effort.

Compared to previous applications, the present invention does not require any additional data processing resources in the chain or pipeline from the recording camera to the segmentation device in order to achieve an improved segmentation result. If necessary, bandwidth can even be saved by additional compression of the original image data, which would previously have led to unusable segmentation results, without reducing the functionality, reliability and security of applications that are based on the semantic segmentation, i.e. use it as an input or database. Likewise, for example, in cases where compression of the original image data has already been used, an improved, ie in particular more accurate and/or more reliable, segmentation result can be achieved. This can make applications based on it more accurate, more reliable and/or more secure. In the application of fully or partially automated driving, more robust decisions can be made or more situations can be dealt with automatically by the present invention, for example by appropriate driving or vehicle guidance assistance systems.

The application of the correction model proposed in the present invention to the preliminary semantic segmentations generated by the segmentation model can be understood or used as a final post-processing step of the preliminary semantic segmentations be det, in particular regardless of how the preliminary semantic segmentations were created or generated. Thus, the correction model provided according to the invention can easily be combined with various compression schemes and/or segmentation methods that have already been proposed for improving the final segmentation result.

In one possible embodiment of the present invention, several trained correction models are provided, which are trained to correct segmentation errors that occur during the semantic segmentation of images compressed in different compression stages, that is, made by the segmentation model. Here the compression level, a dedicated correction model trained for or on the basis of this, in particular only this, compression level is generated or provided. For example, the different correction models can be stored in a data store of the segmentation device mentioned. The compression level is then automatically determined for the compressed image recorded in each case. This can, for example, be read from metadata of the respective compressed image or estimated based on its file size.

The correction model corresponding to the determined compression level of the respective compressed image is then selected, ie determined, from the correction models provided. The respectively corresponding correction model can be the correction model specifically trained for the determined compression level. If no correction model is available for the determined compression level, the correction model provided that comes closest to the compression level determined for the respective compressed image in terms of the compression level for which it is trained can be used as the corresponding correction model

The respectively selected corresponding correction model is then used to generate the corrected semantic segmentation based on the preliminary semantic segmentation of the respective compressed image. For this purpose, for example, only this correction model can be loaded, for example into a working memory of the segmentation device. As also mentioned elsewhere, a number of different correction models can be trained and made available for use or kept available with comparatively little effort, for example in comparison to a number of different segmentation models. Due to the different correction models, each trained for a specific compression level, particularly good segmentation results can also be achieved in more complex or dynamic applications or across a large number of different applications in which different compression levels are used.

For example, the level of compression used in an application can be dynamically adjusted depending on a particular available bandwidth of an unstable or changing data link used to transmit or capture the compressed images. In an ongoing or continuous operation, different frames or frame sequences of a video stream can then be compressed in different compression stages, for example. Accordingly, the correction model used in each case can then be dynamically changed in order to generate a corresponding stream or a corresponding sequence of corrected semantic segmentations for the frames. Thus, particularly good, in particular independent of the compression level used - i.e. the compression level or degree of compression used - of the compressed images, at least essentially consistently good or reliable corrected semantic segmentations can be generated in a practicable manner, since the considerable effort for a Training and deploying according to different segmentation model can be saved.

In a further possible embodiment of the present invention, a respective artificial neural network with a completely convolutional encoder-decoder layer structure or architecture is used as the at least one correction model. The correction model can in particular include an autoencoder or be designed as an autoencoder. According to one finding on which the present invention is based, such an embodiment of the correction model can deliver particularly good results with particularly little effort. In particular, particularly good segmentation results can be achieved in applications in which there are a large number of different classes of objects depicted in the image data, which also differ significantly in terms of their shapes and/or relative positions and/or can vary from image to image. Such an application is, for example, the semantic segmentation of image data of a respective vehicle environment recorded in a real traffic situation by means of a camera of a motor vehicle. The completely folding layer structure can minimize a loss of spatial information, for example in contrast to use of fully connected layers in latent space.

In one possible development of the present invention, the artificial neural network used as the at least one correction model has a number of downsampling layers followed by a number of upsampling layers. A rectifier activation function—also known as a rectifier or rectifying linear unit (ReLu)—is applied after each of these layers, with the exception of the last layer on the output side. The at least one correction model can thus be implemented in a particularly simple and effective manner. This is based on the knowledge that such a comparatively simple architecture, which can be used with correspondingly little calculation effort, is sufficient for the application described here, since semantic segmentations or semantic maps are already used as input for the correction model, which are already used in the processing of the segmentation model in in a highly abstract space, for example in comparison to the original image data. In contrast to the segmentation model, the correction model does not have to extract any low-dimensional features from RGB image data, for example. Instead, the correction model only needs to learn or have learned how the semantic classes predicted by the segmentation model, ie output and specified in the provisional semantic segmentation, are related to the actually correct semantic classes.

Due to the relatively low effort associated with training, providing and using the at least one correction model, the at least one correction model can be supplemented particularly easily as a post-processing step in almost any previous segmentation application in the manner proposed here.

In a further possible embodiment of the present invention, to generate the input for the correction model, based on which the respective correction model then generates the corrected semantic segmentation, a 1-out-of-n coding is applied to the respective provisional semantic segmentation generated by the segmentation model referred to as one-hot encoding - applied. This can be particularly useful when the semantic classes of the objects represented in the imagery do not have a natural order or hierarchy. In such cases, one-hot coding can improve the accuracy of the correction model and avoid the occurrence of unexpected or unusable outputs, such as predictions lying between two classes or categories, by not allowing the correction model to follow a specific order of the semantic to accept classes or categories.

In a further possible embodiment of the present invention, a softmax evaluation output by the segmentation model when processing the respective compressed image is also used for the input of the correction model, based on which the respective correction model then generates the corrected semantic segmentation. The soft-max evaluation can be a pixel-precise specification of results or outputs of the or a soft-max function that is present or generated on the output side of the segmentation model. The softmax rating used here is also referred to as the softmax score. The softmax evaluation can indicate to the correction model a confidence of the segmentation model in its semantic classification, ie in the provisional semantic segmentation. By using the softmax evaluation, the correction model can be provided with more information about which pixels of the provisional semantic segmentation are expected to be corrected or have a particularly high probability of being corrected. A further improved segmentation result can ultimately be achieved by the embodiment of the present invention proposed here.

In a possible development of the present invention, the 1-out-of-n coding of the provisional semantic segmentation is combined with the softmax evaluation to generate the input for the correction model, based on which the respective correction model then generates the respective corrected semantic segmentation. For this purpose, for each pixel of the provisional semantic segmentation, the respective 1-out-of-n vector, ie the vector of the one-hot coding assigned to the respective pixel, and the softmax evaluation for the respective pixel are multiplied with one another. In this way, the data or information mentioned can flow into the input of the correction model without changing its dimensionality, for example. This can enable a particularly simple use of the input thus generated for the correction model and a particularly simple generation of the corrected semantic segmentation in the same dimensionality or size as the preliminary semantic segmentation. Thus, ultimately, the preliminary invention can be applied or implemented in a particularly simple manner.

Another aspect of the present invention is a method of manufacturing a segmentation device for generating a semantic segmentation of images. in one Method step of this manufacturing method, segmentation training data for a segmentation model is generated or provided or recorded. This segmentation training data comprises a plurality of data tuples, each comprising an image in an uncompressed form and/or at least one compressed form as input and a correct semantic reference segmentation of the uncompressed image as target output. Different compression levels can be used for different images. Likewise, the data tuples or at least some of the data tuples can, for example, contain forms or versions of the same image compressed to different extents, that is to say in different compression stages.

In a further method step of the method according to the invention, the segmentation training data is used to train one, in particular precisely one, segmentation model for the semantic segmentation of images, in particular independently of their compression level.

In a further method step of the manufacturer method according to the invention, correction training data are provided for at least one correction model. This correction training data comprised a large number of data tuples, each comprising at least one error-prone semantic segmentation generated by the trained segmentation model by processing a compressed image as input and the associated correct semantic reference segmentation of the uncompressed form of the respective image as target output.

In a further method step of the manufacturer method according to the invention, at least one correction model for generating corrected semantic segmentations of images from error-prone semantic segmentations is trained using the correction training data. In this case, in particular, a plurality of correction models can each be trained for a specific compression level.

In a further method step of the method according to the invention, the trained segmentation model and the at least one trained correction model are loaded into a data memory of the segmentation device. The segmentation device can also include a processor device coupled to this data memory, ie for example a microchip, microcontroller or microprocessor or the like, for executing the models stored in the data memory.

The segmentation device generated by means of the manufacturer method according to the invention can in particular be or correspond to the segmentation device mentioned in connection with the method according to the invention for generating a semantic segmentation. The same can apply to the training data mentioned, the segmentation model and the at least one correction model. The segmentation device generated according to the method according to the invention can therefore be generated or manufactured in such a way that it is set up for, in particular automatically or semi-automatically, executing at least one configuration or development relationship or embodiment of the method according to the invention for generating a semantic segmentation of an image.

In a possible development of the present invention, several different sets of correction training data are generated or provided. These different sets of correction training data differ from one another in terms of the compression levels of the compressed images from which the erroneous semantic segmentations contained in the correction training data were generated. Each set of correction training data from compressed images only includes error-prone semantic segmentations generated from a specific compression level. Then, by training with one of the sets of correction training data, several different trained correction models are then generated and loaded into the data memory of the segmentation device. In this way, the different correction models can be specialized for a specific compression level and thus particularly accurately and reliably correct provisional semantic segmentations that were generated by the segmentation model from images compressed in these respective compression levels. In this way, particularly accurate corrected semantic segmentations can ultimately be generated even when using different compression stages for compressing the original image data.

For example, JPEG 90% quality, JPEG 50% quality and JPEG 10% quality can be used as different compression levels. However, other and/or further quality gradations as well as other compression methods can also be used.

Another aspect of the present invention is a segmenter that includes an input interface for detecting compressed th images, a processor device, a data memory coupled thereto and an output interface for outputting corrected semantic segmentations generated from the compressed images captured. The segmentation device according to the invention is set up to execute the method according to the invention for generating a semantic segmentation of an image and/or produced according to the method according to the invention for producing a segmentation device for generating a semantic segmentation of images. The segmentation device according to the invention can therefore in particular be or correspond to the segmentation device mentioned in connection with one or both of the methods according to the invention. The segmentation device according to the invention can be or include, for example, a server device, a backend or cloud server, a data processing device or at least part of an assistance system of a motor vehicle, a motor vehicle and/or the like.

Further features of the invention can result from the claims, the figures and the description of the figures. The features and feature combinations mentioned above in the description and the features and feature combinations shown below in the description of the figures and/or in the figures alone can be used not only in the combination specified in each case, but also in other combinations or on their own, without going beyond the scope of the invention to leave.

In the single figure, the drawing shows a schematic overview representation to illustrate the generation of a corrected semantic segmentation for a compressed image.

A semantic segmentation of image data allows an automated, pixel-precise understanding of a displayed scene, for example in the context of computer vision. Therefore, for example, semantic segmentation can be an important task for various applications such as automated driving or driving a motor vehicle. However, one hundred percent accuracy cannot be achieved with previous architectures and devices for the semantic segmentation of images, so that segmentation errors, ie semantic misclassifications, occur again and again. This can be the case in particular when the image quality of the semantic image data to be segmented is reduced, for example due to compression.

For applications in the field of automated driving, image data can be recorded by motor vehicles on the road, i.e. collected and sent to a central server. In order to make this practicable, however, depending on the conditions or qualities or bandwidths of available network or data connections from the vehicles to the server, compression of the image data can be used or be necessary. Different compression levels can be used in different situations or at different times, ie the image data can be compressed to different extents in order to enable reliable and sufficiently fast transmission to the server. However, bottlenecks in available bandwidths, transmission capacities, storage capacities and/or calculation resources can also occur within a vehicle, for example, which may require compression of recorded image data.

The image data compressed in this way can be segmented semantically, depending on the situation or application, for example in the recording vehicle or by the server. Due to compression-related data loss and compression artefacts, there can be a significant degradation of the performance or the quality of the results of the semantic segmentation compared to the semantic segmentation of uncompressed image data. However, due to the relatively high expenditure of time and money, it is not practicable, for example, to train a specific model for the semantic segmentation for each individual compression level.

Against this background shows 1 a schematic overview to illustrate how the mentioned problems and challenges can be met.

A compressed image 1 can be generated from uncompressed original image data (not shown here) by using any desired compression method. This can then be provided as a respective input to a segmentation device 2 or recorded by it. For this purpose, the segmentation device 2 can have an input interface 3 indicated schematically here.

The compressed image 1 recorded in this way is then processed in the segmentation device 2 by means of a predetermined segmentation model 4 trained for the semantic segmentation of images. The segmentation model 4 generates a provisional semantic segmentation 5 and a pixel-by-pixel softmax evaluation 6 as output Softmax evaluation 6 indicates the associated softmax score calculated by segmentation model 4 for each pixel of compressed image 1 or provisional semantic segmentation 5 . The provisional semantic segmentation 5 specifies the semantic class determined by the segmentation model 4 for each pixel. Due to the compression of the compressed image 1, however, the segmentation model 4 will probably make errors during the semantic segmentation, so that the provisional semantic segmentation 5 contains corresponding segmentation errors 7, some of which are indicated here by way of example.

wobei S das jeweilige Modell, hier also das Segmentierungsmodell 4 bezeichnet.For a given data set of images I _i and corresponding pixel-precise labels or annotations L _i , the segmentation model 4 can be or will be trained to assign a prediction, ie a predicted annotation or predicted label L _{i,pred ,} to each pixel of the images I _i . So it applies $${\text{L}}_{\text{i}\text{,preach}}=\text{S}\left({\text{I}}_{\text{i}}\right),$$

where S denotes the respective model, ie the segmentation model 4 here.

Pixel-by-pixel differences or discrepancies between the predictions L _i,pred and actually correct annotations or ground truth labels L _i can be interpreted as errors introduced by the segmentation model 4 . In other words, it can be assumed that the segmentation model 4 generates an error function Es or applies it to the ground truth label L _i : $${\text{L}}_{\text{i}\text{,preach}}={\text{E}}_{\text{S}}\left({\text{L}}_{\text{i}}\right).$$

The ground truth labels L _i are only known for given training data, i.e. during the training of the segmentation model 4, but not for a later intended or productive use of the segmentation model 4 for the semantic segmenting of any or unknown image data, i.e. during a test use of the segmentation model 4 for the semantic Segment test data. During such a test use, only the predicted L _i,pred , but not the ground truth label L _i for the respective image data, are available. The error function E _S can be a highly complex function that is not explicitly known but implicitly encoded or represented in the structure and the learned parameters of the segmentation model 4 . If it were possible to invert the error function E _S , the ground truth labels L _i could be reconstructed from the predicted labels L _i,pred according to $${\text{L}}_{\text{i}}={\text{E}}_{\text{S}}{}^{-1}\left({\text{L}}_{\text{i}\text{,preach}}\right).$$

In practice, however, the error function Es is not explicitly known and/or cannot be inverted or used, for example in cases in which the segmentation model 4 does not even recognize an object that is present in the image data, i.e. that is mapped, and accordingly also not in the predicted labels L _{i ,pred} indicated or represented.

In order to counteract this problem, a model is trained to effectively approximate the inverse error function Es ^-1 as best as possible. Such a model is provided here as a correction model 9 and is stored in the segmentation device 2 .

In order to generate an input for the correction model 9, the output of the segmentation model 4 is processed 8. As part of this processing 8, one-hot coding can be applied to the provisional semantic segmentation 5 and weighting with the softmax rating 6.

The input generated in this way for the correction model 9 thus includes the labels L _i,pred predicted by the segmentation model 4 for the respective compressed image 1 or is at least based on these. The correction model 9 is trained to predict or reconstruct and output the ground truth label L _i from such an input.

Due to the nature of the data to be processed here, the correction model 9 can be generated or provided with a convolutional autoencoder architecture. The correction model 9 can have three downsampling layers and three upsampling layers following these, for example. Ultimately, depending on the requirements or application, different numbers, types and/or linkages of the layers as well as different kernel sizes and/or increments are possible in order to implement the correction model 9 .

Since the compressed image 1 or a multiplicity of compressed images 1 to be processed, ie to be semantically segmented, can be compressed in different compression levels, the correction model 9 can be trained accordingly for different compression levels. In other words, the correction model 9 can be trained to use the ground truth label L _j from different preliminary semantic segmentations 5, which come from the segmentation model 4 in different compression stages primed images 1 were generated to predict or to reconstruct.

In order to achieve improved results for different compression levels of the compressed images 1, however, a number of further correction models 10 specifically trained for each compression level are provided in the segmentation device 2. The segmentation device 2 can then determine or read out the compression level used for the compressed image 1 to be segmented and automatically load the corresponding correction model 9, 10 for use. The input generated by the processing 8 is then only supplied to the respectively corresponding or loaded correction model 9, 10.

The correction models 9, 10 can be trained using preliminary semantic segmentations 5 generated by the segmentation model 4 actually used. When training models or artificial neural networks, overfitting to the respective training data can be problematic. However, it has been shown that learning errors made by a model 4 when processing training data can sometimes be useful, for example for predicting segmentation errors 7 in the semantic segmentation of image data. Such segmentation errors 7, which the segmentation model 4 made when processing the training data, will also be made with a relatively high probability when processing non-annotated test data, in this case, for example, when processing the compressed image 1. The segmentation errors 7 are therefore a useful source of information or data on which segmentation errors 7 of the segmentation model 4 are also to be expected at least for test data, ie when the segmentation model 4 is used productively. Thus, by means of the correspondingly trained correction models 9, 10, an at least partial correction of these segmentation errors 7 can be achieved.

An output generated by the respective correction model 9, 10 can be output or made available, for example, via an output interface 11 of the segmentation device 2. This output can be a corrected semantic segmentation 12 with corrections 13 instead of the segmentation errors 7 still contained in the provisional semantic segmentation 5 .

This corrected semantic segmentation 12 can then be provided or used as a basis or database for further applications.

Overall, the examples described show how errors in a semantic segmentation, which were caused by a compression of a respective image to be semantically segmented, can be corrected in a particularly simple and effective manner in order ultimately to produce an improved semantic segmentation of the respective image.

Reference List

1

compressed image

2

segmentation facility

3

input interface

4

segmentation model

5

preliminary semantic segmentation

6

Softmax rating

7

segmentation error

8th

processing

9

correction model

10

another correction model

11

output interface

12

corrected semantic segmentation

13

correction

Claims (10)

Method for generating a semantic segmentation of an image, wherein - a compressed image generated from original image data by compression is captured, - the compressed image is processed independently of its compression level using a predefined trained segmentation model in order to generate a preliminary semantic segmentation, - a corrected semantic segmentation is generated based on the provisional semantic segmentation by means of a predetermined correction model which is trained to correct segmentation errors due to compression artifacts. procedure after claim 1 , characterized in that - several trained correction models are provided, which are trained to correct segmentation errors occurring during the semantic segmentation of images compressed in different compression levels, - the compression level of the respective compressed image is determined automatically, and - from the correction models provided, the the determined compression level of the respective compressed image corresponding correction model is selected and used to generate the corrected semantic segmentation. Method according to one of the preceding claims, characterized in that an artificial neural network, in particular an autoencoder, with a completely convolutional encoder-decoder layer structure is used as the correction model. procedure after claim 3 , characterized in that the artificial neural network used as a correction model has a plurality of downsampling layers followed by a plurality of upsampling layers, a rectifier activation function being applied after each of these layers, with the exception of the last output-side layer. Method according to one of the preceding claims, characterized in that a 1-out-of-n coding is applied to the provisional semantic segmentation generated by the segmentation model in order to generate an input for the correction model. Method according to one of the preceding claims, characterized in that a soft-max evaluation output by the segmentation model when processing the respective compressed image is also used for an input of the correction model. procedure after claim 5 and 6 , characterized in that to generate the input for the correction model, the 1-out-of-n-coded provisional semantic segmentation and the softmax evaluation are combined with each other by the respective 1-out-n vector and for each pixel of the provisional semantic segmentation the respective Softmax rating are multiplied together. Method for producing a segmentation device for generating a semantic segmentation of images, in which - segmentation training data are provided for a segmentation model, which comprise a large number of data tuples, the data tuples each comprising an image in an uncompressed form and/or at least one compressed form as input and a correct semantic reference segmentation of the uncompressed image as target output, - using the segmentation training data, a segmentation model for the semantic segmentation of images is trained, in particular independently of their compression level, - correction training data are provided for a correction model, which comprise a large number of data tuples from at least one error-prone semantic segmentation generated by the trained segmentation model by processing a compressed image as input and the correct semantic reference segmentation of the uncompressed form of the respective image as target output, - a correction model for generating corrected semantic segmentations of images is trained by means of the correction training data, and - The trained segmentation model and the trained correction model are loaded into a data memory of the segmentation device. procedure after claim 8 , characterized in that - several different sets of correction training data are provided, which differ from each other by the compression levels of the compressed images from which the error-prone semantic segmentations contained in the correction training data were generated, each set of correction training data from compressed images of only one specific Compression level generated error-prone semantic segmentations, - are generated by training with one of the sets of correction training data several different trained correction models and loaded into the data memory of the segmentation device. Segmentation device, comprising an input interface for capturing compressed images, a processor device, a data memory coupled thereto and an output interface for outputting corrected semantic segmentations generated from captured compressed images, the segmentation device for executing a method according to one of Claims 1 until 7 established and/or according to a method claim 8 or 9 is made.

Images