DE102021103200B3 - Method for determining a degree of degradation of a recorded image, computer program product, computer-readable storage medium and assistance system - Google Patents

Abstract

The invention relates to a method for determining a degree of degradation (3) of an image (5) recorded by means of a camera (4) of an assistance system (2) of a motor vehicle (1) using the assistance system (2), with the steps: 5) using the camera (4); - carrying out a deep feature extraction of a large number of pixels (8) of the image (5) using a coding module (9) of an electronic computing device (6) of the assistance system (2); - clustering the large number of pixels (8) by means of a feature point cluster module (10) of the electronic computing device (6); - regressing the clustered pixels (8) by means of a regression module (11) of the electronic computing device (6); and- determining the degree of degradation (3) as a function of an evaluation by applying a sigmoid function (20) after regression using a sigmoid function module (12) of the electronic computing device (6) as the output of the sigmoid function module (12).Furthermore, the invention relates to a computer program product, a computer-readable storage medium and an assistance system (2).

Description

The invention relates to a method for determining a degree of degradation of an image recorded by a camera of an assistance system of a motor vehicle using the assistance system. Furthermore, the invention relates to a computer program product, a computer-readable storage medium and an assistance system.

In particular, cameras are known from motor vehicle construction which can record an image of the surroundings of the motor vehicle. In particular, what are known as surround-view cameras are known, which can be adversely affected in particular by harsh environmental conditions such as snow, rain, mud and the like. In order to detect such an impairment, it is already known that special pollution classes, such as soil and water droplets, can be determined and recognized depending on the pollution. This creates major challenges when annotating only these few classes. In particular, the transition areas are to be annotated subjectively and it is difficult to obtain realistic pollution images with the corresponding variety. In particular, it is not yet possible to determine lens contamination based on mud, sand, water droplets and frost, as well as other adverse weather conditions such as snow, rain or fog, which in particular produce lighting-related deterioration such as weak light, glare, shadows or motion blur, whereby these must also be taken into account in the future. However, it is impractical to create and annotate additional data sets for each of these individual classes and with all combinations of degradation.

the U.S. 2018/315167 A1 discloses an object recognition method in which an object such as a vehicle can be recognized from an image captured by an on-board camera even if a lens of the on-board camera is dirty. In order to achieve the object, in this object recognition method for recognizing an object included in a captured image, an original image containing the object to be recognized is created from the captured image, a processed image is created from the created original image by applying predetermined processing to the original image is applied, and learning is performed on reconstructing an image of the object to be recognized using the original image and the processed image.

the EP 36 57 379 A1 discloses a neural network image processing apparatus that obtains a first image and a respective additional image, wherein a first image capturing device has a different field of view than each additional image capturing device. Each image is processed by respective instances of a common feature extraction processing network to produce a respective first map and at least one additional map. Feature classification includes processing the first map to generate a first classification indicative of contamination of the first image. Processing the or each additional map to generate at least one additional classification indicative of a corresponding contamination of the or each additional image. The first and each additional classification are combined to produce an enhanced classification indicative of contamination of the first image. If the reinforced classification indicates no contamination of the first imaging device, further processing can be performed.

the US2016/0379067 discloses a camera system for a vehicle, the camera having an image sensor and a lens that is exposed to the environment outside of the vehicle. The image sensor is adapted to process multiple frames of image data captured by the camera and processes captured image data to detect a blob in a frame of captured image data.

In SoildNet: Soiling Degradation Detection in Autonomous Driving by Arindam Published in arXiv preprint arXiv:1911.01054v2, 2019, a Deep Convolutional Neural Network (DCNN)-based baseline network as well as exploiting several network remodeling techniques such as . B. using static and dynamic group convolution, channel reordering to compress the baseline architecture and make it suitable for low power embedded systems with ~1 TOPS are proposed. Thereby different result metrics of all intermediate networks intended for a contamination deterioration detection on the tile level of the size 64×64 on the input resolution 1280×768 are compared.

The object of the present invention is to create a method, a computer program product, a computer-readable storage medium and an assistance system, by means of which an improved determination of a degree of degradation of a camera can be carried out.

This object is achieved by a method, a computer program product, a computer-readable storage medium and an assistance system according to the independent patent claims. Advantageous embodiments are specified in the dependent claims.

One aspect of the invention relates to a method for determining, by means of the assistance system, a degree of degradation of an image recorded by means of a camera of an assistance system of a motor vehicle. The image is captured by the camera and a deep feature extraction of a large number of pixels of the image is carried out by means of a coding module of an electronic computing device of the assistance system. The plurality of pixels are clustered using a feature point clustering module of the electronic computing device. The clustered pixels are regressed using a regression module of the electronic computing device. The degree of degradation is determined as a function of an evaluation by applying a sigmoid function after the regression using a sigmoid function module of the electronic computing device as the output of the sigmoid function module.

In particular, an improved determination of the degree of degradation can thus be carried out. In particular, the degree of degradation can be between 0 and 1, for example, where 0 can indicate that there is no degradation, ie the lens is clear or clean, for example, and 1 can mean that the lens is dirty. The closer the degree of degradation is to 0, the cleaner the lens. In particular, this is therefore independent of a classification of the contamination. The degradation is only generally determined and whether, for example, the image with the corresponding degree of degradation can then be used further, for example in order to use an evaluation of this image for a driving function.

In particular, the invention thus solves the problem that a corresponding degree of degradation can nevertheless be determined independently of a classification and thus also independently of a large number of data sets for contamination.

The assistance system is therefore easy to train, with a small number of data sets being necessary for this in particular.

In particular, the deterioration in degradation is thus estimated between 0 and 1 using the sigmoid function, with 0 meaning clean and 1 meaning opaque. In particular, the two cases mentioned above are extremes for which corresponding annotations are known. The quality of degradation deterioration is determined in the range between 0 and 1 due to the presence of the regression module.

According to an advantageous embodiment, the coding module is provided as a convolutional neural network. The coding module can also be referred to as an encoder. In particular, a simple convolutional neural network (CNN) is used to extract depth features from the corresponding input images. In particular, if there is sufficient memory and computing support, for example, a neural network with a higher capacity can also be used. In particular, a feature extraction can thus be carried out reliably.

It has also proven to be advantageous if the multiplicity of pixels is clustered using a K-means algorithm of the feature point cluster module. However, especially since the annotations are only available for two extreme cases each, namely clean and opaque, during the inference time the input image will most likely show a varying degradation in the range between 0 and 1. In order to learn this variation, the K-means clustering method with a number of, for example, n clusters, where n can be 5, for example, is used in particular. It is advantageous to first distinguish the depth feature points in n clusters and later each of these clusters is passed through a regression module. The K-means algorithm is in particular a method for vector quantization which is also used in particular for cluster analysis. A previously known number of K-groups is formed from a set of similar objects.

It is also advantageous if the clustered pixels are regressed using a regression module designed as a long-term storage module. The long-short-term memory module is in particular what is known as a long-short-term memory module (LSTM). In particular, the problem presented is not a classification problem, but a regression problem.

The output from the K-means clustering in particular is nothing more than a series of depth feature points separated into clusters, which is why the LSTM module in particular is advantageous for regressing. The core concept of such an LSTM module is the cell state and its various gates. Cell states act like a memory of the network and can provide relevant information during the processing of the sequence carry information. The gates are various neural networks that decide what information is allowed on the cell status. During training, the gates can learn which information is relevant in order to remember or forget it. Each gate contains sigmoidal activations. This is useful for updating or forgetting dates, since any number multiplied by 0 is 0, making values disappear or "forget". Any number that is multiplied by 1 is the same value, therefore this value remains the same or is "kept". In such an LSTM module in particular the so-called forget gate is formed first. This gate decides what information to discard or keep. Information from the previous hidden state and information from the current input are passed through the sigmoid function. This results in values between 0 and 1. The closer to 0, the more likely it is to be forgotten, and the closer to 1, the more likely it is to be retained.

In a further advantageous embodiment, the clustered pixels are transmitted unidirectionally from the feature point cluster module to the regression module. In particular, the connection from the encoder to the LSTM module via the K-means algorithm is unidirectional and is only used for the forward pass since unsupervised K-means clustering is used.

Furthermore, it has proven to be advantageous if the regressed pixels are propagated back to the coding device. In particular, to create an unconnected graph, the back propagation is done via a separate connection from the LSTM module to the encoder. It is also discouraged to use supervised K-means and make the connection bi-directional as it has few trainable parameters which are insufficient in backpropagation since the gradient flow is done by a large number of trainable parameters from the LSTM module.

In a further advantageous embodiment, a sigmoid loss is trained for the application of the sigmoid function in a first training phase for the assistance system. In particular, sigmoid is an advantageous function here so that the values in the range between 0 and 1 are obtained. The reason for this is that there is no continuous annotation, only annotations for 0 and 1. In particular, the sigmoidal loss function is trained in the first phase, which estimates the quality of the deterioration in perception, i.e. the degradation, from an input image. However, this flow cannot output the degradation deterioration quality per pixel, so a functionality is added that generates the output in the form of the input image.

It has also proven to be advantageous if the regressed pixels are discriminated by means of a self-awareness module of the electronic computing device and the discriminated pixels are transmitted to the sigmoid function module. The self-awareness module is in particular a so-called attention map. In particular, so-called “self attention” can be carried out using the Attention Map. In particular, this is an advantageous step to discriminate the output of the LSTM.

It is also advantageous if the self-awareness module is provided in the form of global averaging. The global averaging is in particular a so-called Global Average Pooling (GAP). This is applied to the output of the LSTM and multiplied by a one-dimensional vector.

In a further advantageous embodiment, the output of the sigmoid function module is transmitted to a decoding device of the electronic computing device for decoding the output. In particular, a decoder is thus added which accepts the output of the sigmoid function module in one-dimensional format. In particular, this one-dimensional vector is converted into a two-dimensional vector by reshaping and then in turn fed to the decoder for reconstruction. This reconstruction can then in turn be transmitted to a higher-level assistance system, in which case the higher-level assistance system can then decide on the basis of the reconstruction whether the image from the camera can be used for the evaluation or not. If, for example, the image cannot be used for the evaluation, an alarm signal can be generated for a user of the motor vehicle. Furthermore, it can also be provided that, for example, the credibility of the image for an at least partially automated operation of the motor vehicle is downgraded if a corresponding degradation value is present. Safe operation of the motor vehicle can thus be implemented.

In a further advantageous embodiment, the decoding device is provided in the form of a fully folded neural network. The fully convoluted neural network is in particular a so-called fully convoluted network (FCN). In particular, it has turned out that this is very advantageous in order to be able to further process the output of the sigmoid function module.

In a further advantageous embodiment, the decoding device is trained for the assistance system in a second training phase, with only the decoding device being trained in the second training phase, which occurs after the first training phase. In particular, the pre-trained model from the first phase is used. It is known that these are two simultaneous problems. In particular, the degree of degradation is estimated in the first phase and the output is reconstructed in the second phase. In the second phase, therefore, only the decoder is trained, the remaining components of the assistance system or the so-called "pipeline" remain untrainable. Otherwise, the decoder's gradient would destroy most of the trained weights, including the encoder, in the first few epochs. Therefore, the components used in the first training phase are used as feature extractors only in the second phase.

The method presented is in particular a computer-implemented method. The method is carried out in particular on an electronic computing device, in which case the electronic computing device can have circuits, for example integrated circuits, processors and other electronic components, in order to carry out the corresponding method steps.

A further aspect of the invention therefore relates to a computer program product with program code means, which are stored in a computer-readable storage medium, in order to carry out the method for determining a degree of degradation according to the preceding aspect, when the computer program product is processed on a processor of an electronic computing device.

Yet another aspect of the invention relates to a computer-readable storage medium with a computer program product, in particular an electronic computing device with a computer program product, according to the preceding aspect.

Yet another aspect of the invention relates to an assistance system for determining a degree of degradation of an image recorded by a camera of a motor vehicle, with at least one camera and with an electronic computing device, the assistance system being designed to carry out a method according to the preceding aspect. In particular, the method is carried out using the assistance system.

Yet another aspect of the invention relates to a motor vehicle with an assistance system according to the preceding aspect. The motor vehicle is designed in particular as an at least partially autonomous, in particular as a fully autonomous, motor vehicle.

Advantageous configurations of the method are to be regarded as advantageous configurations of the computer program product, the computer-readable storage medium, the assistance system and the motor vehicle. For this purpose, the assistance system and the motor vehicle have specific features which enable the method or an advantageous embodiment thereof to be carried out.

The invention will now be explained in more detail using preferred exemplary embodiments and with reference to the accompanying drawings.

• 1 eine schematische Draufsicht auf ein Kraftfahrzeug mit einer Ausführungsform eines Assistenzsystems;
• 2 ein schematisches Blockschaltbild einer Ausführungsform eines Assistenzsystems; und
• 3 eine schematische Ansicht einer Ausführungsform eines Regressionsmoduls einer Ausführungsform einer elektronischen Recheneinrichtung einer Ausführungsform des Assistenzsystems.

show:

• 1 a schematic plan view of a motor vehicle with an embodiment of an assistance system;
• 2 a schematic block diagram of an embodiment of an assistance system; and
• 3 a schematic view of an embodiment of a regression module of an embodiment of an electronic computing device of an embodiment of the assistance system.

Elements that are the same or have the same function are provided with the same reference symbols in the figures.

1 shows a schematic plan view of an embodiment of a motor vehicle 1 with an embodiment of an assistance system 2. The assistance system 2 is for determining a degree of degradation 3 ( 2 ) of an image 5 recorded by a camera 4 of the motor vehicle 1 is formed. For this purpose, the assistance system 2 has in particular the camera 4 and an electronic computing device 6 . In particular, an area 7 of the motor vehicle 1 can be captured with the camera 4 . In the present case, the motor vehicle 1 is in particular an at least partially autonomous, in particular a fully autonomous motor vehicle 1 . In addition, the assistance system 2 can also be designed for at least partially autonomous operation or for fully autonomous operation of the motor vehicle 1 . For this purpose, the assistance system 2, for example, interventions in a steering and Accelerating device of the motor vehicle 1 make. On the basis of the detected surroundings 7, in turn, the assistance system 2 in particular can then, for example, carry out corresponding control signals for the steering or braking intervention.

2 shows a schematic block diagram of an embodiment of the assistance system 2, in particular the electronic computing device 6 of the assistance system 2.

In the method for determining the degree of degradation 3 , the image 5 is captured by the camera 4 . A deep feature extraction of a multiplicity of pixels 8 of the image 5 is carried out by means of a coding module 9 of the electronic computing device 6. The coding module 9 can also be referred to as an encoder. The multiplicity of pixels 8 are clustered using a feature point cluster module of the electronic computing device 6 . The clustered pixels 8 are regressed using a regression module 11 of the electronic computing device 6 . The degree of degradation 3 is then determined again as a function of an evaluation by applying a sigmoid function 20 ( 3 ) after the regression by means of a sigmoid function module 12 of the electronic computing device 6 as an output of the sigmoid function module 12.

2 also shows that a sigmoid loss 13 is trained for the application of the sigmoid function 20 in a first training phase for the assistance system 2 . Furthermore, the assistance system 2 or the electronic computing device 6 has a self-awareness module 14 by means of which the regressed pixels 8 are discriminated and the discriminated pixels 8 are transmitted to the sigmoid function module 12 . In particular, the self-awareness module 14 is provided in the form of global average pooling.

2 Figure 12 further shows that the coding module 9 is provided as a convolutional neural network. Furthermore, the multiplicity of pixels 8 are clustered using a K-means algorithm 15 of the feature point cluster module 10 . The clustered pixels 8 are in turn regressed using a regression module 11 embodied as a long-short-term memory module 16 . Furthermore, the clustered pixels 8 are transmitted unidirectionally, which is represented by the arrows 17, from the feature point cluster module 10 to the regression module 11. The pixels 8 are in turn transmitted bidirectionally from the regression module 11 to the self-awareness module 14, from the self-awareness module 14 to the sigmoid function module 12 and from the sigmoid function module 12 bidirectionally to the sigmoid loss 13. This is shown in particular by arrows 18 . Furthermore, the degree of degradation 3 is also forwarded bidirectionally to a decoding device 19 of the assistance system 2. In particular, the output of the sigmoid function module 12 is thus to the decoding device 19 for decoding the output. The decoding device 19 can be provided in particular in the form of a fully folded neural network. In particular, the decoding device 19 is trained for the assistance system 2 in a second training phase, with only the decoding device 19 being trained in the second training phase, which occurs after the first training phase.

3 FIG. 12 shows a schematic block diagram of an embodiment of the regression module 11 according to FIG 2 . In the present case, it is in particular a long-short-term memory module 16, which can also be referred to as a long-short-term memory module (LSTM). In the present case, the long-short-term storage module 16 has at least three sigmoid functions 20 and at least two tanh functions 21 . Furthermore, the long-short-term memory module 16 has at least two point-by-point multiplications 22 and two point-by-point additions 23 . The results of the regression are in turn propagated back to the coding device 9, which is shown in particular by the arrows 24.

Such a long-short-term storage module 16 is particularly suitable since the K-means algorithm 15 is a series of depth feature points which are separated into clusters, so that a regression problem is involved.

The core concept of the long-short-term storage module 16 is the cell state and its various gates. Cell states act like a memory of the network and can carry relevant information during the processing of the sequence. The gates are various neural networks that decide what information is allowed on the cell status. During training, the gates can learn which information is relevant in order to remember or forget it. Each gate contains sigmoidal activation. This is useful for updating or forgetting dates, since any number multiplied by 0 is 0, making values disappear or "forget". Any number multiplied by 1 is the same value, so that value stays the same or is "kept." In the long-short-term storage module 16, the forget gate is first provided. This gate decides what information to discard or keep. Information from the previous hidden state and information from the current one output are passed through the sigmoid function 20. This results in values between 0 and 1. The closer to 0, the more likely it is to be forgotten, and the closer to 1, the more likely it is to be retained.

Next is the input gate, where first the previous hidden state in the current input is passed to a sigmoid function 20 which decides which values to update by transforming the values to be between 0 and 1. It also passes the hidden state on the current input to the tanh function 21 to "crunch" values between -1 and 1 to regulate the network. Later, the outputs of both are multiplied. The sigmoid output decides what information is important to keep from the tanh output.

To calculate the cell state, the cell state is first multiplied point by point by the forgetting vector. Then the output of the input gate is used and a pointwise addition 23 is performed, updating the cell status to new values that the neural network deems relevant.

An output gate then decides what the next hidden state should be. The hidden state contains information about the previous inputs. This state is also used for predictions.

It should be noted that the connection from the encoder to the LSTM module is unidirectional via K-means and is only used for the forward pass as unsupervised K-means clustering is used. To produce a disconnected graph, the back propagation 24 is done via a separate connection from the regression module 11 to the encoder 9. It is also discouraged to use supervised K-means and make the connection bi-directional as there are few trainable parameters has, which are not sufficient in the back propagation 24, since the gradient flow takes place through a large number of trainable parameters from the LSTM.

Furthermore, temporal consistency is an important aspect of image processing-based algorithms in order to keep system performance constant. The solution proposed in the figures can be expanded to include a further coding device, so that both encoders have consecutive image frames or frames of the video sequence at their disposal. This is advantageous for the assistance system 2 to output predictions without a flickering effect.

Overall, a semi-supervised algorithm is thus proposed that estimates the degree of degradation 3 on a pixel basis from the input image. The assistance system 2 is so flexible that it does not require any hard annotation, which is annoying and very expensive. The assistance system 2 can be used in particular for at least partially autonomous driving of the motor vehicle 1 .

Claims (15)

Method for determining a degree of degradation (3) of an image (5) recorded by a camera (4) of an assistance system (2) of a motor vehicle (1) using the assistance system (2), with the steps: - Capturing the image (5) by means of the camera (4); - Carrying out a deep feature extraction of a plurality of pixels (8) of the image (5) by means of a coding module (9) of an electronic computing device (6) of the assistance system (2); - clustering the plurality of pixels (8) by means of a feature point cluster module (10) of the electronic computing device (6); - Regression of the clustered pixels (8) by means of a regression module (11) of the electronic computing device (6); and - Determining the degree of degradation (3) as a function of an evaluation by applying a sigmoid function (20) after the regression using a sigmoid function module (12) of the electronic computing device (6) as the output of the sigmoid function module (12). procedure after claim 1 , characterized in that the coding module (9) is provided as a convolutional neural network. procedure after claim 1 or 2 , characterized in that the plurality of pixels (8) is clustered by means of a K-means algorithm (15) of the feature point cluster module (10). Method according to one of the preceding claims, characterized in that the clustered pixels (8) are regressed by means of a regression module (11) designed as a long/short-term memory module (16). Method according to one of the preceding claims, characterized in that the clustered pixels (8) are transmitted unidirectionally at least from the feature point cluster module (10) to the regression module (11). Method according to one of the preceding claims, characterized in that the regressed pixels (8) are propagated back to the coding device (9). Method according to one of the preceding claims, characterized in that a sigmoid loss (13) is trained in a first training phase for the assistance system (2) in order to apply the sigmoid function (20). Method according to one of the preceding claims, characterized in that the regressed pixels (8) are discriminated by means of a self-awareness module (14) of the electronic computing device (6) and the discriminated pixels (8) are transmitted to the sigmoid function module (12). procedure after claim 8 , characterized in that the self-awareness module (14) is provided in the form of global averaging. Method according to one of the preceding claims, characterized in that the output of the sigmoid function module (12) is transmitted to a decoding device (19) of the electronic computing device (6) for decoding the output. procedure after claim 10 , characterized in that the decoding means (19) is provided in the form of a fully folded neural network. procedure after claim 10 or 11 , characterized in that the decoding device (19) is trained in a second training phase for the assistance system (2), wherein in the second training phase, which is chronologically after the first training phase, only the decoding device (19) is trained. Computer program product with program code means which, when the program code means are processed on an electronic computing device (6), this cause a method according to one of Claims 1 until 12 to perform. Computer-readable storage medium containing at least one computer program product Claim 13 . Assistance system (2) for determining a degree of degradation (3) of an image (5) recorded by a camera (4) of a motor vehicle (1), with at least one camera (4) and with an electronic computing device (6), the assistance system (2 ) for performing a method according to any one of Claims 1 until 12 is trained.

Images