DE102018205561A1

DE102018205561A1 - Device for classifying signals

Info

Publication number: DE102018205561A1
Application number: DE102018205561.0A
Authority: DE
Inventors: Muhammad Gondal; Jan Mathias Koehler
Original assignee: Robert Bosch GmbH
Current assignee: Robert Bosch GmbH
Priority date: 2017-08-18
Filing date: 2018-04-12
Publication date: 2019-02-21
Also published as: CN109409188A

Abstract

A tissue classification apparatus (1) comprising an artificial neural network (3) having at least one folding layer (31a-31e), said folding layer (31a-31e) containing a plurality of trainable folding cores (32a-32c) and to each convolutional core (32a-32c) provides an activation card (33a-33c), and the at least one folding layer (31a-31e) at least one classifier layer (35a-35b) is connected downstream. Furthermore, the invention relates to a device, a method (100) for training and method (200) for operating the device and an associated computer program.

Description

The present invention relates to an apparatus for classifying signals applicable to, for example, image data or audio data, a method of training, and a method of operation.
State of the art
When evaluating, for example, image data or audio data, it is always the task to recognize certain objects or other features in the data. Such a task is demanding for a computer, since the image data or audio data initially present to the computer only as numerical values without any deeper meaning and span a problem space of high dimensionality. For example, there are 10 12 different VGA resolution (640x480 pixels) images with 24 bit color depth. This large number of images should be classified based on the content of objects or features on the order of 10-100 classes.
To overcome this complexity, artificial neural networks are used. Such neural networks may, for example, consist of several successive layers in which the dimensionality of the task is significantly reduced by the use of convolution kernels and by downsampling. Such neural networks are further characterized by the fact that the data are massively processed in parallel. The GB 2 454 857 B gives an example of a method in which a microscope image is classified by means of a self-learning neural network as to which objects it contains.
Disclosure of the invention
Within the scope of the invention, an apparatus for classifying an input signal has been developed. This device comprises an artificial neural network with at least one folding layer. The at least one folding layer contains a plurality of trainable folding cores. By successive application of the respective convolution kernel at all possible positions in the input signal, an activation card results for each convolution kernel, which in each case assigns an output value to discrete positions in the input of the at least one convolution layer. This output value is a measure of the local match of the input with the respective convolution kernel.
It is quite possible that several folding layers are connected in series. For example, a first convolution layer that receives input image data may include a plurality of convolution kernels that recognize simple geometric primitives of objects, such as lines and arcs. Further folding layers can then be designed, for example, to recognize simple objects composed of these basic elements (for example a house, a traffic sign or a car). This information, in turn, can be used in further convolutional layers to detect, for example, complex situations, such as a traffic situation composed of several road users, traffic signs and buildings, in which the right-of-way position is to be determined during at least partially automated driving.
If multiple convolutional layers are present, the output of a convolutional layer need not necessarily be passed only to an adjacent convolutional layer, but may also be passed directly to non-adjacent convolutional layers.
This processing chain is not limited to image data, but can be performed just as well, for example, with audio data. Here too, for example, a first folding layer may include folding cores for detecting driving noises of other road users, horns and sirens, and in a later folding layer the traffic situation as a whole can be deduced therefrom.
The at least one convolution layer, for example the last convolution layer, is followed by at least one classifier layer, which maps its input by means of trainable weights to a set of probabilities for the input to belong to one of several classes. For example, a traffic situation as a whole can be classified as to whether the own vehicle may continue driving or is wartepflichtig. An image of tissue can be evaluated in the synopsis of all the complex features recognized by the last convolutional layer to determine if the tissue is healthy or diseased. Likewise, an image of a component can be evaluated as to whether it is damaged or not.
The output of the at least one folding layer is additionally guided into at least one aggregation layer, which is designed to assign at least one aggregation value to each activation card obtained from the at least one folding layer.
If several folding layers are connected in series, preferably the output of the last folding layer can be guided into the aggregation layer. The aggregation layer then assesses the same situation as the Classifier layer, ie, the activation cards indicate very precisely which partial areas in the input signal contributed to the probabilities determined by the classifier layer. Alternatively or in combination, however, the output of another folding layer can also be guided into the aggregation layer.
It has been found that the aggregation values contain a statement as to which subregions in the input signal were relevant for the classification of the input signal by the classifier layer. In this way, for example, the classification can be made plausible. Furthermore, for example, in the case of a classification of a component as defective or in the case of a classification of a tissue as pathologically altered, those damaged areas which have been decisive for this assessment can be identified. When assessing a driving situation to the effect that the own vehicle is waiting, the vehicle to be maintained can be identified.
The aggregation causes a significant compression of the information provided by the folding layer. However, this does not affect the accuracy of the classifier class classification because the classifier layer still receives the full information.
The significance of the aggregation values with respect to the classification made by the classifier layer is greatest when the classifier layer gets its input from the same convolution layer as the aggregation layer. However, this is not mandatory. For example, it may be desirable to analyze the relevance of portions of the input signal at a finer level of abstraction than the level of abstraction applied to the classifier layer.
In a particularly advantageous embodiment of the invention, an evaluation unit is provided, which is designed to determine a relevance map of the output of the at least one folding layer by weighted summation of the activation cards obtained from the at least one folding layer with weights stored in the aggregation layer. This relevance map is abstracted exactly on the plane last examined by the at least one convolution layer. The evaluation unit can for example be integrated in the aggregation layer or form an assembly together with the aggregation layer.
The relevance map is especially obvious when the input signal to be classified is image data. However, it is not limited to this use case. For example, a visually insightful relevance map can also be generated as an input signal for audio data by transforming the audio data into a visually intelligible representation for the respective application. Thus, for example, the signal of a horn or a siren in a visual representation of the audio signal in the frequency domain can be easily identified.
Because each convolution layer reduces the dimensionality of the input signal, the relevance map will typically be of significantly smaller dimensionality than the input signal. In the case of an image as an input signal, this means, for example, that the relevance map is of significantly lower pixel resolution than the original image. The relevance map, for example, can be scaled up accordingly, so that it can be laid out in a meaningful manner for the viewer on the input signal, so for example on the original image. The scaling can be done, for example, by bilinear interpolation.
In a further particularly advantageous embodiment of the invention, the evaluation unit receives at least one probability determined by the classifier layer as input and, depending on this probability, considers different weights stored in the aggregation layer. In this way, in the relevance map, the information obtained from the aggregation values and activation maps can be merged with the classification obtained from the classifier layer so that the parts of the input signal that are relevant for the decision are more precisely outlined.
In a further particularly advantageous embodiment of the invention, the aggregation layer is also designed as a classifier layer, which determines the probabilities for the affiliations to classes by weighting the aggregation values with trainable weights. In this way, it is possible, on the one hand, to train these weights separately from the classification by the classifier layer provided primarily so that they can subsequently be used to determine the relevance map. The recognition of relevant areas in the input signal can then benefit from the same learning data that is also used when training the classifier layer provided primarily. On the other hand, the additional function of the aggregation layer as a classifier layer is not limited to the learning phase. Rather, the classification obtained in this way by the aggregation layer can be used to make the classification obtained from the primary classifier layer plausible and to recognize possible malfunctions. This is important, for example, in safety-critical Applications, such as the at least partially automated control of vehicles in traffic.
In comparison with solutions that use the aggregation layer as the only classifier, one important difference is that the classifier layer primarily provided still relies on the full information provided by the convolution layer. If, on the other hand, the aggregation layer is the only classifier, the classification can only work with the amount of data drastically reduced by the aggregation and becomes significantly less accurate.
The aggregation layer can be designed, for example, to determine at least one aggregation value by summation and / or by averaging and / or maximum value formation and / or minimum value formation and / or variance formation and / or median formation and / or percentile formation over the values contained in the activation card. For example, "global average pooling", GAP, can be used. If the input signal is, for example, image data and the convolution kernel of the convolution layer is designed to detect a particular object, the result of the summation or averaging is, among other things, a measure of how many objects of this kind are contained in the image area as a whole. For example, if the classification that an image contains one or more objects of a particular type is based on only a single occurrence of such an object, then that statement is less robust than if many such occurrences contributed to that decision. That many detections are all flawed is less likely than that a single detection is flawed.
The classifier layer may comprise, for example, at least one fully crosslinked layer of the artificial neural network. Such a fully crosslinked layer determines, with the aid of learning weights, which features recognized by the folding layer correlate particularly well with the particular classes. Thus, for example, a right-of-way sign can be relevant for assessing whether the own vehicle may continue in a traffic situation or is in waiting, while a neon sign is irrelevant for this purpose. The weights of one or more fully networked layers can be trained in a particularly clear way with learning data.
In a further particularly advantageous embodiment of the invention, at least one display device is provided which receives at least one probability determined by the classifier layer and / or at least one relevance map as input and outputs it to a user of the device. In this way, for example, it is possible to study how preprocessing of the input signal, a change in the lighting conditions or a change in other recording conditions have an effect.
In a further particularly advantageous embodiment of the invention, at least one device for the physical detection of the input signal has at least one parameter which has an effect on the process of the detection. A feedback device is provided, which is designed to determine at least one manipulated variable for the parameter from the relevance card.
For example, if the input signal is an image signal that is physically detected by a camera, the exposure setting of the camera has an influence on which features of the examined object fall within the dynamic range of the received image signal. Via the feedback device, this exposure setting, or also any other parameter of the physical detection, can be optimized in such a way that from the outset a meaningful input signal is recorded for the respective application. For example, an image of a component can be taken with an exposure setting that saturates the largest part of the image and, for this purpose, works out the area of a suspected damaged area in a particularly high-contrast manner.
The invention also relates to a method for training the device. In this method, a plurality of learning input signals are first input to the artificial neural network. The probabilities determined for each learning input signal from the classifier layer are compared with the known affiliation of the learning input signals to the classes. At least one trainable convolution kernel in at least one convolution layer is adapted to reduce a deviation determined in the comparison.
Furthermore, a set of weights is stored in the aggregation layer and the effect of the aggregation layer with these weights is evaluated on each learning input signal with a quality measure. Subsequently, at least one weight in the aggregation layer is adjusted to improve the value of the gauge.
In particular, the quality measure may be oriented toward making the weighted summation of the activation cards from the convolutional layer with the weights of the aggregation layer into a relevance map as meaningful as possible. For example, if the aggregation layer is also called Classifier layer formed, the quality measure may include about the accuracy with which the learning input signal is correctly classified using the weights in the aggregation layer. Thus, for example, analogously to the classification with the classifier layer provided primarily, a deviation from the known affiliation of the learning input signals to the classes can be minimized.
It has been found that in this way the classification in the classifier layer on the one hand and the generation of a relevance map with the aggregation layer on the other hand can be trained with the same learning data, but independently of each other.
In particular, different sets of weights may be trained in the aggregation layer for learning input signals belonging to different classes.
The invention also relates to a method for operating the device. In this method, an input signal is input to the artificial neural network, which may have previously been trained in particular by the method provided by the invention. The class to which the classifier unit determines the greatest likelihood is considered the result class to which the input signal belongs.
It is now possible to retrieve weights from the aggregation layer based on the result class, or alternatively fixed weights can be used. The relevance map is determined from the weights and the activation maps of the folding layer.
The weights retrieved from the aggregation layer already contain the information about the result class provided by the classifier layer. This is an important difference to methods in which the aggregation layer is used as the only classifier. If the aggregation layer is the only classifier, the classification is less accurate.
The neural network can be implemented in any desired manner and in any desired mixing ratio between hardware and software. With specially designed hardware for the operation of artificial neural networks or particularly suitable hardware, such as graphics processors (GPUs), the processing speed can be increased. However, the neural network can also be implemented as a pure software solution on a conventional CPU. Furthermore, in particular, the training of the device or its control during operation can be completely or partially implemented in software. Such software is a standalone salable product that may also be distributed as an add-on or update to existing devices. Therefore, the invention also relates to a computer program having machine readable instructions which, when executed on a computer, upgrade the computer to a device provided by the invention and / or cause it to perform a method provided by the invention. Likewise, the invention also relates to a machine-readable data carrier or a download product with the computer program.
Further measures improving the invention will be described in more detail below together with the description of the preferred embodiments of the invention with reference to figures.
list of figures

1 Ausführungsbeispiel der Vorrichtung 1;
2 Ausführungsbeispiel des Verfahrens 100 zum Trainieren;
3 Ausführungsbeispiel des Verfahrens 200 zum Betreiben;
4 Wirkung der Vorrichtung 1 bei der Analyse medizinischer Bilder.

1 Embodiment of the device 1 ;
2 Embodiment of the method 100 to exercise;
3 Embodiment of the method 200 to operate;
4 Effect of the device 1 in the analysis of medical images.

1 shows an embodiment of the device 1 , A camera 6 takes pictures of a component 8th with a crack 8a as input signal 2 on, with the exposure setting via the parameter 61 is controlled. Of course, other settings, such as the viewing angle, the use of filters (such as optical filters, infrared or UV cut filter) or the type of recording (such as color, red-free, autofluorescence or blue) on the parameter 61 to be controlled.
The input signal 2 becomes the artificial neural network 3 fed. The artificial neural network 3 returns a first probability as the final result 36a for being the input signal 2 a proper component 8th shows, as well as a probability 36b for being the input signal 2 a defective component 8th shows.
However, the operation of the invention is not limited to calculating exactly two probabilities, that is to say belonging to exactly two classes is examined. For example, any intermediate stages between the states "completely correct" and "completely defective" can define other classes.
Internally, the neural network exists 3 from five folding layers 31a - 31e , two optional pooling layers 34a and 34b for reducing the dimensionality and classifier layers 35a and 35b , Of those in the folding layers 31a - 31e each contained folding cores are in 1 for clarity, only three in the last convolutional layer 31e marked and with the reference numerals 32a - 32c designated. The folding layer 31e detected to every convolution kernel 32a - 32c the associated activation card 33a - 33c , The classifier layers 35a and 35b are fully connected layers in the neural network 3 ,
It is purely coincidental that in the in 1 As shown in the example just as many (two) classifier layers 35a and 35b are available as probabilities 36a and 36b be calculated. It can also be, for example, a single classifier layer 35a or 35b about belonging to much more, for example, hundreds of classes. For example, in the automatic recognition of objects such as traffic signs, each object to be recognized defines its own class.
The aggregator layer 37 determined to each activation card 33a - 33c an aggregation value 37a - 37c and settle the aggregation values 37a - 37c additionally with trainable weights 38a - 38c to a probability 39a for being the input signal 2 a proper component 8th shows, as well as to a probability 39b for being the input signal 2 a defective component 8th shows. The determination of the probabilities 39a and 39b For example, for the training of weights 38a - 38c be used, but also for the plausibility of the classifier layers 35a and 35b determined probabilities 36a and 36b ,
The weights 38a - 38c be from the evaluation unit 4 in combination with the activation cards 33a - 33c used the relevance map 41 to investigate. The probabilities decide 36a and 36b about what set of weights 38a - 38c is used. The relevance card 41 indicates to what extent ranges of the input signal 2 for the decision of the neural network 3 in terms of probabilities 36a and 36b have been relevant.
The probabilities 36a and 36b be together with the relevance card 41 via the output device 5 output. In addition, the relevance card 41 a feedback facility 7 fed, which is a manipulated variable 42 for the exposure parameter 61 the camera 6 determined. This will ensure that the camera 6 always operated in an optimal working area to possible cracks 8a in the examined component 8th to recognize.
2 shows an embodiment of the method 100 to train in 1 shown device 1 , Training becomes learning input signals 2 used, their affiliations 21 to the classes to which the probabilities 36a . 36b . 39a and 39b relate to, are known in advance.
In step 110 be the learning input signals in the neural network 3 entered, including folding layers 31a-31e contains. As a result, the neural network provides the probabilities 36a and 36b for belonging to classes.
In step 120 become the probabilities 36a and 36b with the familiar affiliation 21 the learning input signals 2 compared to the classes. The comparison provides a deviation D.
In step 130 will be at least one trainable convolution kernel 32a-32c in at least one folding layer 31a-31e adapted such that after repeating step 120 the deviation D is reduced.
Parallel the activation cards 33a - 33c from the folding layer 31a-31e the aggregation layer 37 fed and there to aggregation values 37a-37c processed. In conjunction with trainable weights 38a-38c are from the aggregation layer 37 probabilities 39a and 39b for the affiliation of the learning input signals 2 determined to classes. The probabilities 39a and 39b So put in this embodiment, the effect W of the weights 38a-38c represents.
In step 140 this effect W is associated with the known affiliation 21 the learning input signals 2 evaluated to classes and issued a corresponding quality measure M. The better the true affiliation 21 consistent with the probabilities 39a and 39b stands, the better is the quality measure M.
In step 150 become the weights in the aggregation layer 37 adapted so that in a repeat of step 140 the quality measure M is improved.
The weights gained in this way 38a - 38c can according to 1 for creating relevance maps 41 be used.
3 shows an embodiment of the method 200 for operating the device 1 , In step 210 becomes an input signal 2 into the artificial neural network 3 entered. the one Class to which the classifier layer 35a - 35b , and thus the neural network 3 as a whole, the greatest probability 36a respectively. 36b determined in step 220 as the result class 22 evaluated, to which the input signal 2 belongs.
In step 230 are determined by the result class 22 weights 38a-38c from the aggregation layer 37 accessed. These weights will be in step 240 along with the activation cards 33a-33c the folding layer 31a-31e of the neural network 3 to the relevance map 41 processed.
4 shows an application example of the device 1 in health care. Partial images a and c each show camera images 2 a retina. The task is to classify the images as to whether a pathological change (here: diabetic retinopathy) is present.
Field b is an overlay 2 + 41 of the camera image 2 from partial image a with the relevance map derived from it 41 , Similarly, field d is an overlay 2 + 41 of the camera image 2 from part c with the relevance map obtained from it 41 ,
Both partial image a and partial image c each show a pathologically altered retina, which was determined by a physician. This was also done by the neural network 3 so recognized. In the partial images b and d, the physician has in each case those areas on which he has based his diagnosis and of which two each have the reference number 9 are marked marked. The comparison with the relevance card 41 shows that in many of these areas 9 also areas 41a lie according to the relevance map 41 are particularly important.
QUOTES INCLUDE IN THE DESCRIPTION
This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.
Cited patent literature

GB 2454857 B [0003]

Claims

Device (1) for classifying a tissue, in particular a retina, comprising an artificial neural network (3) with at least one folding layer (31a-31e), wherein the convolution layer (31a-31e) contains a plurality of trainable convolution cores (32a-32c) and provides to each convolution kernel (32a-32c) an activation map (33a-33c) representing discrete positions in the input of the at least one convolution layer (31a-32c). 31e) each assigns an output value which is a measure of the local match of the input with the respective convolution kernel (32a-32c), wherein the at least one convolution layer (31a-31e) is followed by at least one classifier layer (35a-35b) which maps its input by trainable weights to a set of probabilities (36a-36b) for making that input one of a plurality of predetermined classes, in particular whether the tissue is healthy or abnormally altered, is associated, wherein an evaluation unit (4) is provided which is designed by weighted summation of the activation cards (33a-33c) obtained from the at least one folding layer (31a-31e) with weights (38a-38c) deposited in an aggregation layer (37) Relevance map (41) of the output (33a-33c) to determine the at least one folding layer (31a-31e), wherein the relevance map (41) shows those parts of the tissue that have been relevant for the assessment.
Device (1) according to Claim 1 wherein the evaluation unit (4) receives as input at least one probability (36a-36b) determined by the classifier layer (35a-35b) and, depending on this probability (36a-36b), different weights (38a-38c) stored in the aggregation layer (37) ) considered.
Device (1) according to one of Claims 1 or 2 in that the activation card (33a-33c) of the at least one folding layer (31a-31e) is additionally guided by the aggregation layer (37), which is further adapted to receive each activation card (33a) obtained from the at least one folding layer (31a-31e). 33c) assigns at least one aggregation value (37a-37c), the aggregation layer (37) also being designed as a classifier layer containing the probabilities (39a-39b) for the classifications by weighting the aggregation values (37a-37c) with trainable weights (38a-38c).
Device (1) according to Claim 3 wherein the aggregation layer (37) is adapted to at least one aggregation value (37a-37c) by summation and / or by averaging and / or maximum value formation and / or minimum value formation and / or variance formation and / or median formation and / or percentile education on in the Activation card (33a-33c) to determine the values contained.
Device (1) according to one of the preceding claims, wherein the classifier layer (35a-35b) comprises at least one fully crosslinked layer of the artificial neural network (3).
Device (1) according to one of the preceding claims, wherein at least one display device (5) is provided, the at least one of the classifier layer (35a-35b) determined probability (36a-36b), and / or at least one relevance card (41), as Input receives and outputs to a user of the device (1).
Device (1) according to one of the preceding claims, wherein at least one device (6) for the physical detection of the input signal (2) has at least one parameter (61) which has an effect on the process of detection, and in that a feedback device (7 ) is provided, which is adapted to determine from the relevance card (41) at least one manipulated variable (42) for the parameter (61).
Method (100) for training a device (1) according to one of the preceding claims with the steps: A plurality of learning input signals (2) are input to the artificial neural network (3) (110); The probabilities (36a-36b) determined for each learning input signal (2) from the classifier layer (35a-35b) are compared (120) with the known affiliation (21) of the learning input signals (2); • at least one trainable convolution kernel (32a-32c) in at least one convolution layer (31a-31e) is adapted (130) to reduce a deviation (D) determined in the comparison (120); In the aggregation layer (37) a set of weights (38a-38c) is deposited, and the effect (W) of the aggregation layer (37) with these weights (38a-38c) on each learning input signal (2) is given a quality measure ( M) (140); • at least one weight (38a-38c) in the aggregation layer (37) is adapted (150) to improve the value of the gauge.
Method (200) for operating a device (1) according to one of the preceding claims with the steps: • an input signal (2) is input to the artificial neural network (3) (210); The class to which the classifier layer (35a-35b) has the highest probability (36a-36b) is determined to be the result class (22) to which the input signal (2) belongs; Using the result class (22), weights (38a-38c) are retrieved from the aggregation layer (37) (230), or fixed weights (38a-38c) are used; From the weights (38a-38c) and the activation cards (33a-33c) of the convolution layer (31a-31e), the relevance map (41) is determined (240).
A computer program containing machine readable instructions which, when executed on a computer, move the computer to a device (1) according to any one of Claims 1 to 7 upgrade, and / or cause, a method (100, 200) according to one of Claims 8 or 9 perform.