DE102020211214A1 - System and method for detecting anomalies in images - Google Patents

Abstract

The invention describes a method for generating a system (6) for detecting anomalies in images, which comprises the following steps: - providing a generative network and/or an autoencoder (10) ("G/A network"), a Siamese network (8), a first training data set (T1), which includes normal images, and a second training data set (T2), which includes abnormal images,- training the G/A network (10) to generate latent data (LD) based on input images ( IP) and output images (OP) based on the latent data (LD), the training being carried out with images of the first training data set, a loss function being used for the training at least at the beginning of the training, the loss function being the similarity of the input images ( IP) and respective output images (OP), - training the Siamese network (8) to generate similarity measures (S) between input images (IP) and respective output images (OP), the Trai ning with images of the first training data set (T1) and the second training data set (T2) is carried out in that images of both training data sets (T1, T2) are used as input images (IP) for the G/A network (10) and output images (OP ) of the G/A network (10) are compared with their respective input images (IP) through the Siamese network (8). The invention further describes a related system and method for detecting anomalies in images and a related imaging system.

Description

The invention describes a system and a method for detecting anomalies in images and a method for generating such a system and a medical imaging system. The anomalies in images are preferably detected with confidence levels generated by the system.

In the technical field of machine learning, systems are sometimes used, in particular comprising a deep neural network framework, capable of learning from a set of training images and generating new images with the same characteristics as the training images. There are several embodiments of such systems, which differ in their internal arrangement, but which have in common that they have a first neural network (“input network”) that converts an input image into a set of systematic data, such as features or a code (hereinafter “latent Data”), converted, and a second neural network (“Output Network”) that generates images from the latent data. The output images generated from latent data of input images should look similar to the input images.

Such systems are, for example, “generative networks”, specific embodiments being generative adversarial networks (GAN) or generative query networks (GQN). Other examples are autoencoders, particularly variable autoencoders (VAE). Aside from being used for compressing images, they (especially their parent mesh) can also be used to generate photorealistic objects, such as faces, that are entirely fictitious.

An example of a network in the technical field of the invention is an autoencoder, i. H. disclose an artificial neural network architecture capable of learning a representation (encoding) for an image (typically for dimensionality and/or noise reduction) based on unsupervised learning of data encodings. In addition, a reconstruction side (decoding) is trained with the coding, with the autoencoder generating output images using the reduced coding (latent data), with the output images being representations that come as close as possible to the original input images. Accordingly, an autoencoder comprises an input layer, an encoding network (as an input network), a latent space having the encoding (latent data), a decoding network (as an output network), and an output layer. Special autoencoders are variational autoencoders (VAE), which are generative models.

The networks are often used for the generation of images or the recognition and classification of images.

A disadvantage of generative networks is that the training is very complicated and can lead to systematic errors.

A serious problem of recognition and classification networks is the lack of reliability, especially when using deep learning neural networks, because the credibility of the output depends heavily on the training. For example, training on noisy or spurious images could easily result in high confidence mispredictions being made. In particular, in the classification of medical images, such as in anomaly detection tasks, the problem of unbalanced data arises, making the deep learning training process difficult.

Accordingly, a major disadvantage of recognition and classification networks is that there is no measure of the reliability of the output. With regard to medical images in particular, it should be noted that precise annotation of medical images requires a great deal of effort and expense and can be subject to uncertainties in the images. In addition, medical images could be noisy, for example as a result of dose reduction. As mentioned above, because training a deep learning model on noisy or uncertain ground truth data could lead to incorrect results with a high level of confidence, this is particularly problematic for clinical decision making.

The object of the present invention is to improve the known systems, devices and methods in order to enable an improvement in image processing and in particular in the training of a recognition and classification network. Another preferred task is to generate a (confidence) score as a measure of reliability for the detection and classification of anomalies in images.

This object is solved by a method according to claim 1, a system according to claim 10, a method for detecting anomalies according to claim 12 and an imaging system according to claim 13.

This invention relates to neural networks, which are algorithms or models that need to be trained. The basic principles of machine learning are well known to those skilled in the art. Apart from an appropriate loss function for solving a spe Of the specific problem (also well known), the nature of the training data and the ground truth (labels often applied to the data) are crucial. Accordingly, if the loss function is clear, a (trained) neural network or a group of (trained) neural networks can be defined by the specific training procedure.

The system according to the invention for detecting anomalies in images is trained using a special method.

• - Bereitstellen eines generativen Netzes und/oder eines Autoencoders.

A method according to the invention for generating a system for detecting anomalies in images, in particular in medical images, comprises the following steps:

• - Provision of a generative network and/or an autoencoder.

• - Bereitstellen eines siamesischen Netzes.

Typically either a generative mesh or an autoencoder (mesh) is provided, however a combination of a generative mesh and an autoencoder could also be provided. The generative mesh and/or the autoencoder (the mesh) are hereinafter also referred to as "G/A mesh" to improve readability. As mentioned above, the basics of G/A networks are well known to those skilled in the art. The respective networks include a first neural network (“input network”) that converts an input image into a set of systematic data such as features or an encoding (hereinafter “latent data” or “latent features”), and a second neural network (“ output network") that generates images from the latent data. The output images generated from latent data of input images should look similar to the input images. Loss functions for such G/A networks are well known to those skilled in the art. In particular, the G/A network has not yet been trained. However, it could also be an already trained G/A network that is now further trained according to the method.

• - Providing a Thai network.

• - Bereitstellen eines normale Bilder umfassenden ersten Trainingsdatensatzes.

A Siamese network (sometimes referred to as a "Siamese neural network" or "twin neural network") is an artificial neural network that uses the same weights while acting on two different input vectors in parallel to compute comparable output vectors. In relation to the field of the invention, the input vectors here are input images and output images of a generative mesh or its latent data (compared to latent data generated from the output images). The output vector of this Siamese network is a measure of similarity. Accordingly, images generated by the G/A mesh are compared to their respective original images. The basic principles of Siamese network training are well known. For example, training can be accomplished with a triplet loss or contrastive loss. The Siamese network in particular has not yet been trained. However, it could also be a network that has already been trained, which is then trained further according to the method.

• - Provision of a first training data set comprising normal images.

The term "normal images" means images of objects in their normal state, where "normal" denotes a predefined state that objects should be in or a correct state. In relation to medical images, "normal" means the healthy condition. In terms of product quality, "normal" means defect-free or desired condition. The nature of images here determines what kind of problem the system will later solve. It is preferred that the images are medical images (for medical use) or product images (for quality assurance). For example, the images of the first dataset are images of a healthy organ or body area.

• - Bereitstellen eines abnormale Bilder umfassenden zweiten Trainingsdatensatzes.

The images of the first training data set are preferably labeled as normal images in order to provide ground truth. However, just knowing that images of the first data set are used for training could also be used as ground truth.

• - Providing a second training data set comprising abnormal images.

The term "abnormal images" means images of objects in a condition different from normal, i.e. H. a state in which objects are not supposed to be, or an incorrect state. In relation to medical images, "abnormal" means a pathological condition. In terms of product quality, "abnormal" means a defective or defective condition. It is clear that the abnormal images show the same type of objects as the normal images, except that the objects are now non-normal. According to the previous example, the images of the second data set are images of an unhealthy organ or body area.

• - Trainieren des G/A-Netzes, um latente Daten anhand Eingangsbildern und Ausgangsbilder anhand der latenten Daten zu erzeugen, wobei das Training mit Bildern des ersten Datensatzes ausgeführt wird, wobei eine Verlustfunktion für das Training zumindest zu Beginn des Trainings verwendet wird (später könnte das siamesische Netz möglicherweise die Verlustfunktion ersetzen), wobei die Verlustfunktion die Ähnlichkeit der Eingangsbilder und der jeweiligen Ausgangsbilder erhöht.

The images of the second training data set are preferably labeled as abnormal images in order to provide ground truth. However, the mere knowledge that images from the second data set are used for the training could also be used as ground truth.

• - training the G/A network to generate latent data from input images and output images from the latent data, where the training is performed on images of the first data set, using a loss function for training at least at the beginning of the training (later the Siamese network could possibly replace the loss function), the loss function increasing the similarity of the input images and the respective output images.

The G/A network uses its input network to generate latent data, specifically a coding or feature set, from the normal images of the first training data set. To later compare the input and output of the G/A mesh, the output mesh generates images from the latent data.

Suitable loss functions that increase the similarity of the input images and the respective output images are well known to those skilled in the art, for example reconstruction loss functions relating to the similarity of images, or perceptual loss functions relating to the latent data of images. With respect to perceptual loss, the source images must be reprocessed by the first mesh (or a mesh identical to the first mesh) to generate latent data, specifically an encoding or feature set, of the source images. Accordingly, the G/A network is trained to learn how to reconstruct examples from normal images, e.g., a majority class, which often includes the normal examples for a medical anomaly detection task. For example, the G/A network could be one of the architectures such as a variational autoencoder or a generative adversarial network.

• - Trainieren des siamesischen Netzes, um Ähnlichkeitsmaße (auch als „Ähnlichkeitsmetrik“ bezeichnet) zwischen Eingangsbildern und jeweiligen Ausgangsbildern zu erzeugen, wobei das Training mit Bildern des ersten Trainingsdatensatzes und des zweiten Trainingsdatensatzes dadurch ausgeführt wird, dass Bilder beider Datensätze als Eingangsbilder für das G/A-Netz verwendet werden und Ausgangsbilder des G/A-Netzes durch das siamesische Netz mit ihren jeweiligen Eingangsbildern verglichen werden.

It should be noted that a G/A network does not aim to produce an identical output of images, ie an output in which the pixels of the input images are processed pixel by pixel. The latent images always represent features or an encoding of the image that does not include the direct values and/or coordinates of pixels. Accordingly, by training only on normal images (e.g., images of a healthy organ), the G/A network does not “understand” abnormal states of the object. Accordingly, output images of abnormal images (e.g. images of a pathological organ) that are later (not during training) processed by the G/A network have a lower resemblance to their respective input images than when normal images are processed. The G/A network is preferably trained using a large training data set with report statements, with no explicit annotation being required.

• - Training the Siamese network to generate measures of similarity (also referred to as "similarity metrics") between input images and respective output images, the training being performed on images of the first training dataset and the second training dataset by using images of both datasets as input images for the G/ A-net are used and output images of the G/A-net are compared by the Siamese network with their respective input images.

Of course, the Siamese network needs to know whether a normal or abnormal image is input in order to learn similarity differences. As mentioned above, direct labels of the images can be used, or the training data set used is itself the label. The basic principles of generating a measure of similarity are well known, the measure of similarity typically being a value which is higher the better the similarity is and lower for a lower similarity. In short, the task of the Siamese network is to learn whether two input images are similar or different, so that a similarity metric is learned based on the reconstructed output of a G/A network. It is preferably trained to maximize the similarity measure when there is a normal sample and its reconstruction provided, and to minimize the similarity measure when an abnormal sample is measured and its reconstruction provided.

The similarity measures generated by the Siamese network can preferably be used in an active learning arrangement to guide an annotation process with recommendations, in particular for further training of the G/A network.

• - ein generatives Netz und/oder einen Autoencoder, das und/oder der durch das erfindungsgemäße Verfahren trainiert wurde,
• - ein siamesisches Netz, das durch das erfindungsgemäße Verfahren trainiert wurde. Dieses siamesische Netz wird mit dem G/A-Netz verbunden, so dass es in der Lage ist, Eingangsbilder mit ihren durch das G/A-Netz erzeugten Ausgangsbildern zu vergleichen.

A system according to the invention for detecting anomalies in images comprises the following components:

• - a generative network and/or an autoencoder which and/or which has been trained by the method according to the invention,
• - a Siamese network trained by the method according to the invention. This Siamese network is connected to the G/A network so that it is able to compare input images with their output images generated by the G/A network.

While there could be alternative arrangements for training, it is preferred that this arrangement (or any other arrangement related to the system described below relates) is also used for training the networks.

• - Bereitstellen eines Bilds als Eingabe für ein erfindungsgemäßes System,
• - Empfangen eines Ähnlichkeitsmaßes für dieses Bild (durch die Ausgabe des siamesischen Netzes),
• - falls das Ähnlichkeitsmaß jenseits einer vordefinierten Ähnlichkeitsschwelle liegt, Klassifizieren einer Abnormität im Bild, insbesondere abhängig vom Bereich der Abnormität, wobei dies vorzugsweise mit einem nachfolgend beschriebenen Klassifikationsnetz erreicht werden kann.

A method according to the invention for detecting anomalies in images comprises the following steps:

• - providing an image as input for a system according to the invention,
• - Receiving a similarity measure for this image (through the output of the Siamese network),
• - if the degree of similarity is beyond a predefined similarity threshold, classifying an abnormality in the image, in particular depending on the area of the abnormality, it being possible for this to be achieved preferably with a classification network described below.

Optionally, input images may be processed multiple times to produce multiple output images and respective similarity measures, with this multiple processing preferably being performed on each input image. Accordingly, there are two or more similarity metrics for each input image. It is thus possible and advantageous to generate a probability distribution (or a non-normalized similarity distribution) via the similarity measures.

It is clear that "beyond" a threshold means "in a range of abnormal events". In terms of a typical measure of similarity, where good similarities result in a high measure and poor similarities result in a low measure, "beyond" means below the threshold.

A control device according to the invention for controlling an imaging system comprises a system according to the invention. Alternatively or additionally, it is designed to carry out the method according to the invention. The control device can have additional units or devices for controlling components of an imaging system, for example a sequence control unit for measuring sequence control, a memory, a transmission device that generates, amplifies and emits radiation, a magnet system interface, a radiation receiving device for detecting signals and/or a reconstruction unit for reconstructing image data.

An imaging system according to the invention comprises a control device according to the invention. Accordingly, an imaging system according to the invention comprises a system according to the invention and/or is designed to carry out a method according to the invention. Preferred imaging systems are medical imaging systems, such as computed tomography (CT) systems or magnetic resonance imaging systems.

Some units or modules of the above-mentioned system or controller may be implemented in whole or in part as software modules running on a processor of a system or controller. An implementation largely in the form of software modules can have the advantage that applications already installed on an existing system can be updated with relatively little effort in order to install and run these units of the present application. The object of the invention is also achieved by a computer program product with a computer program that can be loaded directly into the memory of a device of a system or a control device of an imaging system and program units for executing the steps of the method according to the invention when the program is executed by the control device or the system is included. In addition to the computer program, such a computer program product can also include other parts such as documentation and/or additional components, including hardware components such as a hardware key (dongles, etc.) to enable access to the software.

A computer-readable medium in the form of a memory stick, a hard disk or another transportable or permanently installed carrier can be used to transport and/or store the executable parts of the computer program product so that they can be read by a processor unit of a control device or system . A processor unit may include one or more microprocessors or their equivalents.

Particularly advantageous embodiments and features of the invention are indicated by the dependent claims, as set out in the following description. Features from different claim categories can be combined as appropriate to create further embodiments not described here.

According to a preferred method, the G/A network is trained by comparing an input image with its output image generated by the G/A network. The images can be compared directly or indirectly using data generated from the images by identical processes. It is preferred that the comparison is performed using a reconstruction loss function. It is preferred to directly compare the images (comparison of the image data). Alternatively or additionally, data that is th mesh of the G/A mesh, in particular latent data, is compared with data additionally processed by the first mesh, in particular using a perceptual loss function. The first network of the G/A network is preferably an encoder network such that the data is an encoding or feature set of the image. It is preferred that the training is supported by an already trained (second) Siamese network to replace a reconstruction loss function and/or a perceptual loss function.

According to a preferred method, the Siamese network is trained to generate similarity measures directly by comparing the images (comparing the image data) between the input images and respective output images. Alternatively or additionally, the Siamese network is trained to generate similarity measures between the input images and respective output images indirectly with data generated by a first network of the G/A network, in particular latent data, which is compared with data additionally processed by the first network output images will. The first network of the G/A network is preferably an encoder network such that the data is an encoding or feature set of the image.

According to a preferred method, the G/A network is a generative adversarial network (GAN) or an autoencoder, in particular a variable autoencoder (VAE). A generative adversarial network (GAN) is a class of machine learning frameworks where two neural networks compete with each other.

According to a preferred method, the Siamese network is trained to generate a similarity threshold on the similarity measures, the similarity threshold being indicative of the processing of an abnormal image by the G/A network. It preferably learns the threshold on the similarity measure using both normal and abnormal examples from the training data sets. This is beneficial for a final binary decision ('normal' or 'abnormal'), but not required when ranking samples based on their similarity. It is clear that the threshold is chosen to separate normal images from abnormal images.

According to a preferred method, the Siamese network is designed in such a way that it normalizes similarity measures, the normalization preferably being based on a validation data set. A validation dataset is a separate training dataset. Validation dataset validation data is mainly used to tune hyperparameters of a network and monitor the behavior of loss during training. Standardized similarity measures can be defined as "confidence values". Using this confidence value (or any other comparison of the measure of similarity with predefined values), automated annotations can be made based on predictions with a high confidence value. Using the measure of similarity as a confidence value has the advantage that clinicians can be provided with a measure of the reliability of the system output.

Preferably, the system (and/or the training arrangement) comprises a classification network connected to the output of the Siamese network, preferably such that it receives a measure of similarity if it is beyond a predefined similarity threshold, in particular only in that case, so that normal Images (with a high degree of similarity) are not classified. The classification network receives output images or latent data of the G/A network or input images as input and classifies its input data. For example, normal images of a healthy coronary artery are not classified because they have a high similarity score (the G/A mesh is trained on healthy coronary arteries), while abnormal images of diseased coronary arteries have a low similarity score. Accordingly, the image of a diseased coronary artery is preferably further classified into the categories of "calcified", "non-calcified", or "mixed".

According to a preferred method, a classification network is additionally trained so that it is able to classify images using similarity measures generated by the Siamese network and/or data generated by the G/A network. The classification network is preferably trained using images of the second training data set and/or the latent data of the G/A network and/or the output images of the G/A network generated using the second training data set as the input feature set. Preferably, the classification network is trained using the latent data and/or the output images of the G/A network and/or the similarity measure of the Siamese network. The similarity measure is preferably used to filter out "normal examples". Accordingly, by training a classifier on abnormal examples, the preferred method can be extended to support multi-class prediction.

According to a preferred method, the Siamese network is trained to generate a spatially resolved similarity measure of images. This means monitoring where in the image the similarity is high and where it is low. In particular, this can be done by segmenting an image into partial images and/or segmenting an image stack (e.g. a 3D image) into image slices and/or using the coordinates of pixels of the images can be achieved. It is preferred that a classification depends on an area in an image and on the respective degree of similarity of this area. For example, if a low resemblance is found in an area where the heart normally resides, it is assumed that there is a disease of the heart.

According to a preferred method, the training is end-to-end training. Alternatively or additionally, results generated by the Siamese network are used to further train the G/A network and/or results generated by the G/A network are used to further train the Siamese network.

In a preferred system according to the invention, components of the system are part of a data network, the data network and a (especially medical) imaging system preferably being in data communication with one another, the data network preferably comprising parts of the Internet and/or a cloud-based computing system, the system according to the invention or a Number of components of this system are preferably realized in this cloud-based computing system. For example, the components of the system are part of a data network, with the data network and a medical imaging system, which provides the image data, preferably being in communication with one another. Such a networked solution could be implemented through an internet platform and/or in a cloud-based computing system.

The method can also have elements of “cloud computing”. In the technical field of "cloud computing", an IT infrastructure is provided via a data network, for example a storage area or processing power and/or application software. Communication between the user and the "cloud" is achieved through data interfaces and/or data transfer protocols.

In connection with “cloud computing”, according to a preferred embodiment of the method according to the invention, data is provided via a data channel (for example a data network) for a “cloud”. This "cloud" includes a (remote) computing system, such as a computer cluster, that typically does not include the user's local machine. In particular, this cloud can be made available by the medical facility that also provides the (medical) imaging systems. In particular, the image acquisition data is sent to a (remote) computer system (the “Cloud”) via a RIS (Radiology Information System) or a PACS (Image Archiving and Communication System).

Within the scope of a preferred embodiment of the system according to the invention, the components mentioned above are located on the "cloud" side. A preferred system also includes a local processing unit, which is connected to the system via a data channel (for example a data network designed in particular as RIS or PACS). The local processing unit has at least one data receiving interface for receiving data. Furthermore, it is preferred if the local computer also has a transmission interface for sending data to the system.

With the invention, annotation processes can be done more efficiently and less expensively through active learning, with the Siamese network selecting high-uncertainty cases from an unlabeled pool for annotation or high-uncertainty outliers from a labeled pool for further inspection.

Other objects and features of the present invention will become understood from the following detailed descriptions considered in conjunction with the accompanying drawings. However, it is to be understood that the drawings are for the purpose of illustration only and not to define the limits of the invention.

• 1 ein vereinfachtes CT-System gemäß einer Ausführungsform der Erfindung,
• 2 eine Ausführungsform für das Training eines G/A-Netzes mit einer Schätzung eines Rekonstruktionsverlusts,
• 3 eine Ausführungsform für das Training eines G/A-Netzes mit einer Schätzung eines perzeptuellen Verlusts,
• 4 eine schematische Ausführungsform eines erfindungsgemäßen Systems mit einem siamesischen Netz,
• 5 eine schematische Ausführungsform eines erfindungsgemäßen Systems mit einem siamesischen Netz und einem Klassifikationsnetz,
• 6 eine schematische Ausführungsform eines erfindungsgemäßen Systems mit einem siamesischen Netz und
• 7 ein Blockdiagramm des Prozessablaufs eines bevorzugten erfindungsgemäßen Trainingsverfahrens.

Show it:

• 1 a simplified CT system according to an embodiment of the invention,
• 2 an embodiment for training a G/A network with an estimation of a reconstruction loss,
• 3 an embodiment for training a G/A network with a perceptual loss estimation,
• 4 a schematic embodiment of a system according to the invention with a Siamese network,
• 5 a schematic embodiment of a system according to the invention with a Siamese network and a classification network,
• 6 a schematic embodiment of a system according to the invention with a Siamese network and
• 7 a block diagram of the process flow of a preferred training method according to the invention.

Like numbers refer to like objects throughout the diagrams. Objects in the diagrams are not necessarily drawn to scale.

1 shows a simplified computer tomography system 1 with a control device 5, which includes a system 6 for carrying out the method according to the invention. As usual, the computer tomography system 1 has a scanner 2 with a gantry, in which an X-ray source 3 with a detector 4 rotates around a patient, and it records raw data RD, which are later reconstructed by the control device 5 into images.

It should be noted that the exemplary embodiment according to this figure is only an example of an imaging system and that the invention can also theoretically be implemented in any imaging system used in a medical and non-medical environment. Likewise, only those components that are essential for explaining the invention are shown. In principle, such imaging systems and associated control devices are known to those skilled in the art and therefore do not need to be explained in detail.

The imaging system (here the CT system 1) records images that are used for training the system 6 according to the invention, and after training, images of the imaging system are processed by the system 6 according to the invention.

To generate a training data set (first training data set T1 and second training data set T2), a user can examine CT images and label them as normal or abnormal images (e.g., images showing disease-induced changes). The examination can be carried out on a terminal 7 which can communicate with the control device 5 . This terminal can also be used to examine results from the system 6 according to the invention.

2 12 shows an embodiment for training a G/A network 10, here an autoencoder 10, with an estimate of a reconstruction loss. The G/A network in this example is preferably a variational autoencoder (VAE) and comprises an input layer 11, an encoding network 12 as the first network 12, a latent space 13 (can also be referred to as "feature space") for storing the latent data LD, a decoding network 14 as second network 14 and an output layer 15.

Input images IP are provided to the input layer 11 and encoded by forming latent data LD (for example, a feature set) in the latent space 13 by the encoding network 12 . The latent data LD is then decoded again by a decoding network 14 and the output layer provides output images OP which should serve (but are not identical to) the input images.

By comparing the input images IP to their respective output images OP, it can be determined how well the G/A network 10 is tuned. Training can be achieved by applying a loss function that maximizes similarity.

In this example, input images IP are directly compared to their output images OP by a reconstruction loss function.

3 Figure 12 shows an embodiment for training a G/A network 10 with a perceptual loss estimate. The network is similar to that in 2 G/A network 10 shown, with the difference that the output images OP are again coded by a coding network 12. This coding network 12 should have and could have identical preferences as the coding network 12 encoding the input images.

In this example, encoded input images IP are compared to encoded output images OP by a perceptual loss function.

4 shows a schematic embodiment of a system according to the invention with a Siamese network 8. A G/A network 10 (as in 2 shown) generates output images OP based on input images IP. In contrast to 2 a Siamese network 8 compares the input images IP with their respective output images OP. In the training phase, the Siamese network 8 is trained with normal and abnormal images and is able to determine a measure of similarity S (also referred to as “similarity metric”). An example of a training procedure is in 7 shown. The output of the Siamese network 8 is the similarity measure S.

5 shows a schematic embodiment of a system 6 according to the invention with a Siamese network 8 and a classification network 9. An in 4 The arrangement shown is expanded by a classification network 9 which processes the results of the Siamese network 8 . Here normal results with a high Similar ness not processed by the classification network 9. However, when the similarity falls below a certain predefined threshold, the classification network 9 processes the results of the Siamese network 8 and derives possible classifications for the input images IP (which are then abnormal and show pathologies, for example).

6 shows a schematic embodiment of a system 6 according to the invention with a Siamese network 8. This example is similar to that in FIG 4 example shown with the difference that encoded input images IP are compared by the Siamese network 8 with encoded output images OP.

7 shows a block diagram of the process flow of a preferred training method according to the invention.

In step I an (untrained) G/A network 10 is provided.

In step II an (untrained) Siamese network 8 is provided.

In step III, a first training data set T1, which includes normal images, is provided.

In step IV, a second training data set T2, which includes abnormal images, is provided.

In step V, the G/A network 10 is trained to generate latent data LD from the input images IP and output images OP from the latent data LD (see for example 2 ), the training being carried out with images of the first training data set T1, a loss function being used for the training at least at the beginning of the training, the loss function increasing the similarity of the input images IP and the respective output images IP.

In step VI, the Siamese network 8 is trained to generate similarity measures S between input images IP and respective output images OP, the training with images of the first training data set T1 and the second training data set T2 being carried out by using images of both training data sets T1, T2 as input images for the G/A network 10 and output images OP of the G/A network 10 are compared by the Siamese network 8 with their respective input images IP.

While the present invention has been disclosed in terms of preferred embodiments and variations thereof, it should be understood that numerous additional modifications and variations could be made therein without departing from the scope of the invention. In the interest of clarity, it is to be understood that the use of "a/an" in this application does not exclude a plurality and that "comprising" does not exclude other steps or elements. Mention of a "unit" or "device" does not preclude use of more than one unit or device.

Claims (15)

Method for creating a system (6) for detecting anomalies in images, comprising the following steps: - providing a generative network and/or an autoencoder (10), - providing a Siamese network (8), - providing a first training data set (T1) which includes normal images, - providing a second training dataset (T2) that includes abnormal images, - Training the generative network and/or the autoencoder (10) to generate latent data (LD) based on input images (IP) and output images (OP) based on the latent data (LD), the training being carried out with images of the first training data set , wherein a loss function is used for the training at least at the beginning of the training, the loss function increasing the similarity of the input images (IP) and respective output images (OP), - training the Siamese network (8) to generate similarity measures (S) between input images (IP) and respective output images (OP), the training being carried out with images of the first training data set (T1) and the second training data set (T2) thereby, that images of both training datasets (T1, T2) are used as input images (IP) for the generative network and/or the autoencoder (10) and output images (OP) of the generative network and/or the autoencoder (10) by the Siamese network (8 ) are compared with their respective input images (IP). procedure after claim 1 , wherein the generative network and/or the autoencoder (10) is trained by comparing an input image (IP) with its output image (OP) generated by the generative network and/or the autoencoder (10), in particular using a reconstruction loss function and/or that data generated by a first network (12) of the generative network and/or the autoencoder (10) are compared with data from output images (OP) that were additionally processed by the first network (12). , in particular using a perceptual loss function, wherein the training is preferably supported by a second Siamese network (8), which already has been trained to replace a reconstruction loss function and/or a perceptual loss function. Method according to any one of the preceding claims, wherein the Siamese network (8) is trained to generate similarity measures (S) between the input images (IP) and respective output images (OP) directly by comparing the images and/or indirectly by comparing data , which were generated by means of a first network (12) of the generative network and/or the autoencoder (10), which is in particular an encoder network (12), with data which are generated by additional processing of the respective output images (OP) were generated with the first mesh (12). Method according to one of the preceding claims, in which the generative network and/or the autoencoder (10) is a generative adversarial network (GAN) or a variable autoencoder (VAE). A method according to any one of the preceding claims, wherein the Siamese network (8) is trained to generate a similarity threshold on the similarity measures (S), the similarity threshold indicating the processing of an abnormal image by the generative network and/or the autoencoder (10). . Method according to one of the preceding claims, in which the Siamese network (8) is designed in such a way that it normalizes similarity measures (S), the normalization preferably being based on a validation data set. Method according to one of the preceding claims, in which a classification network (9) is additionally trained so that it is able to classify images using similarity measures (S) generated by the Siamese network (8), the classification network (9) being preferred using images of the second training data set (T2) and/or the latent data (LD) and/or the output images (OP) generated by the generative network and/or the autoencoder (10) using the second training data set (T2) and preferably also of the similarity measure (S) of the Siamese network (8) is trained. Method according to one of the preceding claims, wherein the Siamese network (8) is trained to generate a spatially resolved similarity measure (S) of images, with a classification preferably depending on an area in an image and the respective similarity measure (S) of this area . Method according to any one of the preceding claims, wherein the training is end-to-end training and/or wherein results generated by the Siamese network (8) are used to further train the generative network and/or the autoencoder (10). , and/or wherein results generated by the generative network and/or the autoencoder (10) are used to further train the Siamese network (8). System for detecting anomalies in images, comprising: - a generative network and/or an autoencoder (10) which and/or has been trained by the method according to any one of the preceding claims, - a Siamese network (8) trained by the method according to any one of the preceding claims and connected to the generative network and/or the autoencoder (10) in order to encode input images (IP) with theirs through the generative network and/or the Autoencoder (10) to compare generated output images (OP). system after claim 10 comprising a classification network connected to the output of the Siamese network (8) such that it preferably receives a similarity measure (S) if it is beyond a predefined similarity threshold. Method for detecting anomalies in images with a system (6). claim 10 or 11 , which comprises the following steps: - providing an image as an input image (IP) for the system (6), - receiving a similarity measure (S) for this image with the system (6), - if the similarity measure (S) is beyond a predefined similarity threshold lies, classifying an abnormality in the image, in particular depending on the area of the abnormality, preferably with a claim 7 trained classification network, - where input images (IP) are optionally processed multiple times to generate multiple output images (OP) and respective similarity measures (S) based on an input image (IP) and in particular to generate a probability distribution over the similarity measures (S). Imaging system (1) according to a system (6). claim 10 or 11 includes and / or is designed to a method claim 12 to execute. Computer program product comprising a computer program that can be loaded directly into a system (6) or a control device (5) for an imaging system (1), which program elements for carrying out steps of the method according to any one of Claims 1 until 9 or 12 when the computer program is executed by the system (6) or the control device (5). A computer-readable medium storing program elements that can be read and executed by a computer unit to perform steps of the method according to any one of Claims 1 until 9 or 12 to be executed when the program elements are executed by the computer unit.

Images