DE102021201124A1 - TRAINING IMAGE CLASSIFIER NETWORKS - Google Patents

Abstract

One aspect of the present disclosure relates to a method for generating a training data set for an image classifier network, receiving an initial training data set comprising a first plurality of image samples, each image sample being labeled with a class identifier of a plurality of known classes, training a partial segmentation network on the Based on the initial training data set, dividing each image sample from the first plurality of image samples into a plurality of regions and generating a set of region features for each image sample, each region feature being based on image data in a respective one of the plurality of regions, and generating a training data set that includes a plurality of new ones includes image samples based on the first plurality of image samples. The generating step includes extracting the set of region features of a first image sample and a second image sample of the first plurality of image samples using the trained sub-segmentation network, the first image sample being labeled with a first class identifier and the second image sample being labeled with a second class identifier that is different from the first class designation. The step of generating the training data set further comprises replacing at least one region feature from the set of region features of the first sample image with the corresponding region feature from the set of region features of the second sample image to thereby generate a new sample image for the training data set.

Description

TECHNICAL AREA

The present description relates to a method for generating a training data set for an image classifier network. Another aspect relates to a method for training an image classifier network using the generated training data set and using a trained image classifier network. Associated aspects relate to a computer-implemented system configured to create and/or use an image classifier network for a device and a computer program.

BACKGROUND

There is growing interest in implementing deep learning techniques for a variety of complex image classification tasks. Many prior art methods are limited to a closed set assumption, where all possible test classes are "known" during training. This makes some of the previously known methods unsuitable for some realistic classification scenarios where it is not possible to cover all potentially occurring classes, since this may not be computationally feasible and/or due to non-existent training data sets for certain classes. Recently, open-set recognition (OSR) techniques have extended standard classification tasks to deal with unknown classes that are not in a given training data set, and are thus able to separate known from unknown samples distinguish as well as classify known. The OSR techniques can be used in many applications, e.g. B. range from the detection of failures in manufacturing systems to the detection of traffic signs or other elements in autonomous driving applications. However, in some real world areas, the difference in visual appearance across classes can be subtle. In this context, the OSR techniques should be able to distinguish between different objects at a fine-grained level, which is a difficult task.

However, some known OSR-based techniques focus on the areas where visual differences across classes are not fine-grained. Therefore, there is a need to develop new efficient fine-grained OSR techniques for image recognition.

SHORT VERSION

A first aspect of the present disclosure relates to a method for generating a training data set for an image classifier network. The method of the present disclosure includes receiving an initial training data set that includes a first plurality of image samples, wherein each image sample is labeled with a class label of a plurality of known classes. The method further comprises training a sub-segmentation network based on the initial training data set for dividing each image sample from the first plurality of image samples into a plurality of regions and generating a set of region features for each image sample, each region feature being based on image data in a respective one of the plurality of regions based. The method of the first aspect further comprises generating a training data set comprising a plurality of new image samples based on the first plurality of image samples. The generating step includes extracting the set of region features of a first image sample and a second image sample of the first plurality of image samples using the trained sub-segmentation network, the first image sample being labeled with a first class identifier and the second image sample being labeled with a second class identifier that is different from the first class designation. The step of generating the training data set further comprises replacing at least one region feature from the set of region features of the first sample image with the corresponding region feature from the set of region features of the second sample image to thereby generate a new sample image for the training data set.

In a second aspect, the present disclosure relates to a method for training an image classifier network using the training data set generated according to the first aspect or its embodiments.

A third aspect provides a computer-implemented system adapted to create and/or use an image classifier network for a device according to the techniques of the second aspect or its embodiments.

A fourth aspect provides a computer program arranged to carry out any of the steps of the techniques according to the first and/or second aspect.

The technique of the first to fourth aspects can have advantageous technical effects.

First, in some situations, the techniques of the present disclosure include generating a training data set for an image classifier network that can be implemented without complicated settings and/or parameter tuning of network-based image recognition architectures. This can lead to a reduction in training costs and an opportunity to implement the image classifier network even with limited hardware resources.

Second, complex non-linear generative techniques of some prior art techniques may require comparatively large amounts of data to train network-based architectures. In some instances, using the present techniques, such complex and computationally expensive techniques can be avoided. Therefore, compared to some prior art techniques, a size of the training dataset can be smaller and the requirements on storage devices can be less restrictive.

Third, by recombining local features across image samples of the initial training data set, the present techniques can provide efficient generation of novel image samples that differ in fine-grain detail from the initial training samples. As a result, image recognition networks trained with these newly generated image samples can detect fine-grain differences between image samples better than certain prior art techniques. Additionally and as a result, image classifier networks trained with the training data sets according to the present disclosure may perform better in solving OSR tasks (e.g., detecting defective parts or components in a manufacturing or inspection system).

The term "image sample" may refer to an image (e.g., digital image represented by an area of pixels) captured by an imaging device, e.g. a digital camera, which may also be mounted on or be part of other devices. Furthermore, as used herein, the term "image sample" includes not only an original digital image recorded by an imaging device, but also information obtained by digitally post-processing the corresponding digital image recorded by the imaging device. In some embodiments, image samples may be represented in an image space involving ordinary pixel representation, where each pixel is represented by a set of discrete quantities, e.g. B. its spatial coordinates and a color is defined. In other embodiments, image samples may be represented in a comparatively smaller feature space, such as 2D feature maps obtained by applying the convolution operation known in the art to original digital image samples (e.g., using a convolutional neural network).

The term "region" can refer to a spatially restricted part of a scene depicted in an image. For example, a region may refer to any set of pixels of an image sample (e.g., an original digital image sample or a post-processed digital image sample). In some examples, the set of pixels forms one or more contiguous groups (e.g., a single contiguous group). A region may represent a visually distinguishable part of a scene (e.g., part of an object) depicted in an image. Thus, "multiple regions" may include multiple sets of pixels, each set of pixels representing a respective (spatial) portion of an object in a recorded or processed image sample.

A “Region Feature” describes a region as discussed above, e.g. B. a spatially limited part of a scene depicted in an image and/or a set of pixels corresponding to a respective region of an image sample. For example, a region feature may be determined by processing low-level features (e.g., edges or pixel attributes) of the region. A "region feature" may include multiple elements (e.g., organized into a vector, tensor, or any other set of finite discrete elements known in the art).

Accordingly, a “set of region features” refers to a plurality of region features for respective regions. For example, a set of region features may include a first region feature for a first region of an image and a second region feature for a second region of the image, and so on.

The term "image classifier network" as used herein can refer to any type of neural network known in the context of machine learning and artificial intelligence. According to certain embodiments, the image classifier network may be a convolutional neural network (CNN) (defined e.g. by the number of filters, filter sizes, strides, etc.), which may include one or more common transformations of digital images known to those skilled in the art. based for example on convolution, non-linearity (ReLu), pooling or classification operations. In other embodiments, the image classifier network may be implemented based on other types of neural networks, such as a multilayer perceptron, a feedforward neural network, a modular neural network, or a recurrent neural network, which is a list of several non-exhaustive examples. An image classifier network can be designed to process low-level features (e.g., edges or pixel attributes) of an image to classify the image.

Similar to the term "image classifier network", the term "sub-segmentation network" can refer to any type of neural network mentioned above. In addition, the "partial segmentation mesh" can disentangle an image sample to obtain "region feature sets" representing individual object parts.

character list

• 1A Figure 12 is a flow chart of a method for generating a training data set for an image classifier network according to the first aspect.
• 1B and 1C are flow charts of further possible method steps according to the first aspect.
• 2 Figure 12 shows schematically an architecture in which a training data set of the first aspect can be generated.
• 3 is an example of partial segmentation of an image sample. Different numbers represent different parts of the object. A panel with part indices below illustrates an ordering of each object part addressed in a set of region features (Z). The sub-index 1, 2, 3, 4 and 5 can be assigned to "lower body", "back", "belly" and "head" of a "bird" and "background" object, respectively.
• 4 Figure 12 is a flow chart of a method for training an image classifier network according to the second aspect using the generated training data set according to the first aspect.
• 5 shows the performance of the method according to the present techniques. The micro-F-measure metric of OSR (Open Set Recognition) is displayed as a function of a weighting factor α that weights a cross-entropy loss contribution of the generated samples (L _produced ) with respect to that of the known samples (L _known ).

DETAILED DESCRIPTION

First, some general aspects related to the generation of a training data set for an image classifier network and the training of an image classifier network using the generated trained data set will be discussed before explaining some possible implementations. An overview of the first aspect of the present disclosure relating to generation of a training data set for an image classifier network is provided in connection with in 1A-1C flowcharts shown. Then in 2 an exemplary architecture in which a training data set of the first aspect can be generated is shown. An example for the partial segmentation of an image sample according to the first aspect is in 3 pictured. The second aspect relating to training of an image classifier network using the generated training data set according to the first aspect is then described in an in 4 shown flowchart summarized. Finally, the performance of the method in connection with 5 demonstrated.

In 1A-1C discloses and proposes a method for generating a training data set for an image classifier network according to the first aspect of the present disclosure. The method steps of the corresponding independent claim are summarized in boxes that appear in 1A-1C are drawn in solid lines, while the method steps of dependent claims are shown in boxes drawn in dashed lines. The present technique for generating the training data set includes receiving 100 an initial training data set comprising a first plurality of image samples (I _i ). With the present techniques, each image (which, for example, is a recorded th object represented) can be identified with a class identifier of a plurality of known classes (N). In some examples, image samples may be digital images of objects, e.g. B. be recorded by an imaging device. In an example, image samples may be digital images of human faces or digital images of tissues used in diagnostics, for example. In other examples, image samples may be bird, mammalian, or plant species. In still other examples, image samples may represent scanned digital images of objects, such as manufactured products (e.g., circuit boards) from a production pipeline. In other examples, image samples may be digital images of moving cars, pedestrians, or stationary objects in the context of autonomous driving. In one example, the known classes may relate to known species of birds, which may exhibit noticeable differences in visual appearance across classes (e.g., different structure and/or size of wings within different bird classes), for example. In other examples, the known classes of birds may exhibit subtle differences in visual appearance across classes (e.g., the difference between two classes of birds may be attributed to their different eye colors).

The next step of the method may include training 110 a sub-segmentation network 111 based on the initial training data set. In preferred examples, the sub-segmentation mesh may divide each image sample from the first plurality of image samples into multiple regions 112 (see labels "Part 1", "Part 2", ..., "Part "K" of schematic block 112 in 2 ). In one example, the trained sub-segmentation network can generate a set of region features (Z) for each image sample (see the architecture of 2 "primary region feature set" shown as an example). In this case, each region feature (z _k ) from the set of region features can be based on image data in a respective region (k) of the plurality of regions (K). In other words, the method of the first aspect may be able to disentangle global object features in an image into multiple region features representing individual object parts (e.g., "lower body" 1, "back" 2, "belly" 3, "Head" 4 of an object "bird" and "background" 5 are assigned, as in the upper right image of 2 and in 3 shown). In some examples, the present techniques may be based on the recently developed region grouping method, which aims to infer unraveled, spatially local features of images corresponding to individual object parts (see the article by Z. Huang and Y. Li entitled " Interpretable and Accurate Fine-grained Recognition via Region Grouping, available at https://arxiv.org/pdf/2005.10411vl.pdf). Thus, in contrast to some prior art methods, the present method may allow to work with image samples in feature space rather than image space. In some examples, formulating the problem in feature space may make the techniques of the present disclosure less necessary for capturing fine-grain details of digital images in high-dimensional image space. Tackling image sampling in feature space can thus be particularly useful when differences in visual appearance across different classes of objects and differences in their resulting digital images are subliminal, so hardware implementation of the method involving training the sub-segmentation network in ordinary image space, computationally expensive or not possible at all due to limited hardware resources. In some examples, the sub-segmentation network trained by the present techniques may be a CNN (convolutional neural network), which may include one or more common transformations on digital images known to those skilled in the art, based on, for example, convolution, non-linearity (ReLu), Pooling or classification operations are based. In some examples, training the CCN-based sub-segmentation network of the present disclosure may include producing feature maps from image samples, e.g. B. can be represented by matrices of much smaller dimensions compared to those standing for the image samples of the initial data set. For example, in one example, the size of the feature maps may vary depending on the complexity of the initial training data set (e.g., number of image samples, expected number of object parts and the resulting region features, distinguishability between different objects and the resulting image samples within the initial data set, etc.) to get voted.

In this context, an appropriate number of filters, filter size, stride, architecture, etc. for the CCN-based sub-segmentation network can be selected before the training process, and the CNN can learn the filter values alone during the training process.

The next step of the present techniques may include generating 140 a training data set comprising a plurality of new image samples based on the first plurality of image samples I _i . In some examples, the techniques of the present disclosure may take advantage of the fact that image samples from new (ie, unknown) classes may share much of their features with image samples from known classes. For example, unknown birds are still birds, can but have somewhat different attributes than birds from known classes. Generating unknown image samples from known ones can be advantageous because it does not require storing and processing a training data set with a significant number of image samples that can only differ from one another at a fine-grained level. For example, birds of unknown classes may have head and body attributes similar to those obtained by merging a head attribute from one known class with a body attribute from another known class. In addition, it may be possible to supplement the initial data set of images given by generating the plurality of new image samples, which in turn may lead to better performance of an image classifier network trained with the generated training data set. As mentioned earlier, it is useful to approach image sampling in feature space rather than in image space. In preferred examples, therefore, the plurality of new image samples can be generated in feature space to avoid the unnecessary complexity of fine-grain image construction.

In the present techniques, the step of generating 140 the training data set may include extracting 120 the set of region features of a first sample image and a second sample image of the first plurality of sample images using the trained sub-segmentation network 111 . In this case, the first image sample can be labeled with a first class identifier (e.g., "Class 1"), and the second image sample can be labeled with a second class identifier (e.g., "Class 2") that is different from the first class identifier . Referring again to the bird species example, the first class designation may refer to a "duck" while the second class may represent a "goose". In the next step, the generation step can replace 130 at least one region feature (z _k ) from the set of region features 112; Z of the first image sample with the corresponding region feature (z _k ) from the set of region features 112; Z of the second image sample. In this way, a new image sample can be generated for the training data set. Given the discussions above, it may be preferable in some examples to only deal with image samples that belong to different classes (e.g., to "Class 1" and "Class 2", to "Class 1" and "Class 3", or to one belong to any combination of the plurality of known classes (N)) because only the image samples sharing the same class could have region features that differ from each other at the marginal level. Consequently, newly generated images may be indistinguishable from those in the initial training data set, rendering their use redundant. In one example, a number of new image samples of the plurality of new image samples generated in this manner may coincide with the number of image samples of the first plurality of image samples from the initial training data set. In other examples, the number of new image samples may be less than the number of image samples from the initial data set (e.g., one half, one quarter, or one eighth that number). In still other examples, the number of new image samples may be greater than the number of image samples from the initial training data set (e.g., one and a half times, twice, or three times that number).

In some examples of the present techniques, training 110 the sub-segmentation network may further classify 150 each image sample from the first plurality of image samples of the initial training data set by a (in the embodiment 2 (referred to as "attention-based classifier" 151 ) into one of the plurality of known classes (N) based on the set of region features of the image sample generated by the sub-segmentation network 111 . In preferred examples, training 110 of the sub-segmentation network may aim to improve a classification performance of the second image classifier network.

According to the present description, the second image classifier network 151 can be trained 160 based on the initial training data set. In some examples, the training 160 of the second image classifier network 151 can aim to train 110 the sub-segmentation network 111. For example, after it has been trained , the second image classifier network 151 generate the highest prediction probability for an image sample labeled with a known class (e.g. "Class 1") that this image belongs to the known class (e.g. "Class 1"). In other words, an output prediction probability for each class of the plurality of known classes (N) generated by the second image classifier network 151 can be optimized. In this way, the sub-segmentation network 111 can also be trained 110 because the set of region features generated by the sub-segmentation network can be used as input for the second image classifier network 151 . Consequently, a splitting and/or generating performance of the sub-segmentation network 111 can also be improved. Also, in some examples, use of the trained second image classifier network can facilitate training 110 of the sub-segmentation network 111 (e.g., due to combined training of both the sub-segmentation network 111 and the second image classifier network 151). ter and accelerate. In some examples, the sub-segmentation network 111 may be directly connected to the second image classifier network 151 . In other examples, the sub-segmentation mesh 111 may be connected to the second image classifier mesh 151 via intervening elements 114 to classify the set of region features 112 generated by the sub-segmentation mesh 111; Z to transform as in the architecture of 2 illustrated and disclosed in more detail below. In some examples, the trained second image classifier network 151 of the present techniques may be implemented based on a CNN (convolutional neural network) whose parameters may be selected according to a strategy similar to that used for the CNN-based sub-segmentation network. In some examples, once the sub-segmentation network 111 and the second image classifier network 151 have been trained, their parameters can be frozen for further use of both networks according to the disclosure below. In some examples, the plurality of new image samples may be classified using the trained second image classifier network 151 .

In a further step, the replacement step 130 can include at least one region feature (z _k ) from the set of region features 112; Z of the first image sample further comprises: iteratively replacing 130 region features (z _k ) of the set of region features 112; Z of the first image sample with the corresponding region features (z _k ) of the set of region features (112; Z) of the second image sample. In some examples, the second image sample may be a single image sample selected from the first plurality of image samples in that context. In other examples, the second image sample may correspond to two or more image samples selected from the first plurality of image samples when the iterative replacement steps are performed. For example, at the nth iteration, an mth image sample may be selected from the first plurality of image samples, while at the (n+1)th iteration, an (m+k)th image sample may be selected from the first plurality of image samples . (Here m, n and k are some integers).

In the next step, a class of the first image sample can be determined after each iteration using the second image classifier network 151 . In the techniques of the present disclosure, iterative replacement as defined above may be terminated when the particular class predicted by the network of the second image classifier (151) is different from the class of the first image sample of the first plurality of image samples (i.e. different from the class of the first Image sample taken from the initial dataset before iterative replacement). In a further step, the first image sample generated before the ending step can be used as the new image sample. The new image sample generated in this way can then be included in the large number of new image samples of the training data set. In other words, with the present techniques, new image samples can be generated that represent novel recombinations of local features extracted from image samples of the initial training data set, such that these new feature combinations are not found in the training data set. At the same time, the termination step defined above can ensure that the newly generated image samples can differ from those of the known class of the first image sample (i.e. a sample selected from the initial training data set before the replacement step) at a local fine-grain level, while preserving the overall realism of the generated image samples by keeping them in the same class. In some examples, this technique may correspond to real-world scenarios where visual appearances of samples from the unknown classes may be mixtures of local appearances from known classes. At the same time, the entire disclosed generation process can be performed without requiring complex non-linear generative techniques commonly used for fine-grain scenarios.

In an example, the iterative step of replacing the region features (z _k ) from the set of region features 112; Z are executed in a sequential order. In some examples, region features in the set of region features may be arranged in a particular order (e.g., as a set of rows, columns, vectors, etc.) such that a structure of the sets of region features for image samples may be the same for all image samples can. Thus, for example, rows from the set of region features of the first image sample can be successively replaced one by one with the corresponding rows from the set of region features of the second image sample as arranged in the sets of region features. In other examples, the iterative step of replacing the region features (z _k ) may be performed in a random order. In this case, for example, the region features can be selected iteratively in a random order from the set of region features of the first image and replaced by the corresponding region features of the set of region features of the second image. In some examples, the first image may be randomly selected from the first plurality of image samples. In other In other examples, the first image may be selected sequentially from the first plurality of image samples. In either case, the present techniques may allow the collection of the new image samples created.

The techniques of the present disclosure may further include: learning a dictionary of object parts 113; D, on the basis of which, for example, the sub-segmentation mesh 111 converts any given image in feature space (e.g. a 2D feature map) into a set of region features (i.e. into "object part" segments, see labels 1 to 5 in 3 as an example) can divide (or in other words group). In some examples, the method of the first aspect may use the dictionary of object parts 113; Learn D during the training process of the sub-segmentation network 111 based on the initial training data set. In alternative examples, the method may include providing a predetermined dictionary of object parts 113; D (e.g. the predetermined dictionary of object parts may be pre-stored data stored for further use in the method of the first aspect). In a next step, the learned or predetermined dictionary of object parts 113; D can be used to allocate all region features (z _k ) from the set of region features of the image sample to a corresponding object part. In some examples, this step can be implemented based on the region grouping method mentioned above (see arXiv:2005.10411 for more details).

In some examples, the learned or predetermined dictionary of object parts may include a set of dictionary vectors (d _k ). In these examples, a number of the dictionary vectors in the set of dictionary vectors may correspond to a number of object parts (K). In some examples, the learned dictionary can be represented mathematically as D=[d ₁ , d ₂ ,...,d _k ]. In this example, each dictionary vector can be a D-dimensional vector d _k ∈ RD, and the learned or predetermined dictionary can be defined in the space D ∈ ℝ ^{D ×K} . In some examples, the set of region features of the image sample from the first plurality of image samples or from the plurality of new image samples may be represented by a set of feature vectors. In this case, each region feature from the set of region features can correspond to the respective feature vector, and a number of the feature vectors in the set of feature vectors can coincide with the dictionary vectors (ie with the number of object parts (K)). Similar to the set of dictionary vectors, the set of region features can be written as Z=[z ₁ ,z ₂ ,...,z _k ] in some examples. In this case, each feature vector can be a D -dimensional vector z _k ∈ R ^D and the set of feature vectors can be defined in the space Z ∈ ℝ ^D×K .

umfassen. In einem Beispiel kann jedes Element der Wahrscheinlichkeitszuweisungsmatrix eine Wahrscheinlichkeit zuweisen, dass das Pixel des Regionenmerkmals der Bildstichprobe an einem gegebenen Ort auf einer Bildmerkmalabbildung (zum Beispiel an einem Ort x_ij) der Bildstichprobe zu dem Objektteil k gehört, der durch den jeweiligen Wörterbuchvektor (d_k) repräsentiert wird. In einem spezifischen Beispiel kann die Zuweisungsmatrix folgendermaßen lauten: $$q_{ij}^k=\frac{\text{exp}\left(-\Vert \left(x_{ij}-d_k\right)/{\sigma }_k\Vert /2\right)}{{\displaystyle {\sum }_{l=1}^K\text{exp}\left(-\Vert \left(x_{ij}-d_k\right)/{\sigma }_k\Vert /2\right)}},$$

wobei σ_l für einen Glättungsfaktor steht.In some examples, dividing each image sample may further include calculating a distance from each pixel of the region feature of the image sample to each object part of the learned or predetermined dictionary of object parts. In some examples, the calculating step may involve calculating a distance from each pixel of the image sample's region feature (e.g., defined in feature space) to each dictionary vector (d _k ). In the present technique, dividing each image sample may further include calculating a probability that the pixel of the region feature of the image sample can be assigned to an object part (k) from the learned or predetermined dictionary of object parts. In some examples, this probability may be based on the distance of pixels of the region feature of the image sample to each object part of the learned or predetermined dictionary of object parts (e.g. to each dictionary vector (d _k )). In some examples, calculating the probability can further calculate a probability assignment matrix $\left(q_{ij}^k\right)$

include. In one example, each element of the probability assignment matrix may assign a probability that the pixel of the image sample's region feature at a given location on an image feature map (e.g., at a location x _ij ) of the image sample belongs to the object part k defined by the respective dictionary vector (i.e _k ) is represented. In a specific example, the assignment matrix can be: $$q_{ij}^k=\frac{\text{ex}\left(-\Vert \left(x_{ij}-{\mathrm{i.e}}_k\right)/{\sigma }_k\Vert /2\right)}{{\displaystyle {\sum }_{l=1}^K\text{ex}\left(-\Vert \left(x_{ij}-{\mathrm{i.e}}_k\right)/{\sigma }_k\Vert /2\right)}},$$

where σ _l stands for a smoothing factor.

von einer Differenz zwischen dem gegebenen Ort (z. B. zwischen dem Ort x_ij) des Pixels des Regionenmerkmals und dem jeweiligen Wörterbuchvektor (d_k) abhängen kann. In einigen Beispielen kann Berechnen der Menge von Merkmalvektoren 112; Z ein in der Technik bekanntes nichtlineares Merkmalcodierungsschema involvieren. Somit kann jeder berechnete Vektor aus der Menge von Merkmalvektoren (z. B. der Vektor z_k) als das Regionenmerkmal aus Pixeln interpretiert werden, die dem entsprechenden Wörterbuchvektor (z. B. dem Wörterbuchvektor d_k) zugewiesen sind. In einem spezifischen Beispiel kann der (normierte) Merkmalvektor z_k durch den folgenden Ausdruck gegeben werden: $$z_k={\tilde{z}}_k/\Vert {\tilde{z}}_k\Vert ,\text{mit }{\tilde{z}}_k=\frac{1}{{\displaystyle \sum {}_{ij}{q}_{ij}^k}}{\displaystyle \sum {}_{ij}{q}_{ij}^k}\left(x_{ij}-d_k\right)/{\sigma }_k.$$

In some examples of the present techniques, computing the set of feature vectors 112; Z Pooling of region features from an image feature map (X) of the image sample sen - the step that reduces the dimensionality of feature maps but preserves the key information contained within. In some examples, each computed vector (z _k ) after pooling region features can be represented by a non-linear function derived from the probability assignment matrix $\left(q_{ij}^k\right),$

may depend on a difference between the given location (e.g. between the location x _ij ) of the pixel of the region feature and the respective dictionary vector (d _k ). In some examples, computing the set of feature vectors 112; Z involve a non-linear feature coding scheme known in the art. Thus, each computed vector from the set of feature vectors (e.g., vector z _k ) can be interpreted as the region feature of pixels associated with the corresponding dictionary vector (e.g., dictionary vector d _k ). In a specific example, the (normalized) feature vector z _k can be given by the following expression: $${\mathrm{e.g}}_k={\widetilde{\mathrm{e.g}}}_k/\Vert {\widetilde{\mathrm{e.g}}}_k\Vert ,\text{With}{\widetilde{\mathrm{e.g}}}_k=\frac{1}{{\displaystyle \sum {}_{ij}{q}_{ij}^k}}{\displaystyle \sum {}_{ij}{q}_{ij}^k}\left(x_{ij}-{\mathrm{i.e}}_k\right)/{\sigma }_k.$$

umfassen, die aus der Wahrscheinlichkeitszuweisungsmatrix $\left(q_{ij}^k\right)$

erhalten wird, indem alle Elemente der Wahrscheinlichkeitszuweisungsmatrix $\left(q_{ij}^k\right),$

die Beiträge von den Hintergrundpixeln beschreiben, einem konstanten Wert zugewiesen werden. In einem spezifischen Beispiel kann die modifizierte Wahrscheinlichkeitszuweisungsmatrix folgendermaßen beschrieben werden: $$\widetilde{q_{ij}^k}=\{\begin{array}{l}\frac{\text{exp}\left(-\Vert \left(x_{ij}-d_k\right)/{\sigma }_k\Vert /2\right)}{{\displaystyle {\sum }_{l=1}^{K-1}\text{exp}\left(-\Vert \left(x_{ij}-d_l\right)/{\sigma }_l\Vert /2\right)}}\text{ im FALL }{M}_{ij}\equiv \mathrm{1,}\\ 1ifk\equiv K\text{ andernfalls 0 im FALL }{M}_{ij}\equiv 0.\end{array}$$

According to the present description, training 110 of the partial segmentation network 111 can further use a foreground segmentation mask (see M _i in 2 ) to assign all background pixels of the image sample to a single object part (k) from the number of object parts (K). The object part attributed to background pixels can be chosen from the interval k ∈ [1,...,K]. In this way, background pixels whose differences may not be useful in the context of the training data set with image samples representing recorded objects can be grouped and treated as a single part. This can lead to a reduction in the dimension of, for example, the probability assignment matrix, speed up calculations and potentially reduce the complexity of the sub-segmentation and second classifier network (e.g. in the architecture of 2 used) decrease. In some examples, the method of the first aspect can further calculate a modified probability assignment matrix $\left(\widetilde{q_{ij}^k}\right)$

include those from the probability assignment matrix $\left(q_{ij}^k\right)$

is obtained by dividing all elements of the probability assignment matrix $\left(q_{ij}^k\right),$

describing contributions from the background pixels are assigned a constant value. In a specific example, the modified probability assignment matrix can be described as follows: $$\widetilde{q_{ij}^k}=\{\begin{array}{l}\frac{\text{ex}\left(-\Vert \left(x_{ij}-{\mathrm{i.e}}_k\right)/{\sigma }_k\Vert /2\right)}{{\displaystyle {\sum }_{l=1}^{K-1}\text{ex}\left(-\Vert \left(x_{ij}-{\mathrm{i.e}}_l\right)/{\sigma }_l\Vert /2\right)}}\text{in the case}{M}_{ij}\equiv \mathrm{1,}\\ 1ifk\equiv K\text{otherwise 0 in CASE}{M}_{ij}\equiv 0\end{array}$$

der Bildstichprobe (I_i) zu erhalten. In einigen Beispielen kann die sekundäre Menge von Regionenmerkmalen, die durch Verarbeiten der Menge von Regionenmerkmalen 112; Z der Bildstichprobe mit dem Faltungsnetz 114 erhalten wird, semantische Merkmale auf höherer Ebene repräsentieren, die die Unterscheidungsfähigkeit der folgenden Bildklassifizierernetze (zum Beispiel des zweiten Bildklassifizierernetzes 151) verbessern können. In einigen Beispielen kann das Faltungsnetz 114 ein 1×1-Faltungsnetz sein. In einem Beispiel kann die sekundäre Menge von Regionenmerkmalen 115; $\overline{Z}$

der Bildstichprobe eine Menge von sekundären Merkmalvektoren $\left({\overline{z}}_k\right)$

umfassen. Die Menge von sekundären Merkmalvektoren (z_k) kann somit durch Verarbeiten der Menge von Merkmalvektoren (z_k) mit dem 1×1-Faltungsnetz (114) erhalten werden. Bei der in 2 dargestellten Ausführungsform können die sekundären Mengen von Regionenmerkmalen 115; $\overline{Z}$

von Bildstichproben als Eingabe für das zweite Bildklassifizierernetz (siehe „auf Aufmerksamkeit basierender Klassifizierer“ 151 in 2) verwendet werden, das Ausgangsvorhersagewahrscheinlichkeiten für die bekannten Klassen (N) von Bildstichproben aus der anfänglichen Trainingsdatenmenge erzeugt.The present techniques can further process the set of region features 112; Z of the image sample (e.g., any image from the first plurality of image samples or from the plurality of new image samples) by a convolution mesh 114 to form a secondary set of region features 115; $\overline{Z}$

of the image sample (I _i ). In some examples, the secondary set of region features obtained by processing the set of region features 112; Z of the image sample obtained with the convolution network 114 represent higher level semantic features that can improve the discrimination ability of the following image classifier networks (e.g. the second image classifier network 151). In some examples, the convolution mesh 114 may be a 1×1 convolution mesh. In one example, the secondary set of region features 115; $\overline{Z}$

the image sample a set of secondary feature vectors $\left({\overline{\mathrm{e.g}}}_k\right)$

include. The set of secondary feature vectors (z _k ) can thus be obtained by processing the set of feature vectors (z _k ) with the 1×1 convolutional network (114). At the in 2 illustrated embodiment, the secondary sets of region features 115; $\overline{Z}$

of image samples as input to the second image classifier network (see "Attention-based classifier" 151 in 2 ) can be used, which produces output prediction probabilities for the known classes (N) of image samples from the initial training data set.

der Bildstichprobe und auf einem Aufmerksamkeits-Gewichtsparameter (α) basieren. In einigen Beispielen kann der Aufmerksamkeits-Gewichtsparameter einen Beitrag verschiedener Regionenmerkmale der Menge von Regionenmerkmalen der Bildstichprobe zu einem End-Vorhersageergebnis des zweiten Bildklassifizierers 151 steuern. Bei einigen Ausführungsformen kann der Aufmerksamkeits-Gewichtsparameter durch einen Aufmerksamkeits-Gewichtsvektor (α) beschrieben werden. In diesem Fall kann der Klassifizierungsschritt auf dem Aufmerksamkeits-Gewichtsvektor (α), auf der Menge von sekundären Merkmalvektoren (z_k) und Gewichten eines zusätzlichen linearen Klassifizierers W, der eine Anzahl von Ausgängen (N) gleich der gegebenen Anzahl bekannter Klassen (N) aufweisen kann, basieren. In bevorzugten Beispielen kann der Aufmerksamkeits-Gewichtsparameter (a) auf der Basis der sekundären Menge von Regionenmerkmalen $\left({\overline{z}}_k\right)$

statt der Menge von Merkmalvektoren (z_k) berechnet werden. Im ersteren Fall können, wenn der Aufmerksamkeits-Gewichtsparameter (α) auf der Basis der sekundären Menge von Regionenmerkmalen $\left({\overline{z}}_k\right)$

berechnet wird, die erzeugten Bildstichproben direkt in das zweite Bildklassifizierernetz 151 (siehe 2) geleitetet werden. In einigen Beispielen kann der auf Aufmerksamkeits-Gewichtsvektor (α) einen Beitrag jedes Pixels des Regionenmerkmals auf einer Bildmerkmalabbildung (X) zu dem End-Vorhersageergebnis des zweiten Bildklassifizierers 151 steuern.According to the present description, classifying each image sample from the first plurality of image samples by the second image classifier on the secondary set of region features 115; $\overline{Z}$

of the image sample and on an attention weight parameter (α). In some examples, the attention weight parameter may control a contribution of different region features of the set of region features of the image sample to a final prediction result of the second image classifier 151 . In some embodiments, the attention weight parameter can be described by an attention weight vector (α). In this case the classification step can be based on the attention weight vector (α), on the set of secondary feature vectors (z _k ) and weights of an additional linear classifier W having a number of outputs (N) equal to the given number of known classes (N) may have based. In preferred examples, the attentional weight parameter (a) may be based on the secondary set of region features $\left({\overline{\mathrm{e.g}}}_k\right)$

instead of calculating the set of feature vectors (z _k ). In the former case, if the attentional weight parameter (α) based on the secondary set of region features $\left({\overline{\mathrm{e.g}}}_k\right)$

is calculated, the generated image samples directly into the second image classifier network 151 (see 2 ) to be conducted. In some examples, the attentional weight vector (α) may control a contribution of each pixel of the region feature on an image feature map (X) to the final prediction result of the second image classifier 151 .

repräsentiert werden. In einigen Beispielen kann die sekundäre Menge von Regionenmerkmalen 116; $\overline{Z}\text{'}$

der neuen Bildstichprobe durch die Menge von sekundären Merkmalvektoren $\left({\overline{z}}_k^{\text{'}}\right)$

repräsentiert werden. In diesem Sinne können neue Bildstichproben und Bildstichproben aus der anfänglichen Datenmenge durch Mengen und Vektoren mit ähnlicher Struktur repräsentiert und auf ähnliche Weise (z. B. in dem Beispiel von 2 durch das Faltungsnetz 114) verarbeitet werden.In the preferred examples, each new image sample from the plurality of new image samples for the training data set can be characterized by the secondary set of region features 116; $\overline{Z}\text{'}$

be represented. In some examples, the secondary set of region features 116; $\overline{Z}\text{'}$

of the new image sample by the set of secondary feature vectors $\left({\overline{\mathrm{e.g}}}_k^{\text{'}}\right)$

be represented. In this sense, new image samples and image samples from the initial data set can be represented by sets and vectors with similar structure and in a similar way (e.g. in the example of 2 processed by the convolution network 114).

The techniques of the present disclosure may also include receiving 170 a validation data set that includes a plurality of image samples for validation. In some examples, the second image classifier network 151 can be trained 160 based on the received validation data set. In a further step, the present techniques can include rejecting zero, one or more newly generated samples that can potentially degrade the prediction accuracy of known classes. In preferred examples, this rejection procedure can be performed based on image samples from the validation and initial data set. In some examples, rejecting may include selecting image samples 180 from the plurality of image samples for validation that are labeled with the class identifier from the given number of known classes (N) for validation. Then the following steps can be performed for each selected image sample from the plurality of image samples for validation. In some examples, by comparing each image sample from the generated plurality of new image samples to the selected image sample, a first closest image sample from the generated plurality of new image samples to the selected image sample can be determined 190. In a next step, a second closest image sample from the first plurality of Image samples of the initial training data set to the selected image sample are determined by comparing each image sample from the first plurality of image samples with the selected image sample 200. In a further step, the new image sample from the generated plurality of new image samples corresponding to the first closest image sample can be rejected 210 if the first closest image sample is closer to the selected image sample than the second closest image sample. The rejection technique disclosed above as an example can thus accept a generated candidate image sample as a new image sample under the idea that a good new sample should not come closer to validation image samples than its compared closest training image samples. Using this principle, the generated image sample can be rejected if it more closely approximates a validation image sample than its compared training image samples. In some examples, for computational efficiency, this proximity condition may not be checked for all validation image samples, but only for closest validation image samples with respect to the generated image samples (e.g., the number of closest validation image samples can be chosen between 5 and 100, or 50 and 1000, or 5000 and 10000 will). In preferred examples, the present technique may include regenerating 220 a new supplemental image sample according to the previously disclosed extracting and replacing steps if the new image sample is rejected from the generated plurality of new image samples.

jeder neuen Bildstichprobe aus der erzeugten Vielzahl neuer Bildstichproben und der entsprechenden Menge von Merkmalvektoren der ausgewählten Bildstichprobe. In diesen Beispielen kann Bestimmen der zweiten nächstliegenden Bildstichprobe ähnlich Berechnen einer zweiten nächsten charakteristischen Distanz zwischen der zweiten Menge von Merkmalvektoren $\left({\overline{z}}_k^{\text{'}}\right)$

jeder Bildstichprobe aus der ersten Vielzahl von Bildstichproben und einer entsprechenden Menge von Merkmalvektoren der ausgewählten Bildstichprobe involvieren.In some examples, determining the first closest image sample may be implemented by computing a first closest characteristic distance between the secondary set of new feature vectors $\left({\overline{\mathrm{e.g}}}_k^{\text{'}}\right)$

each new image sample from the generated plurality of new image samples and the corresponding set of feature vectors of the selected image sample. In these examples, determining the second closest image sample may be similar to computing a second closest characteristic distance between the second set of feature vectors $\left({\overline{\mathrm{e.g}}}_k^{\text{'}}\right)$

every image stitch sample from the first plurality of image samples and a corresponding set of feature vectors of the selected image sample.

A method for training an image classifier network is also disclosed and proposed. The present techniques may include generating 140 a training data set for the image classifier network according to the first aspect of the present description. Then, in some examples, an image classifier network 301 can be trained 300 on the basis of the generated training dataset. In some examples, training 300 of the image classifier network can thus include not only the common initial dataset but also the generated training dataset, thereby providing a total dataset used to train the image classifier network will, to supplement. In some examples, training the image classifier network includes treating the generated plurality of new image samples as a plurality of unknown image samples. In addition, each image sample of the plurality of unknown image samples can be classified 310 by the image classifier network 301.

In the present techniques, classification by the image classifier network 301 may include classifying a given image sample as being from the plurality of known classes (N) or as belonging to an unknown class. In preferred examples, therefore, the second image classifier network 151 may have a number of outputs (N) equal to the given number of known classes (N), while the image classifier network 301 may have a number of outputs (N+1) equal to the given number of known classes by at least one, representing a prediction of unknown classes.

In some examples, training 300 the image classifier network may train the image classifier network by using 320 the set of region features 112; Z for one or more images from the first plurality of image samples of the initial training data set. In preferred examples, training 300 of the image classifier network can train the image classifier network by using 320 the secondary set of region features 115; Z for one or more images from the first plurality of image samples of the initial training data set. In some examples, training 300 of the image classifier network may also include training the image classifier network using the received validation data set.

_erzeugt) auf der Basis der Vielzahl unbekannter Bildstichproben unter Verwendung der erzeugten Vielzahl neuer Bildstichproben berechnet werden. In einem weiteren Schritt kann durch Summieren der ersten Verlustfunktion und einer gewichteten zweiten Verlustfunktion eine Zielfunktion (

) berechnet werden. In einigen Beispielen kann die gewichtete zweite Verlustfunktion ein Produkt der zweiten Verlustfunktion und eines Gewichtungsfaktors (α) sein, so dass die Zielfunktion folgendermaßen lautet:

=

_bekannt + α ·

_erzeugt. In einem letzten Schritt kann die Zielfunktion minimiert werden, um das Bildklassifizierernetz zu trainieren.In a next step, the step of training the image classifier network may comprise computing a first loss function (L _known ) based on known image samples using the first plurality of image samples of the initial training data set. Then a second loss function (

_generated ) based on the plurality of unknown image samples using the generated plurality of new image samples. In a further step, a target function (

) be calculated. In some examples, the weighted second loss function may be a product of the second loss function and a weighting factor (α), such that the objective function is:

=

_known + α

_generated . In a final step, the objective function can be minimized to train the image classifier network.

_bekannt) trainiert wird. Je höher der Wert von α ist, desto mehr Effekt aus

_erzeugt wird berücksichtigt. Die Werte von α werden 0 bis 0,4 mit der Schrittgröße 0,1 durchlaufen gelassen. Die Experimente basieren auf der CUB-Datenmenge (verfügbar bei http://www.vision.caltech.edu/visipedia/CUB-200.html). Die gesamte Datenmenge von Bildstichproben wird in drei Aufteilungen aufgeteilt: Trainings-, Validierungs- und Testdatenmenge. 150 bekannte Klassen werden als bekannte Klassen zugewiesen. Die übrigen 50 Klassen werden als unbekannte Klassen behandelt. Die Ergebnisse von Berechnungen zeigen, dass F-Maße abhängig von dem betrachteten Generator mit zunehmenden Werten von α bis zum Optimalpunkt bei etwa 0,2-0,3 zunehmen. Es ist auch ersichtlich, dass MPGen-R durchweg besser als MPGen ist. Diese Ergebnisse zeigen die Wichtigkeit des in die MPGen-R-Techniken eingeführten Zurückweisers. Es sollte auch beachtet werden, dass die vorliegenden Techniken (mit oder ohne Zurückweiser) besser abschneiden als einige vorbekannte Verfahren, was aus den Ergebnissen von Berechnungen des F-Maßes (in 5 nicht gezeigt) folgt.For illustration is in 5 Figure 12 shows the effect of the weighting factor on the performance of the image classifier network trained on the training data set comprising the plurality of new image samples generated according to the first aspect. Shown in this figure are the results of calculations for an F-measure metric as a function of the weighting factor α for two cases of generating new image samples according to the first aspect, namely (a) without rejecting newly generated image samples (minimal partial edit generator or MPGen); (b) with rejection based on the information contained in the validation data set as explained above (MPGen with rejection or MPGen-R). The F-measure is defined on the basis that unknown classes should not be treated as a single class during evaluation and larger values of the F-measure correspond to better performance. The value α = 0 means that the mesh with only the known entropy function (

_known ) is trained. The higher the value of α, the more effect

_generated is taken into account. The values of α are stepped from 0 to 0.4 with a step size of 0.1. The experiments are based on the CUB data set (available at http://www.vision.caltech.edu/visipedia/CUB-200.html). The total data set of image samples is divided into three partitions: training, validation and test data set. 150 known classes are assigned as known classes. The remaining 50 classes are treated as unknown classes. The results of calculations show that F-dimensions increase with increasing values of α up to the optimal point at about 0.2-0.3, depending on the considered generator. It can also be seen that MPGen-R is consistently better than MPGen. These results demonstrate the importance of the rejector introduced into MPGen-R techniques. It should also be noted that the Present techniques (with or without rejectors) perform better than some previously known methods, as evidenced by the results of F-measure calculations (in 5 not shown) follows.

The techniques of the present disclosure may further include using the trained image classifier network to classify image data. In some examples, the image data may include still images or video data. Using the trained image classifier may include monitoring an environment by a monitoring device. Additionally or alternatively, the methods of the first and second aspects may comprise monitoring a device by a monitoring device. In some examples, image data of the environment may be recorded by the monitoring device (e.g., by an imaging device such as a digital camera, for example, located on the monitoring device or in communication with the monitoring device separately). In a next step, the trained image classifier network can be used to classify the recorded image data.

In some examples, an environmental condition of the monitored environment can be controlled based on the classified image data. Additionally or alternatively, the device can be controlled based on the classified image data. In some examples, a status indicator regarding an environmental condition of the monitored environment may be generated based on the classified image data. Additionally or alternatively, a status indicator regarding a status of the device may be generated based on the classified image data.

In one example, the monitoring device can monitor the surroundings of at least one semi-autonomous robot. In other examples, the monitoring device may be deployed in a manufacturing or inspection system, or in a medical imaging system. In some examples where the monitoring device is deployed in a medical imaging system, the present techniques may further include the medical imaging system outputting diagnostic results based on the classified image data.

In some examples, the semi-autonomous robot can be an industrial robot, a domestic robot, or at least a semi-autonomous vehicle. The techniques of the present disclosure may further include controlling a manufacturing process and/or quality assurance of products by the industrial robot or an operation of the autonomous vehicle based on the classified image data. In some examples, an automatic response may be triggered based on the classification step of the image classifier network. In some examples, the response may include indicating the status in a graphical user interface and/or providing information regarding the status of a remote device. Additionally or alternatively, the response may involve generating an alarm and/or changing or stopping operation of a monitored device.

In the present techniques relating to the monitoring device used in a manufacturing or inspection system, a response can be automatically triggered based on the classification step of the image classifier network. In some examples, this response may include indicating whether a component or part being manufactured and/or inspected is classified as normal or abnormal. In some examples where the monitoring device is used in a manufacturing or inspection system (e.g., a printed circuit board (PCB) manufacturing or inspection line), the techniques of the present disclosure may further scan a variety of components or parts (e.g., by an imaging device such as a camera). In a next step, a component or part of the plurality of components or parts can be classified as normal with no defect or defective as having a defect. In some examples, in response to classifying the component or part as defective, the present techniques may involve determining whether the defect is of the known class of the plurality of known classes or the unknown class. In some examples, the present techniques may be efficient when the objects/products in the entryway share a similar structure. For example, in the context of the PCB product pipeline, which may produce different designs of PCBs, all PCBs may have common components (e.g., transistors, chips, etc.). If they have some defects, the defects will usually show up as subtle (ie, fine-grained) changes in images, such as accidental mixing of wrong types of transistors or chips across PCB designs. In some examples, by using the present technique of generating a training data set for an image classifier network based on a recombination of the local features extracted from the image samples of the initial training set, it may be possible to have the image classifier network learn efficiently on such an incorrect mixture. For example it can trained image classifier network can classify multiple PCBs as belonging to the same defect class even though the same part (e.g. transistor or chip) of that PCB is erroneously located at different spatial locations. In the present techniques, it may be sufficient if this classification provides only a single "representative" image sample with such defective component location on a PCB in the initial training data set.

In some embodiments, if the component or part is classified as normal, the component or part (e.g., a PCB) may be forwarded to a next stage of the production pipeline. In some examples, the defective component or part may be forwarded to a predefined handling module if the particular defect is of the known class. In other examples, if the particular defect is of the unknown class, the defective product may be forwarded for detailed investigation.

A third aspect provides a computer-implemented system adapted to create and/or use an image classifier network for a device according to the techniques of the second aspect or its embodiments. The computer-implemented system may include at least one processor, at least one memory (which may include programs that, when executed, perform the method steps according to the first and/or second aspect) and at least one interface for inputs and outputs. In some examples, the computer-implemented system may be a standalone computing device. In other examples, the computer-implemented system may be integrated into a computing device or system that also serves purposes other than performing the steps of the techniques of the present disclosure. In still other examples, the computer-implemented system may be a distributed system that communicates over a network (e.g., the Internet).

A fourth aspect provides a computer program arranged to carry out any of the steps of the techniques according to the first and/or second aspect.

Claims (12)

A method of generating a training data set for an image classifier network, comprising: receiving (100) an initial training data set comprising a first plurality of image samples (I _i ), each image sample being labeled with a class identifier of a plurality of known classes (N); training (110) a sub-segmentation network (111) based on the initial training data set to: divide each image sample from the first plurality of image samples into a plurality of regions; and generating a set of region features (112; Z) for each image sample, each region feature (z _k ) being based on image data in a respective region (k) of the plurality of regions; Generating (140) a training data set comprising a plurality of new image samples based on the first plurality of image samples, comprising: extracting (120) the set of region features of a first image sample and a second image sample of the first plurality of image samples using the trained A partial segmentation network (111), wherein the first image sample is labeled with a first class identifier (class 1) and the second image sample is labeled with a second class identifier (class 2) different from the first class identifier; Replacing (130) at least one region feature (z _k ) from the set of region features (112; Z) of the first image sample with the corresponding region feature (z _k ) from the set of region features (112; Z) of the second image sample, thereby creating a new to generate an image sample for the training data set. procedure after claim 1 , wherein training (110) the partial segmentation network further comprises: classifying (150) each image sample from the first plurality of image samples of the initial training data set by a second image classifier network (151) into one of the plurality of known classes (N) based on the Set of region features of the image sample generated by the sub-segmentation network (111), the training (110) aiming to improve a classification performance of the second image classifier network. procedure after claim 1 or 2 , wherein replacing (130) at least one region feature (z _k ) from the set of region features (112; Z) of the first image sample further comprises the following: - iterative replacement (130) of region features (z _k ) of the set of region features (112; Z ) the first image sample with the corresponding region features (z _k ) of the set of region features (112; Z) of the second image sample; - determining a class of the first image sample after each iteration using the second image classifier network; - stopping iteratively performing the replacing step if the particular class predicted by the network of the second image classifier (151) is different from the class of the first image sample of the first plurality of image samples; and - using the first image sample generated before the terminating step as the new image sample. Procedure according to one of Claims 1 until 3 the method further comprising: learning a dictionary of object parts (113; D) during the training process of the partial segmentation network (111) based on the initial training data set or providing a predetermined dictionary of object parts (113; D); Using the learned or predetermined dictionary of object parts (113; D) to assign all region features (z _k ) from the set of region features of the image sample to a corresponding object part, optionally dividing each image sample further comprising calculating a probability that a pixel of the region feature of the image sample can be assigned to an object part (k) from the learned or predetermined dictionary of object parts, further optionally the probability being based on a distance from the pixel of the region feature of the image sample to each object part of the learned or predetermined dictionary of object parts. Procedure according to one of claims 2 until 4 , the method further comprising receiving (170) a validation data set comprising a plurality of image samples for validation and training (160) the second image classifier network on the basis of the received validation data set. procedure after claim 5 , the method further comprising rejecting zero or one or more new image samples from the generated plurality of new image samples that can potentially degrade a prediction accuracy of known classes, the rejecting comprising: selecting image samples (180) from the plurality of image samples for Validation marked with the class identifier from the given number of known classes (N); performing the following steps for each selected image sample from the plurality of image samples for validation: determining (190) a first closest image sample from the generated plurality of new image samples to the selected image sample by comparing each image sample from the generated plurality of new image samples to the selected image sample; determining (200) a second closest image sample from the first plurality of image samples of the initial training data set to the selected image sample by comparing each image sample from the first plurality of image samples to the selected image sample; rejecting (210) the new image sample from the generated plurality of new image samples that corresponds to the first closest image sample if the first closest image sample is closer to the selected image sample than the second closest image sample; the method optionally further comprising: regenerating (220) a new supplemental image sample according to the extracting and replacing steps of any one of Claims 1 until 5 , if the new image sample is rejected from the generated plurality of new image samples. A method of training an image classifier network, comprising: generating (140) a training data set for the image classifier network as in any one of Claims 1 until 6 set forth; and training (300) an image classifier network (301) based on the generated training data set. procedure after claim 7 , wherein training the image classifier network comprises: treating the generated plurality of new image samples as a plurality of unknown image samples; classifying (310) each image sample of the plurality of unknown image samples by the image classifier network (301), classifying optionally further comprising classifying a given image sample as belonging to the plurality of known classes or to an unknown class. procedure after claim 7 or 8th , wherein the second image classifier network (151) has a number of outputs (N) equal to the given number of classes (N) and wherein the image classifier network (301) has a number of outputs (N + 1) that the given number of known classes by exceeds at least one representing a prediction of unknown classes. Procedure according to one of Claims 7 until 9 , wherein training the image classifier network comprises: computing a first loss function

based on known image samples using the first plurality of image samples of the initial training data set; Calculating a second loss function

based on the plurality of unknown image samples using the generated plurality of new image samples; calculating an objective function by summing the first loss function and a weighted second loss function, the weighted second loss function being a product of the second loss function and a weighting factor (a); Minimize the objective function to train the image classifier network. Method according to any of the preceding Claims 1 until 10 , further comprising: monitoring an environment and/or a device by a monitoring device; recording image data of the surroundings by the monitoring device; Using the trained image classifier network to classify the recorded image data, where the image data may include still images or video data. A computer-implemented system arranged to implement an image classifier network for an apparatus according to any one of Claims 1 until 11 to create and/or use.

Images