JP2024022546A - Training of neural network using knowledge graph - Google Patents

Abstract

PROBLEM TO BE SOLVED: To train a neural network having a feature extractor for generating feature maps.

SOLUTION: A method includes: preparing labeled training examples 2a; preparing a generic knowledge graph 4 in which nodes represent entities and edges represent a relation between the entities; selecting, from the knowledge graph, a subgraph 43 related to a context for solving a specified task; calculating, for each training example, a feature map 2b using a feature extractor 11; calculating, from each training example related to each target output 3a, a representation of the subgraph in a space of the feature map; evaluating 160, 161 an output related to a task set from the feature map; evaluating, using a set cost function 5, to what extent the feature map is similar to the representation of the subgraph; optimizing parameters of a neural network 1; and adjusting the evaluation of the feature map so as to correspond as well as possible to the target output for each training example.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2024,JPO&INPIT

Description

The present invention relates to the training of neural networks that can be used, for example, to classify images or other measurement data related to the presence of particular types of objects.

BACKGROUND OF THE INVENTION Neural networks, for example, for classifying images or other measurement data related to the presence of a particular object, are typically trained by monitoring a large number of training examples labeled with a target output. After completion of training, it is expected that the neural network will send out correct outputs that are relevant to the specific task set, even for images or measurement data that were not observed during training.

This is typically guaranteed at least with respect to the extent to which the measurement data input to the neural network in a later mode of operation belong to a similar distribution or the same domain as the training example. S. Monka et al., “Learning Visual Models using a Knowledge Graph as a Trainer”, arXiv:2102.08747v2 (2021), states that at least one feature extractor of a neural network is also trained by a comprehensive knowledge representation from a knowledge graph. Discloses a training method. This allows training to be better generated on measurement data that no longer exists in complete form within the distribution or domain of the training examples.

S. Monka et al. “Learning Visual Models using a Knowledge Graph as a Trainer”, arXiv:2102.08747v2 (2021)

Disclosure of the Invention The present invention provides a method for training a neural network for evaluating measurement data. The neural network has a feature extractor that generates a feature map from the measurement data. In this way, the measurement data input to the neural network first passes through the feature extractor and is processed into a feature map, and then in the next step, from the feature map, information related to the set task is extracted. The output is calculated.

The feature extractor may include, for example, a stack of convolutional layers. In each of these convolutional layers, at least one feature map whose dimensionality is reduced compared to the input of the convolutional layer is obtained from the input of the convolutional layer by applying one or more filter cores in a fixed pattern. generated.

In the framework of the method, training examples are provided labeled by target outputs related to the set task. Additionally, a comprehensive knowledge graph is provided where nodes represent entities and edges represent relationships between entities. In this way, in a directed knowledge graph, for example,
- prove that the object or other entity represented by the node has a specific property by adding another node with a specific property and making a reference from the original node with the edge "has the property of" ,
- Indicate that the object or other entity represented by the node has a subclass by appending another node with a specific subclass and making a reference to the original node with the edge "is a subclass of" and/or
・By adding another node with a concrete superclass and making a reference from the original node with the edge "is a subclass of...", the object or other entity represented by the node belongs to the superclass. can be encoded.

For example, traffic signs can have shapes, colors and symbols, and are valid in a particular country. The class "traffic signs" may include subclasses for, for example, instruction signs, danger signs, regulation signs, prohibition signs and priority signs. Conversely, the class "traffic sign" may be a subclass of "traffic-related object."

From the comprehensive knowledge graph, subgraphs are selected that are relevant to the context for solving the set task. In this case, a context can be provided. However, as will be explained later, the selection of the subgraph can also be performed fully or partially automatically in the framework of an optimization.

Here, for each training example, a feature map is computed using a neural network feature extractor. Furthermore, from each training example, a representation of a subgraph in the space of feature maps is computed in relation to a respective target output. Here, "in the space of feature maps" specifically means that comparison results and/or similarity or distance measures between one representation and another feature map are being described.

The representation can, in particular, be computed using, for example, any "knowledge graph embedding" method, which moves the subgraph into a representation in which differences between entities can be identified in the form of distances. can. For example, a "Graph Neural Network" or GNN, which takes the subgraph as input, can be used for this purpose. Such a GNN may in particular be able to compute a representation for each node of a subgraph, for example.

From the feature map, outputs related to the set task are evaluated. This can be done, for example, by a task head that is also trainable. However, as will be described further below, this can also be replaced by modeling the evaluation as a Gaussian process specifically for the classification task of the task head.

Here, the set cost function is used to evaluate how similar the feature map is to the representation of the lower graph.

The parameters characterizing the behavior of the neural network are optimized with the aim of predictively improving the evaluation by the cost function during further processing of the training examples. The parameters may in particular include weights for weighting the inputs supplied to neurons or other processing units of a neural network, for example. Additionally, the evaluation of the feature maps is adjusted such that the output for each training example corresponds as closely as possible to the target output for the respective training example.

The variation of the parameters for each subsequent learning step can be calculated, in particular, for example in a gradient descent method, in which the evaluation by the cost function is back-propagated through the neural network and , are converted into gradients for varying the respective parameters.

It is recognized that solving a specific task often involves only a small portion of the knowledge stored in the knowledge graph as a whole. The effects of these small parts may be completely or partially obscured or neutralized by a great deal of other knowledge that is less important to the specific task. This tendency is suppressed by pre-selection of subgraphs.

Furthermore, by focusing on a specific context by selecting a subgraph, the occurrence of ambiguity is suppressed. Such ambiguity can be concretely shown by so-called "ambiguous figures" based on human perception. A "polysemous shape" is subjectively perceived by one of multiple possible semantic meanings, depending on the individual's context of association. For example, people who recall an object when they see an image of a Rubin vase will recognize the vase in the image. In contrast, a person who associates an image with a person will recognize two faces facing each other in the image. Here, a very extensive knowledge graph is transformed into a representation in the space of feature maps, where multiple possible semantic meanings overlap, i.e., "pot" and "face". This may undermine or even offset the intended effect of using the knowledge graph. According to the context selection proposed here, exactly one semantic meaning is selected even in such cases.

In a particularly advantageous embodiment, the neural network additionally comprises a task head for evaluating the feature map with respect to a set task. The task head can be trained as well. In this case, adjusting the evaluation of the feature map requires
- evaluating how well the output of the task head for each training example corresponds to the target output for each training example using a cost function;
・The parameters of the task head are also optimized with respect to evaluation by the cost function,
is included.

For example, a cost function with two contributions can be used. The first contribution is related to the similarity between the feature map and the representation of the subgraph, and the second contribution is related to the match between the output and the target output. During parameter optimization, for example gradient descent optimization, the first contribution acts only on the parameters of the feature extractor. This is because the similarity does not depend on the behavior of the task head. In contrast, the second contribution can very well influence both the task head parameters and the feature extractor parameters, since the feature map forms the basis of the task head's work.

However, while the task head can be trained with a separate, unique cost function, e.g. with respect to the similarity between the feature map and the representation of the subgraph even after optimization, characterizing the behavior of the feature extractor Parameters are frozen.

The task head may in particular include a fully connected layer that accesses, for example, the entirety of the last computed feature map and compresses this feature map for the desired output.

In a particularly advantageous configuration, the similarity of the feature map to the representation of the subgraph is compared to the similarity of the representation of the subgraph to feature maps calculated for other training examples using other target outputs. In other words, the evaluation using the cost function is
- The more similar the feature map calculated from each training example is to the representation of the subgraph calculated based on that training example ("positive example"), and
The less similar the representation of the subgraph is to feature maps computed for other training examples using other target outputs (“negative examples”),
It will be better.

The contribution to the cost function (also referred to as a loss function) associated with the similarity to the representation of the subgraph is thereby a "contrast loss" equivalent to the signal-to-noise ratio in messaging technology in the broadest sense. Similarity or dissimilarity can be measured by any suitable similarity measure, such as cosine similarity.

The target output may in particular include, for example, classification scores associated with one or more classes of the established classification of the measurement data. That is, a neural network can be configured as a classifier for measurement data. In this case, the class may in particular represent, for example, the type of object that the measurement data showed to be present in the area monitored at the time of recording the measurement data. It is precisely in such applications that ambiguity resolution is particularly advantageous.

In this case, in particular, for example vehicles, traffic signs, lane markings, traffic obstacles and/or other traffic-related objects in the vicinity of the vehicle can be selected as the object type. It is precisely in these types of objects that generalization to distributions or domains different from those of the training examples, as contemplated by the use of knowledge graphs, is particularly important. Here, for example, traffic signs with similar semantic meanings may have a somewhat different appearance in other countries. For example, danger signs in many European countries consist of a red triangle with a white side on which a symbol is placed. Other countries have different shades of red for the triangle or use slightly different symbols. Additionally, in Finland and Greece, a yellow side is used instead of a white side inside the red triangle.

Similarly, in the case of traffic signs and other traffic-related objects, the unambiguousness of the identification, which is improved by focusing on the specific context, is important.

In another advantageous arrangement, evaluating the feature map includes assigning the feature map to classes using a Gaussian process. In this case, the steps to adjust the rating include
・Calculating feature maps for all training examples;
- determining decision limits between classes in the space of feature maps based on their respective target outputs;
is included.

In particular, it is possible to calculate the mean value matrix and covariance matrix for each individual class from the totality of all feature maps. From these, decision limits between classes can be determined.

When the trained neural network is supplied with measurement data during subsequent processing operations, this measurement data is first processed into a feature map by a feature extractor. Here, with respect to the feature map, a probability that the feature map belongs to the class is assigned to each class by a Gaussian process. In this case, the class with the highest probability can be evaluated as the final decision of the classifier.

The advantage of using a Gaussian process instead of a task head to assign a feature map to one class is that interactions between the training of one feature extractor and the training of the other task head are avoided. can get. Such interactions can interfere with training a feature extractor with the subgraph. This is because task head training cannot benefit from the use of subgraphs.

In particular, images, audio signals, time series of measurement values, radar data and/or LiDAR data can be selected as measurement data, for example. These data formats are very diverse in the sense that they may contain descriptions related to many aspects. By selecting a subgraph, it is possible to select aspects to be inspected. The subgraph may relate, in particular, to the visual, taxonomic or functional context of the measurement data, for example.

Visual context describes abstract visual properties of objects, such as color, shape or texture. These characteristics may or may not be present within the configured set of training examples. Utilizing subgraphs can advantageously extend them in training examples in this regard. For example, if the training example dataset contains only images in which a conditionally white horse is present, the subgraph conveys additional information to the feature extractor that the horse may have other colors. be able to.

Taxonomic context describes relationships between classes based on a hierarchy. The taxonomy makes it possible, in particular, to take account of external prior knowledge, for example. For this reason, classifications of traffic signs exist, for example, which allow traffic signs to display information, warnings, regulations, rules or prohibitions. The classification method therefore incorporates an important part of the semantic meaning of traffic signs, which is required for example for automatic vehicle guidance.

Functional context includes properties that describe the function of an object. Here, for example, tools can be categorized according to whether they can perform screwing, sawing or drilling.

In another particularly advantageous configuration, the selection of the subgraph is also introduced into the optimizing step in conjunction with the evaluation by the cost function. That is, for example, the subgraph that provides the best match between the representation of the subgraph and the feature map generated from the training example can be calculated. For example, multiple candidate subgraphs can be created, and neural network training can be performed on each candidate subgraph. Then, for example, a final selection of candidate subgraphs can be made that allows obtaining the best value of the cost function. The creation of candidate subgraphs may include, among other things, for example, systematically examining a search space of subgraphs and/or calculating candidate subgraphs based on heuristics. In particular, the search space can be limited by taking into account prior knowledge about the content of the knowledge graph and/or such prior knowledge can be included in the heuristics.

In another advantageous arrangement, the representation of the subgraph in the space of feature maps is called from a pre-computed look-up table based on the training examples and the target output. Therefore, the cost of calculating the representation of the subgraph can be front-loaded.

In another advantageous configuration, another machine learning model is trained with the neural network to generate a representation of the subgraph in the space of feature maps. For example, a graph neural network GNN that already receives as input a subgraph or a global knowledge graph can be used for this purpose. In this case, the feature extractor on the one hand and the subgraph representation generator on the other hand can contribute to the same extent to maximize the similarity between the feature map and the subgraph representation.

In another advantageous configuration, the trained neural network is fed with measurement data. A drive control signal is formed from the output of the neural network. A vehicle, a driving support system, a quality control system, an area monitoring system, and/or a medical image forming system are drive-controlled by the drive control signal. In this case, based on the improved ability of the neural network to generalize beyond the training examples used during training, the response of each controlled system is adapted to the situation detected by the measured data. The probability of it happening increases.

The method is particularly fully or partially computer-implementable. Accordingly, the present invention provides a computer comprising machine readable instructions for performing the method described above when executed on one or more computers and/or computer instances. It also concerns the program. In this sense, embedded systems for vehicle control devices and technical devices that can likewise execute machine-readable instructions can also be considered as computers. Examples of computer instances are virtual machines, containers, or serverless execution environments that execute machine-readable instructions in the cloud.

The invention likewise relates to a machine-readable data carrier and/or download product containing a computer program. A download product is a digital product that can be transmitted over a data network, i.e. a digital product that can be downloaded by a user of the data network, for example available for immediate download in an online shop.

Furthermore, the computer may be equipped with a computer program, a machine readable data carrier or a downloadable product.

Further measures for improving the invention are explained in detail below, together with a description of a preferred embodiment of the invention with reference to the drawings.

1 illustrates an example of a method 100 for training a neural network 1; FIG. FIG. 4 illustrates that various contexts for subgraphs 43 of comprehensive knowledge graph 4 are considered.

Example FIG. 1 is a schematic flowchart of an example of a method for training a neural network 1. The neural network 1 has a feature extractor 11 that generates a feature map 2b from the measurement data 2.

In step 110, training example 2a is provided. The training example 2a is labeled with a target output 3a related to the set task.

In step 120, a comprehensive knowledge graph 4 is prepared in which nodes 41 represent entities and edges 42 represent relationships between entities. For example, an object as the first entity 41 can be linked to a property as the second entity 41 via the relationship "has/is" as the edge 42.

In step 130, a subgraph 43 is selected from the comprehensive knowledge graph 4 that is relevant to the context for solving the set task.

In step 140, a feature map 2b is calculated for each training example 2a by the feature extractor 11 of the neural network.

In step 150, a representation 43a of a subgraph 43 in the space of feature maps 2b is calculated from each training example 2a associated with a respective target output 3a.

According to block 151, a representation 43a of the subgraph 43 in the space of the feature map 2b can be called from a precomputed lookup table based on the training example 2a and the target output 3a.

According to block 152, another machine learning model may be trained with the neural network 1 to generate a representation 43a of the subgraph 43 in the space of the feature map 2b.

In step 160, outputs 3 related to the set task are evaluated from the feature map 2b.

According to block 161, an additional task head 12 of neural network 1 can be used to evaluate output 3 from feature map 2b.

According to block 162, evaluating feature map 2b may include assigning feature map 2b to a class as output 3 using a Gaussian process.

In step 170, the set cost function 5 is used to evaluate how similar the feature map 2b is to the representation 43a of the subgraph 43. Cost function 5 outputs evaluation 5a.

According to block 171, the similarity of the feature map 2b to the representation 43a of the subgraph 43 is the representation of the subgraph 43 to the feature map 2b' calculated for the other training example 2a' using the other target output 3a'. The similarity can be compared with that of 43a.

In step 180, parameters 1a, 11a characterizing the behavior of the neural network 1 (and here in particular the feature extractor 11) are determined in order to predictably improve the evaluation 5a by the cost function 5 during further processing of the training example 2a. Optimized. The final training states of parameters 1a, 11a are labeled 1a ^* , 11a ^* .

According to block 181, the selection of subgraph 43 can also be introduced into the optimization in conjunction with evaluation 5a by cost function 5.

In step 190, the evaluation of the feature map 2b is adjusted such that the output 3 for each training example 2a corresponds as closely as possible to the target output 3a for the respective training example 2a.

If the feature map 2b is evaluated using the task head 12 by block 161, then the output 3 of the task head 12 for each training example 2a is evaluated by block 191 using the cost function 5, 5' for each training example 2a. It is possible to evaluate how well the output corresponds to the target output 3a. In this case, according to block 192, the parameters 1a, 12a of the task head 12 can be optimized with respect to the evaluation 5a, 5a' according to the cost functions 5, 5'. The training here can be performed simultaneously with the training of the feature extractor 11, or only after the training of the feature extractor 11 is completed.

If the feature map 2b is evaluated using a Gaussian process according to block 162, then according to block 193 it is possible to calculate the feature map 2b for all training examples 2a. According to block 194, decision limits between classes in the space of feature maps 2b can then be determined based on the respective target outputs 3a.

In the example shown in FIG. 1, measured data 2 is provided to the trained neural network 1 in step 200, so that the trained neural network 1 generates an output 3. In the example shown in FIG.

In step 210, a drive control signal 210a is formed from the output 3 of the neural network 1.

In step 220, the vehicle 50, the driving support system 60, the quality control system 70, the area monitoring system 80, and/or the medical image forming system 90 are drive-controlled by the drive control signal 210a.

FIG. 2 shows how different contexts regarding the subgraphs 43 of the comprehensive knowledge graph 4 can be taken into account in the training of the neural network 1.

The first subgraph 43 represents the visual context and has an exemplary representation 43a in the space of the feature map 2b. The second subgraph 43' represents the taxonomic context and has an exemplary representation 43a' in the space of the feature map 2b. The third subgraph 43'' represents the taxonomic context and has an exemplary representation 43a'' in the space of the feature map 2b.

Depending on which context is selected, for a given training example 2a, the representation 43a, 43a' or 43a'' of the respective subgraph 43, 43' or 43' is trained using the target output 3a. It is compared with the feature map 2b of Example 2a to evaluate the similarity.

Claims (16)

A method (100) for training a neural network (1) for evaluating measurement data (2), wherein said neural network (1) performs feature extraction to generate a feature map (2b) from measurement data (2). A method (100) comprising a container (11),
- preparing a training example (2a) labeled with a target output (3a) related to the set task (110);
- providing (120) a comprehensive knowledge graph (4) in which nodes (41) represent entities and edges (42) represent relationships between said entities;
- selecting (130) from the comprehensive knowledge graph (4) a sub-graph (43) relevant to the context for solving the set task;
- calculating (140) a feature map (2b) for each training example (2a) using the feature extractor (11) of the neural network;
- calculating (150) a representation (43a) of a subgraph (43) in the space of said feature map (2b) from each training example (2a) associated with a respective target output (3a);
- evaluating (160) an output (3) related to the set task from the feature map (2b);
a step (170) of evaluating how similar the feature map (2b) is to the representation (43a) of the subgraph (43) using the set cost function (5);
- Optimizing the parameters (1a, 11a) characterizing the behavior of the neural network (1) in order to predictably improve the evaluation (5a) by the cost function (5) during further processing of the training example (2a). a step (180) of
adjusting the evaluation of said feature map (2b) such that the output (3) for each training example (2a) corresponds as well as possible to the target output (3a) for the respective training example (2a); 190) and
(100). The neural network (1) additionally comprises a task head (12) for evaluating (160, 161) the feature map (2b) with respect to the set task;
The step (190) of adjusting the evaluation of the feature map (2b) comprises:
- How well the output (3) of the task head (12) for each training example (2a) corresponds to the target output (3a) for each training example (2a) is determined by a cost function (5,5' ), a step (191) of evaluating using
- optimizing parameters (1a, 12a) of the task head (12) with respect to evaluation (5a, 5a') by the cost function (5, 5') (192);
including,
The method (100) of claim 1. The similarity of the feature map (2b) to the representation (43a) of the subgraph (43) is a feature map (2b) calculated for another training example (2a') using another target output (3a'). ') is compared (171) with the similarity of the representation (43a) of the subgraph (43);
A method (100) according to claim 1 or 2. The target output (3a) includes classification scores associated with one or more classes of the set classifications of the measurement data (2).
A method (100) according to any one of claims 1 to 3. The set classification class represents the type of object that is shown to be present in the area monitored by the measurement data (2) at the time of recording the measurement data (2).
A method (100) according to claim 4. other vehicles, traffic signs, lane markings, traffic obstacles, and/or other traffic-related objects around the vehicle (50) are selected as the object type;
The method (100) of claim 5. Evaluating (160) the feature map (2b) comprises assigning (162) the feature map (2b) to a class using a Gaussian process;
The step (190) of adjusting the evaluation comprises:
- calculating feature maps (2b) for all training examples (2a) (193);
- determining (194) decision limits between classes in the space of said feature maps (2b) based on the respective target outputs (3a);
including,
A method (100) according to any one of claims 4 to 6. images, audio signals, time series of measurement values, radar data, and/or LiDAR data are selected as measurement data (2);
A method (100) according to any one of claims 1 to 7. The subgraph (43) is related to a visual context, a taxonomic context, or a functional context.
A method (100) according to any one of claims 1 to 8. The selection of said subgraph (43) is also introduced (181) into said optimizing step (180) in conjunction with evaluation (5a) by said cost function (5);
A method (100) according to any one of claims 1 to 9. The representation (43a) of the subgraph (43) in the space of the feature map (2b) is retrieved from a pre-computed lookup table based on the training example (2a) and the target output (3a). be (151),
A method (100) according to any one of claims 1 to 10. another machine learning model is trained (152) with the neural network (1) to generate a representation (43a) of the subgraph (43) in the space of the feature map (2b);
A method (100) according to any one of claims 1 to 10. - a trained neural network (1) is fed (200) measurement data (2), which causes the trained neural network (1) to generate an output (3);
a drive control signal (210a) is formed (210) from the output (3) of the neural network (1);
- The vehicle (50), the driving support system (60), the quality control system (70), the area monitoring system (80), and/or the medical image forming system (90) are drive-controlled by the drive control signal (210a). be done (220),
A method (100) according to any one of claims 1 to 12. 14. A method (100) according to any one of claims 1 to 13, when executed on one or more computers and/or computer instances, on said one or more computers and/or computer instances. A computer program containing machine-readable instructions for implementation. A machine-readable data carrier comprising a computer program according to claim 14. One or more computers and/or computer instances comprising a computer program according to claim 14 and/or comprising a machine-readable data carrier according to claim 15.