EP4232970A1 - Method for annotating training data - Google Patents

Abstract

The invention relates to a method for annotating training data for an artificial intelligence system, comprising the following steps: - storing, in a database, a data set to be annotated, - storing, in the database, at least one first description of a first facet for selecting data in the data set, the first description being associated with a first task to be executed by the artificial intelligence system, - selecting the first facet in the database, - applying the first facet to data from the data set to obtain first filtered data, - receiving at least one first annotation from the first filtered data, and - storing the first annotation in the database in association with the first facet.

Description

Title of the invention: METHOD FOR ANNOTATING TRAINING DATA

TECHNICAL FIELD OF THE INVENTION

The present invention relates to the field of machine learning (or "machine learning" in English). More particularly, the present invention relates to the annotation of data used for machine learning or performance evaluation.

TECHNICAL BACKGROUND

[0002] In the field of machine learning, so-called “supervised” models must be “trained” on sets of annotated data (called “dataset”). These are, for example, images, videos or the like. These datasets can also be used to evaluate the performance of machines by comparing their predictions on raw data to annotations.

[0003] The data is annotated by concepts that we seek to "teach" to machines so that they are able to predict them automatically on data (images, videos, etc.) which are subsequently given to them in entrance.

[0004] For example, in the case of an image classification problem where one seeks to recognize an object, for example a type of animal, present in an image, an image of a “dataset” containing a chat can be annotated with a label “chat”. In a dataset, a label is thus metadata associated with the initial data which will inform the machine, during the training phase, about the concept that it must recognize there.

[0005] If moreover, in a detection problem, one seeks to know the position of the object in the image, it is possible to annotate the object, in addition to the "cat" label, with a enclosing region (or "box") that delimits the extremities of the "cat". It is also possible that there are several times the same object in the image (that is to say several cats in the image), in which case several "bounding boxes" annotations can be added to the dataset. As with the “chat” tag, the “bounding box” tag is metadata associated with the initial data.

[0006] Finally, for a segmentation problem, it is for example even possible to go so far as to delimit the objects (“cats”) to the nearest pixel.

[0007] In the case of a practical application of artificial intelligence (or “AI”), there are often several levels of concepts to be annotated.

[0008] For example, in the case of an assistance application for technicians coming to connect a house or a building to the optical fiber, one wishes to teach a machine to verify that a certain number of steps have been respected by the technicians on the basis of photographs taken by the latter.

[0009] One of these steps may, for example, involve measuring the quality of the signal leaving the fiber with an optical power meter or “wattmeter”. The technician must then take a photograph of the power meter indicating on the screen a figure representing the quality of the signal. From the point of view of the AI application, it is natural to annotate the images in three hierarchical levels.

[0010] The first level of annotation could be the context in which the photo is taken. For example, it is a question of determining if the photograph represents the connection box in the cellar, the fiber outlet box in the place of life or the wattmeter. This is therefore a classification problem. Determine what is in the images.

[0011] The second level of annotation could be for each context, to annotate the information to be recognized. In the context of the ^wattmeter” context, we want for example to annotate the position of the screen to read the number on the screen. This is therefore a detection problem. It determines where an object is located in the images.

[0012] The third level of annotation could be, in the case of the power meter screen, the annotation of the value indicated on the screen. This is an automatic character recognition (or "OCR") problem. We determine a text on the images.

[0013] These three levels of annotation are an example. In particular, the order in which they are performed strongly depends on the practical case. In other cases, one can very well begin with a problem of detection, then a problem of classification at the second level.

[0014] We can also take the example of the annotation of photographs of meal trays. The objective is to train an AI application allowing automatic invoicing in a company restaurant. Based on a photograph of a meal tray, the AI can determine the price of what's inside.

[0015] In this example, the different annotation levels can be as follows.

[0016] The first level of annotation can for example be the annotation of bounding boxes around the different dishes and a label according to the type of dish. This is a detection problem followed by a classification problem. We determine where an object is then we determine the type of this object (for example a starter, a main course, a dessert etc.).

[0017] The second level of annotation can for example be, for each type of dish, the fine annotation of the nature of the dish (for example for the "starter" type, it can be carrot salad, terrine , etc.).

[0018] It thus appears that the annotation of the data is conventionally done in a hierarchical manner and defines a tree (technically, the underlying structure is a forest because there can be multiple roots that give rise to multiple annotation trees). The tree is not necessarily regular. There may be little depth level on one side and a lot on the other.

[0019] The standard practice of the state of the art is to constitute as many data sets as there are nodes in the tree. In particular, the images are cropped around the objects of interest to normalize the data and facilitate the learning process of the AIs.

[0020] For example, the images of the third level dataset in the fiber connection example (the reading of the power meter value) would all be cropped so that the power meter screen covers almost all of the 'picture. In fact, the image would typically be cropped using the bounding box of the second annotation level (to detect the presence of the screen).

[0021] In the case where there are several boxes in the image (example of the meal tray), each box on an image at a given level produces several images in the lower levels.

This solution has the advantage of being conceptually simple. It is motivated by the technical needs of the AIs to be trained. We thus have as many data sets as models to train, and we precisely annotate in each data set the list of information that the AI must be able to recognize.

[0023] However, this solution has very many drawbacks.

[0024] Firstly, this solution does not make it possible to easily propagate the annotation changes, in particular in the case of the correction of initial annotation errors. For example, in the case of meal trays, if a plate was initially classified as being a main course, it will have been injected into the level 2 dataset corresponding to “Main courses” . However, if it is in fact, for example, an entry and this is corrected in the level 1 dataset gathering all the meal trays, the state-of-the-art method which Breaking the problem down into different separate datasets is not going to reflect those changes. It will therefore be necessary to think of manually deleting the image from the level 2 data set “Main Courses” to put it in “Starters”.

[0025] Secondly, the difficulties stated above can be partially relieved with computer scripts responsible for verifying the integrity of the dataset, or even automating certain tasks such as, for example, the cropping and filtering of images. These scripts are implemented in an ad-hoc way and are most of the time little or not tested, which contributes to increasing the risk of errors.

[0026] Finally, in cases where the annotations relate to a large number of classes, the annotation of a set of data by following the "classic" method of the prior art is difficult because it is necessary to bear in mind the set of classes available for annotate without making mistakes.

[0027] The capabilities of artificial intelligence greatly depend on the quality of the initial training dataset. We can therefore see that the quality of annotations is a major problem encountered in the field of machine learning.

[0028] However, these datasets sometimes include a number of errors that is far too large to allow effective training of the artificial intelligence. It is possible to provide artificial intelligence correction mechanisms, but this costs resources and may also require some time before it reaches the expected level of performance. This can greatly delay deployment on an industrial scale.

[0029] As presented above, the annotation and generation of datasets is very often done in a hierarchical manner. This implies that an annotation error can propagate very deep in the annotation tree. Once the error has been propagated, it is practically impossible to go back, except to completely re-annotate the dataset, which is in practice unthinkable. We can then at best correct an annotation error at one level but not at lower levels, which does not really improve the quality of the dataset since it introduces inconsistencies.

[0030] Thus, there is a need to improve the annotation of datasets used in the design of machines used in artificial intelligence applications. The present invention falls within this context.

Summary of the invention

A first aspect of the invention relates to a learning data annotation method for an artificial intelligence comprising the following steps: storing, in a database, a set of data to be annotated, storing, in said base of data, at least a first description of a first data selection facet in said data set, said first description being associated with a first task to be executed by said artificial intelligence, selecting said first facet in said database data, applying said first facet to data of said data set to obtain first filtered data, receiving at least a first annotation of said first filtered data, and storing said first annotation in the database in association with said first facet.

[0032] For example, said database includes a plurality of descriptions of a plurality of data selection facets in said data set and said first description includes a hierarchical link to a second des- writing a second facet of data in the database, said first facet is applied to second filtered data obtained by applying said second facet to data of said set of data.

[0033] For another example: said second facet relates to a plurality of sub-regions in said set of data, and the first facet is applied to each region to which the second facet relates.

[0034] These sub-regions are sub-portions of the data in said data set.

According to embodiments: annotations are associated with some of said regions to which said second facet relates as well as with said second facet, and the first facet is applied to each region bearing an annotation associated with the second facet.

For example, the description of the first facet comprises a filtering condition applied to the annotations associated with said regions as well as with said second facet and in which the first facet is only applied for the regions for which said filtering condition is verified.

According to embodiments, said filtering condition is associated with the regions annotated by said second facet and in which the first facet is only applied to the data resulting from a cropping by these regions and for which the condition is verified.

[0038] According to embodiments, said annotation generates the definition of a region in said set of data, said region is stored in the database in relation to the region having been used to crop the annotated data and said annotation is stored in said database in relation to said first facet and said region.

According to embodiments, said annotation does not create a new region and in which said annotation is stored in said database in relation to said first facet as well as the region having been used to crop the annotated data.

[0040] The method may further comprise a step of displaying said first filtered data for a user, said annotation being received from said user.

Alternatively, said first filtered data is supplied as input to an artificial intelligence module implementing said task, said annotation being received from said module.

A second aspect of the invention relates to a machine learning method, for the execution of a task by an artificial intelligence, comprising the following steps: accessing a database comprising a set of data and at least one definition of at least one data selection facet in said set of data, said one definition further comprising at least one annotation associated with said facet, applying said data selection facet to said data set to obtain first filtered data, storing said first filtered data in an annotated training data memory, associating said first data filtered to annotations performing said task by said artificial intelligence.

For example, said annotation is generated according to a method according to the first aspect of the invention.

A third aspect of the invention relates to a device comprising a processing unit configured for the implementation of steps according to a method according to the first and/or the second aspect(s).

FIGURES

[0045] [fig-1] [fig.1] illustrates data annotation for a root view according to embodiments.

[fig.2] [fig.2] illustrates data annotation for a region-creating child view according to embodiments.

[fig.3] [fig.3] illustrates a context for using data annotation for a non-region-creating child view according to embodiments.

[fig.4] [fig.4] illustrates the fact that a datum is not annotated by a child view when the condition thereof is not satisfied according to embodiments.

[fig.5] [fig.5] schematically illustrates an annotation interface according to embodiments.

[fig.6] [fig.6] schematically illustrates an annotation statistics interface according to embodiments.

[fig.7] [fig.7] schematically illustrates an interface for accessing non-annotated images according to embodiments.

[fig.8] [fig.8] schematically illustrates an annotation process according to embodiments.

[fig.9] [fig.9] schematically illustrates the structure of a database according to embodiments.

[fig.10] [fig.10] schematically illustrates an example of image annotation from a helpdesk application to a fiber patch according to embodiments.

[fig.11] [fig.11] schematically illustrates an annotation method according to embodiments.

[fig.12] [fig.12] schematically illustrates an annotation method according to embodiments.

[fig.13] [fig.13] schematically illustrates an annotation according to embodiments.

[fig.14] [fig.14] schematically illustrates a use of annotations produced according to embodiments.

[fig.15] [fig.15] schematically illustrates a device according to embodiments.

DETAILED DESCRIPTION OF THE INVENTION

[0046] In the following, embodiments are described that offer data annotation in a hierarchical manner. For example, they allow the design of image or video recognition applications based on machine learning. The invention is not limited to this type of application and other types of data can be used.

The embodiments of the invention make it possible to manipulate and store the hierarchical character of the annotated concepts, to annotate more efficiently and to decouple the notion of machine learning model from the notion of data set (or " dataset” in English).

According to the invention, the embodiments take advantage of the hierarchical structure of the data to be annotated to facilitate the annotation even though this hierarchical structure is a source of problem in the prior art.

It thus becomes possible to divide an annotation work into several independent sub-tasks.

The embodiments make it possible to respond to the problem of error propagation in the models of the prior art. For example, they make it possible to automatically take class changes into account in order to pass them on automatically.

[0051] The embodiments also make it possible to annotate data without causing inflation thereof as the annotation progresses. The generation phase of the actually annotated data can be postponed to an effective training phase.

[0052] As described in detail below, the annotation according to the embodiments takes into account the hierarchical structure of the annotations to be performed. A Automatic cropping of data around relevant regions to be annotated can be performed for annotation without generating new data subsets. It is also possible to filter the images to be annotated to display only the relevant data regions.

In what follows, the principles applicable to the various embodiments of the invention are first presented.

[0054] Unlike the annotation techniques of the prior art, the embodiments of the invention will not attempt to create “sub-datasets” as annotations are made on the data from the datasets. Thus, the inflation of the dataset by the annotation process which is the source of the propagation of errors in the prior art is not repeated.

On the contrary, the initial dataset will be kept and a dynamic dataset construction system (the “facets” or “views”) will be created with which the annotations will be associated. Thus, it will be possible to make any manipulation, including corrections on this construction system, without touching the dataset data. The "sub-datasets" will be generated on demand, according to the desired use (for example the training of an AI) once the construction system has been validated.

The context of implementation of the embodiments of the invention is thus the following.

[0057] A set of unstructured data to be annotated to allow the training of an artificial intelligence on a certain number of tasks is considered.

[0058] “Data” (not annotated) refers to any type of data that can typically be produced by a sensor, a capture device or a manual input device. It can be an image, a video stream, a point cloud, a 3D image made of voxels, a sound stream, a text, a time series, etc. .

[0059] Annotations are the additional information linked to a datum and relating to its content. For example the position of an object in an image and/or its "class" (or "type").

[0060] A “task” designates any action of a machine allowing it to automatically predict the annotations of a datum, or a subset of the annotations. A large number of tasks can be mentioned. A few examples are cited below.

For a classification task, the annotation to be predicted is an object category, otherwise called a “class”, from a predetermined list of possible classes. For example, given a photo of an animal, we try to find out which animal it is.

For a detection task, the annotation to be predicted is a list of objects present in the image from a list of classes of interest. Each predicted object must indicate a simple delimitation of the object, typically in the form of a bounding box as well than his class. For example, given a video captured by an autonomous car, we want to know where all the vehicles, pedestrians, cyclists, etc. are.

For a segmentation task, the annotation to be predicted is the same as for a detection task, but the objects must be delimited to the nearest pixel.

For an OCR task (acronym for “Optical Character Recognition”), the aim is to predict the text present in an image. For example, reading a license plate number from a plate photo.

[0065] For a pose estimation task, it is a question of predicting the “pose” of a deformable object. Typically, key parts of the object are identified beforehand and linked by a tree. It is then a question of predicting the position of the various nodes of the tree if they are visible or of indicating that they are invisible. For example, in the case of estimating the pose of a person, one typically seeks to locate the head, hands, feet, etc. of each person in the image.

For a regression task, it involves predicting a number or a vector (i.e. a list of N numbers, N being known beforehand). For example, it involves predicting a person's age given a photo of a face.

This list of tasks is of course not exhaustive but makes it possible to realize the variety of tasks that can be requested from an AI and therefore the variety of possible annotations in a dataset.

When these different tasks contribute to solving the same problem by relying on one or more automatic recognition algorithms, it is common for these tasks to be organized in a hierarchical manner.

[0069] The hierarchical relationship between a task A and a task B can for example appear when the need to annotate data for task B depends on the annotation of these data for task A. This is for example the case where the need to annotate for task B depends on the class of object indicated in A. In the example of the verification of connection given in the introduction, it is for example the annotation of the power meter screen which is only relevant in the context where the image actually shows a power meter and has been annotated as such in the first level of annotation. It is therefore a question of being able to “filter” the regions to be annotated for task B according to a certain condition on the annotation of task A.

The hierarchy relationship can also appear when task B consists of annotating the same data regions as those defined by task A. In the example given in the introduction concerning the meal trays, it is for example annotate the nature of the dish on the image resulting from the cropping of the original image by the region defined in task A.

[0071] Hierarchy relationships can appear for any other reason that leads to the fact that the annotation for task A can be directly reused to define the need annotation or simplify the annotation process for task B.

As mentioned above, it is common to annotate the same region of a datum in different tasks. For example, this is the case when a classification task attempts to annotate the detailed class of an object created during an upstream detection task. A region is called a sub-part of a datum defined by its extremal values on the different axes of the datum (x, y, z for spatial data and/or t for video). When the temporal axis is involved (for example for a video), the spatial position (i.e. in x, y, z) can vary for each value of t between t_min and t_max. The interest of a region is to be able to reframe a datum to produce a new (sub-)datum to annotate.

In accordance with the embodiments described below, in order to represent the hierarchical relationships between tasks and regions, the notion of “facet” is used. Each annotation task is assigned a unique facet of its own (i.e. a task is linked to a unique facet and vice versa).

[0074] A facet is understood by analogy with the term facet in the field of information retrieval based on a facet classification, which gives users the ability to filter the data according to the selected facet. In what follows, and without losing generality, the term "view" is used.

In addition to being linked to a task, a view can be hierarchically attached to another view, called “parent view” or “parent” (or conversely a “child view” or “child view”). A view that does not have a parent view is called a “root view”.

When a view is a child view, it defines a “filtering condition” on the annotation of the parent view, or “condition”. The data from the parent view satisfying the condition defines the child view. This allows to create sub datasets.

[0077] An annotation process can have several root views. Formally, the relationship between the views induces a forest structure in the sense of a set of disjoint trees as defined in graph theory.

As described below, the views as defined above make it possible, in the embodiments of the invention, to filter and crop the regions produced by the parent view in order to present to the user who performs annotation, valid data and centered on regions of interest. As explained in the following, this allows for more efficient annotation. This also makes it possible to manipulate data on the fly in the annotation phase and to reserve the generation phase of annotated datasets for the AI training phase.

A more formal definition of views is given below.

The set of all possible regions R is defined and a root region r _{i , o} ∈ R is associated with each of the data d _i ∈ D of the dataset. Thus, each data has at least one root view. We define the reframing function crop: DXR → D which takes as input a datum d and a region ^f and returns the sub-data resulting from the reframing of d by ^r . Cropping by a root region does not modify the data:

[0082] We note { v _j | j = 1, . . . , n | the set of ⁿ views and p : NN (with the set of natural numbers) the parent function such that p(J) = 0 if ^v j is a root view and p(j) = k if is the parent view of ^v j.

[0083] We note annotated ( j , r ) e { 0,1 } the function which defines whether the region ^r is annotated for the view ^v j (we also set annotated(0, r) = 1), and A _j the set of possible annotations for ^v j (the exact form of A . depends on the type of task associated with ^v j: classification, detection, etc...).

[0084] We also note the function which makes it possible to recover the annotations of a region already annotated in view ^v j.

The set of regions covered by the annotations of a region ^r by the view ^v j is denoted regions(j, r).

For example, in the context of a region detection task ( j , r ) corresponds to the set of regions defined by the newly annotated bounding boxes. If no region is created (as in the case of a classification task), then we have regions(J , r) = { r) . To handle the case of root views, we set regions(O, r _{i, o} ) = { r _{i, o} } .

[0087] We note C . R {0,1 } the function which is worth 0 or 1 depending on whether the filtering condition of the view ^v j is valid or not (for any root view we define

We call R _{i, j} the set of regions to be annotated for the view ^v j on the data. We set R _{i, 0} = { r _{i, o} }, such that a . and

The data presented to the annotation for the view ^v j are all the data reframed by the regions to be annotated, i.e.

D _j = { crop ( d _i , r) | d _i ∈ D and r ∈ R _i,j } . These data can then be annotated in the same way as the state of the art, ie by considering Dj as a stream of data to be annotated which can be taken in isolation from the rest of the annotation process.

[0090] Thus, the interaction of a view with its parent view (through the notions of region and conditions, defined above) is the way to implement the necessary manipulations during the annotation in order to speed up and make the process of annotation of the task to which it is attached more reliable. This makes it possible to overcome the inflation of the data during the annotation phase according to the techniques of the prior art.

In other words, the creation of the notion of “view” allows dynamic annotation of the data, which has the consequence of not creating sub-datasets as the annotation progresses. According to the embodiments described below, it is possible to annotate the data without modifying the data itself. It is thus possible to modify any annotation at any time and this with the possibility that does not exist in the prior art of passing on these modifications as deeply as desired since the list of data to be annotated for a given task and the cropping on the area of interest are calculated on the fly based on the annotations already present in the parent task.

[0092] In accordance with the definitions and formulas stated above, it is known how to formally define the data to be presented to a user (or an automatic process) from an initial set, for the annotation. As described in what follows, in the annotation process, instead of making direct links between data and annotations, it is a question here of making links between annotations and formalizing the way to obtain the data presented in the user (or to the process) from the initial data. This association improves the annotation process.

In what follows, the formal equations given above are illustrated for different scenarios.

With reference to FIG. 1, the case of a root view ^v j 101 is illustrated where the datum presented to the annotation is exactly the raw datum of the set of data to be annotated. Indeed, the region to be annotated is necessarily the root region 103 which includes all the data d _i 102. This is found via the formalism above because and therefore the data presented to the annotation is The root view 101 therefore behaves like a task of “standard” annotation which would follow the practices of the state of the art according to which one adds the annotations 104 to the data set D for the task linked to ^v j. The difference is that the annotation is attached to the root region 103 and not directly to the data 102. In the case where there are several root views, all the annotations are attached to the same root region 103.

In what follows, we consider cases where the regions to be annotated for a view B are this time those resulting from an annotation of view A.

In the example of [fig.2], we start by applying a parent view A 200 (parent of view B 202) on a datum 203 from the set of data to be annotated. It is assumed that this parent view A creates new regions A 204 and 205 (for example in the case of a detection task) which receive annotations 206 and 207 respectively. root 208 to simplify the figure (the process would be similar for the case of a non-root region).

It is assumed that the annotations of region 205 (solid line) verify a condition B while those of region 204 do not verify it (dotted line). Thus only the datum 203 cropped by region A 205 verifying condition B will be presented for the annotation according to view B. The datum resulting from a cropping by region 204 is not presented.

In general, the data to be annotated for view B 202 are the initial data cropped by the regions annotated by view A 200 whose annotation verifies condition B. After application of view B 202, the The annotation 209 is associated with the region 205 which was used to reframe the annotated data.

The example of [fig.2] is very simplified and only represents a single initial datum 203 . If, for example, view A 200 creates two regions on a first piece of data and three regions on a second piece of data, then there would be 5 data to annotate for view B 202, resulting from the cropping of each of the 5 regions produced by view A 200 .

[0100] It can therefore be seen in the example of [fig.2] that the set of initial data always forms a whole without anything being added to it. It's just different ways of “seeing” it, the views, that are created as you go, and the annotations are associated with those views and the regions that carry them, not directly with any data.

According to the prior art, instead of keeping a single set of data 203, two subsets of data would have been created corresponding to the two tasks of views A and B. For view A, which could for example be a detection view, the dataset would include the datum 203 cropped by the region 208, annotated by two regions (for example bounding boxes and their classes) 204+206 and 205+207. For view B, which could be a classification view, the dataset would include data 203 cropped by region 205 and annotated by 209.

According to the invention, the datum 203 is not modified and the annotations are not added directly to it. Instead, we create a representation of views A and B, as well as regions, and associate the annotations with these regions. During the annotation process, there is no creation of new data subsets. Only views are used to generate the data to be annotated and allow the user (or an automatic process) to annotate it. These same views can be used to effectively generate the annotated data to give them as input to an AI in the training phase. In the meantime, the storage and processing of annotations is greatly facilitated because the hierarchy between the annotations is automatically preserved, which makes the process of updating (modifying or deleting) the annotations more reliable.

In the example of [fig.3], the parent view A does not create new regions (for example in the case of a classification task). In this case the annotation for view B covers the same regions as view A.

We begin by applying a parent view A 300 (parent of view B 301) to a datum 302 resulting from the set of data to be annotated. As indicated, this parent view A 300 does not create new regions as before. We therefore find after the application of the view A 300, the same region 303. This region is associated with the annotation 304. It is assumed that the view A applies to a root region 303 to simplify the figure (one could consider the case of a region that is not).

It is assumed that the annotations 304 verify the condition of view B 301. The datum to be annotated for view B 301 is then the initial datum 302 cropped by region 303. That is to say that already annotated by view A 300. After applying view B 301, annotation 305 is associated with region 303.

[0106] It can therefore be seen in the example of [fig.3] that, as for [fig.2], the set of initial data always forms a whole without anything being added to it. It's just different ways of “seeing” it, the views, that are created as you go and the annotations are associated with those views.

According to the prior art, instead of keeping a single set of data, two subsets of data would have been created corresponding to the two tasks of views A and B. For view A, which could for example be a view classification, the dataset would include data 302 cropped by region 303 and annotated by 304. For view B, which could also be for example a classification view, the dataset would include data 302 cropped by region 303 and annotated by 305.

[0108] The example of [fig.4] is similar to that of [fig.3] except that it is considered this time that the annotations of view A 304 do not satisfy condition B of view B .

[0109] In this case, the data region will not be presented to the user for annotation and no annotation will be added (the annotation 305 is absent).

The annotation of data according to the above principles can take the form described below.

[0111] A user can for example access an interface 500 of an annotation system as illustrated by [fig.5]. This interface is used to display the views and to manipulate the hierarchies between them.

Thus, the interface 500 includes an action zone 501 including various buttons 502 (ACT1), 503 (ACT2), 504 (ACT3). For example, these buttons allow the user to manage the dataset in general, such as adding data, viewing view mapping, deleting images, etc.

The interface 500 also includes a zone 505 for displaying the root views. Here, for brevity, a root view 506 (VI) is shown. Other root views may be present in this area. In this zone, a button 508 allows the user to add root views.

The user can select a view, for example 506, and a zone 509 similar to 500 appears to display the child views of the selected view. For example, views 510 (V1.1), 511 (V1.2), 512 (V1.3) depend on view 506 (VI) which is therefore a parent view for them. In this zone, a button 513 allows the user to add views dependent on the view selected in the zone 500. For example, the user selects the view 506, then clicks on the button 513 to create a view dependent on view 506.

The method continues recursively as long as there is depth in the tree of views. For example, a zone 514 makes it possible to display views dependent on the view selected in the zone 509 and a button 515 makes it possible to add a dependent view to the selected view. In the example illustrated, the view 511 is for example selected but does not contain a child view. The user then clicks on the button 516 to create a first view dependent on this selected view.

[0116] As illustrated by [fig.6], when he clicks on a view of [fig.5], the user can view the statistics linked to this view.

The [fig.6] illustrates an interface 600 comprising an action zone 601 with a set of buttons 602 (ACT4), 603 (ACT5), 604 (ACT6). For example, these buttons allow the user to annotate images, correct annotations, add concepts to recognize, etc.

The interface 600 also comprises an area 605, with a certain number of windows allowing the user to manage the images of a view. For example, a window 606 (DISTRI IMG) gives access to a distribution of the images of the view between different uses (it is possible to find the number of training data, of non-annotated data, or other). This allows you to know what use the images in the view are intended for. A window 607 (DISTR2 CNCPT) gives access to another distribution concerning the concepts that the machine will have to predict. For each concept, we can see the number of images associated with it.

A window 608 (IMG) can give access to the number of images in the view. A window 609 (CNCPT) gives access to the number of concepts associated with the view.

[0120] One of the buttons in area 601 can give the user access to images not annotated in an interface 700 illustrated by [fig.7].

This interface 700 includes an action zone 701 with a certain number of buttons 702 (ACT7), 703 (ACT8), 704 (ACT9) allowing the user to perform a certain number of actions. These buttons are for example the same as those of the 600 interface or can be supplemented by others. According to certain examples, it may comprise a button opening the possibility for the user to annotate an image not yet annotated, thanks to an annotation interface making it possible to carry out an annotation specific to the type of task linked to the view selected in the interface 600. This interface complies with the practices of the state of the art and is not described here. That said, contrary to the state of the art, it does not apply directly to the annotated data but to the region of interest as described in [fig.8] below.

The interface 700 further comprises an area 705 with all the images 706 (IMG) not yet annotated. For example, the user selects an image in the area by clicking on it and is redirected to the image annotation interface for the selected task and view as before.

[0123] With interfaces as presented above, it can be seen that the annotation process is totally different from that of the state of the art. Indeed, the data to be annotated can be displayed to the user "on the fly" according to the different views. We therefore take advantage of the hierarchy of tasks, without creating new data with each annotation.

[0124] It is therefore possible to annotate the data automatically according to the parent views. The hierarchy of views in the form of a tree is therefore different from a generation of annotated data as according to the prior art. This hierarchy allows a display for adding annotation, not on the data but on the regions produced by the parent views of the view being annotated.

[0125] The process is shown schematically in [fig.8]. The user typically annotates the data one after the other. We describe here how the data stream to be annotated is generated. As input, we find the stream of regions annotated 803 (REG) by the parent view of the view selected for the annotation, called the current view below. If the current view is a root view (ie it has no parent view), then this stream consists of all the root regions. Then, during step 800 (FILTR), the regions are filtered by the condition to be applied for the annotation according to the current view. At the output, we then find the useful regions 804 (REG_CHLD) for the desired annotation. During step 801 (CROP), the data to be annotated are presented, cropped by the useful regions 804. At output, the stream of data to be annotated 805 (DAT) is thus obtained which is then annotated during step 802 The annotations are attached to the useful region 804 and not to the data generated by the reframing 805, so that the latter can be deleted once annotated. In Output of the process is the annotated region stream for the current view 806 (REG_ANNOT). It is this stream of annotated regions that will be used instead of stream 803 for all child views of the current view. Unlike the prior art, a stream of annotated data is not directly obtained, but a stream of annotated regions, themselves linked to the data which carries them and which is not duplicated.

The method presented above is described in the case of manual annotation by a human user. However, the annotation can also be performed on the fly by an automatic annotation module. In this case, the process of [fig.8] remains valid. Step 802 is then implemented by the annotation module. Cropping is still performed to present the data to the module but it is no longer useful in this case to display the result. This embodiment can be useful in cases where an artificial intelligence is already sufficiently trained and gives sufficiently satisfactory results to be able to convert a prediction into an annotation when the confidence score is high enough. We then allow the artificial intelligence to enrich itself by giving it new annotated images.

In practice, the notions of data, views, regions, annotations are stored in a database. When the user seeks to visualize the unannotated data (see [fig.7] ), a query to the database is performed to obtain (i) all the root regions if the current view is a root view and (ii ) all regions already annotated in the parent view and which otherwise satisfy the current view's condition.

[0128] For each annotation, a new “annotation” object is created in the database. This object is linked to the view that produced it and to the region it covers.

If the task linked to the view does not create a new region but merely enriches the region passed as input (as for example in the case of classification) then the newly created annotation is linked to this region and it is considered as annotated for the current view. The annotation then becomes available for annotation in child views.

[0130] If the task linked to the view creates new regions (as for example in the case of detection), it is these new regions that become available for annotation in the child views. For practical reasons, these new regions store their parent region (see step 1305 written below) in the database, in particular to allow the deletion of cascading child regions and annotations as explained below.

[0131] The storage of annotations, not in connection with data but in connection with the hierarchical views, makes it possible to make annotation corrections in a very simple manner. For example, in case of deletion of a region, this makes it possible to rely on the cascading deletion mechanism of a database. All regions and years notations that inherit from this deleted region in child views are automatically deleted.

In addition, when an annotation is modified, one of the annotation difficulties consists in verifying whether the condition for filtering the child views is not violated, and if necessary in correctly deleting the annotations that have become invalid. In addition, a region stores the view that created it so that all regions and annotations from a given view and its children can be efficiently deleted when that view is deleted.

[0133] The [fig.9] illustrates a database according to embodiments according to a UML-type schematization.

As can be seen, both the data 901 and the views (with their conditions) 902 are linked to the dataset 900 to which they belong. The regions 903 are for their part linked to the data 901 and to the views (with their conditions) 902 which carry them. As a view must store a reference to its parent view (this reference is null in the case of a root view), the view table is linked to itself in [fig.9]. Similarly, regions can store their parent region as explained above. Finally, the annotations 904 inherit both the regions 903 and the views and conditions since the annotations are linked to one and the other.

In comparison, a diagram of the same type is also represented in FIG. 9 for an annotation database according to the prior art. This time, the structure is much simpler since the annotations 907 are linked to the data 906 which are themselves linked to the dataset 905. The database according to the prior art does not have the notion of view which allows the creation at the fly of data to annotate according to the annotations of the previous step when the annotation task is hierarchical.

[0136] The annotation of images in the case of an assistance application for technicians coming to connect a house or a building to the optical fiber according to embodiments is now described with reference to [fig.10] .

[0200] We first assume a first root view v1 of classification 1000 (VI) named “Context” with two classes (or type): “Wattmeter” and “Cabinet”. This makes it possible to differentiate between two types of photos taken by the technician: (i) either a photo of the device used to measure the power of the signal called a wattmeter (it will then be a question of saying whether the power of the signal displayed on the screen complies with the minimum threshold), (ii) or a photo of the fiber optic connection cabinet (it will then be a question of saying whether the different connection zones are valid).

We also assume a detection view v2 1001 (V2) named “Screen” having the view v1 1000 as a parent and a single class (or type) “Screen”. Its role is to make it possible to locate the screen of the wattmeter to read it. This view 1001 has a condition. The region must be annotated “Wattmeter” in the parent view for an annotation to be associated with an image of this view.

[0160] We also assume a view v3 of OCR 1002 called “Signal Quality” having as parent the view v2 1001 whose purpose is to make it possible to annotate the text on the screen. This view has no condition, ie V r R, c ₃ ( r ) = 1 (according to the formalism described above).

A detection view v4 1003 named “Connection” having as parent the view v1 1000 and two classes (or type) “OK” and “KO”. Its purpose is to identify the connection areas that are compliant or not. This view has a condition. The region must be annotated “Cabinet” in the parent view for an annotation to be associated with an image of this view.

We will now describe the link between data (in this case images in this case) and regions 1004, annotations 1005 and views 1006 for two images 1007 and 1008. Each data item carries regions organized in the form of a tree from of a root region that encompasses all the data. An annotation is attached to a region and a view. Depending on the view type, the annotation can be a class, text, etc. Each view has a type and possibly a condition.

[0142] Image 1007 represents a wattmeter. A first region 1009 is then defined which is a root region. A sub-region 1010 is also defined around the screen of the wattmeter.

The region 1009 is thus annotated 1011 according to the "Wattmeter" class. The region 1010 for its part receives two annotations: one is the “Screen” class 1012 and the other is the text recognized by OCR on the screen 1013 for example “-4.6 dB”. The annotations 1011, 1012, 1013 are thus respectively associated with the views 1000, 1001, 1002.

[0144] Image 1008 shows a connection cabinet. A first region 1014 is then defined which is a root region. A sub-region 1015 and a sub-region 1016 are also defined which correspond to two different areas of the cabinet.

The region 1014 is thus annotated 1017 according to the “Cabinet” class. The region 1015 for its part receives an “OK” annotation 1018 because it comprises a compliant connection zone. Region 1016 receives a "KO" annotation 1019 because it has a nonconforming splice area. Annotation 1017 is associated with view v 1 1000 because the presence of the cabinet is an annotation of "Context". The annotations 1018 and 1019 are both associated with the v4 view 1003 because the good or bad connection is a “Connection” annotation.

In the above formalization, the conditions are applied to the annotations of the parent view in general. The preceding examples show that it can be particularly important to be able to define conditions on the annotated classes in the parent region. Variants according to embodiments will now be described. [0147] The condition of a view ^v j can for example relate to the class annotated in the parent view (if this is unique, see paragraph below for the multi-class case) to restrict the annotation of ^v j to certain classes only. This makes it possible, for example, to specialize views to certain shooting or object contexts, typically in order to structure and simplify the annotation work in order to have fewer classes (or types) to annotate.

[0148] Let us call the class function which associates an annotation of ^v j with its class represented as an integer, and C _j . the set of acceptable classes, then given a region ^r , we have otherwise.

In the multi-class case where the parent view makes it possible to annotate the region with several classes from a possible set of N classes, the class function can be represented by _class ; _' classes acceptable as a logical formula boolean on the classes: - In ^{this case we have} This variant actually encompasses the previous case where there is only one annotated class at a time.

According to embodiments when the annotation of the parent view does not include any notion of class, as for example in the case of the pose estimation or a textual annotation, it is possible to define “clusters” on which you can also apply conditions.

[0151] The notion of clustering is used when the annotation of a view ^v j does not directly provide a class (or type). This is for example the case for a pose estimation task where the annotation corresponds to placing the nodes of a tree on the data. It is then possible in this case to calculate beforehand a partition of the space of the annotations (clustering in English). This method divides the space A . into A ^r groups (or “clusters”) and any annotation can be associated. to the nearest cluster, i.e. we have a class function. : which allows to associate a class to a annotation.

In some variants, each annotation can be associated with several clusters and the class function has the general form class ' A {0,1 } ^/V - We then fall back exactly into the case of the previous paragraph for the case of the multi-class annotation.

In the foregoing, the tasks and the views have been associated bijectively. However, according to embodiments, a task can be divided over several views in order in particular to reduce the number of classes to be annotated in each view. That allows a person (or an annotation module) annotating to focus on fewer classes, thus being more efficient and making fewer errors.

This does not change the general embodiment according to which, in practice, it is sufficient to keep the one-to-one link between view and task. It is only when training a machine learning algorithm on a set of annotated datasets that it is necessary to be able to combine two sister views (i.e. having the same parent) into a single view.

As has been explained, the view system makes it possible to automate the handling of data and annotations, so that the data flow annotated by the different views corresponds to a set of datasets which would share a structure hierarchical. In this scheme, once data has been annotated for a given view, one can then train a machine learning model on the dataset that corresponds to the annotated data for that view.

According to embodiments, this amounts to saying that a model is trained on the data annotated by a certain view. However, it can be interesting to train a model on several views.

[0170] It is then necessary to be able to merge the annotated data from several views into a single set of annotated data.

The steps of a method are now described with reference to [fig.11] and following, using the formalism described above. This method can be implemented in the case of an annotation application implemented by computer, with for example an interface as described above. It is thus a user who interacts with the application, via the interface to annotate the data. Variants of the method can also be implemented in the context of automatic annotation, by an AI for example.

In what follows, it is considered that data (for example images) are already stored in memory and associated with their root region. This association is made as soon as a piece of data is added to the database: a root region is created for each piece of data added. Alternatively, steps for adding or deleting data to an existing dataset can be provided in what follows, either by a user or automatically. These steps are not shown.

In the case of an application, the user can for example create an annotation project in which data to be annotated and annotations will be stored according to the invention. This is, for example, a “Meal trays” or “Fibre Paccordment” project.

[0161] The process for creating a view is described in [fig.11]. During a step 1101, the description of a view, corresponding to a project task, is created. It is then determined (step 1102) whether this view is a root view. If this is not the case (N), the process continues with the selection of the parent view during step 1103. The interface of [fig.5] can for example be used, for example it is determined whether it is the button 513 or 515 which has been activated. In this case the determination of the parent view is done by determining the view previously selected by the user before clicking on one of these buttons. If button 508 has been activated, then it is determined (Y) that the view created is a root view.

[0162] Once the parent view has been selected, a condition on this parent view is defined (step 1104) to make it possible to filter the data to be proposed for annotation for the view created at step 1101. The exact form of this condition depends on the task type of the parent view. For example, if the parent view is a classification or detection view where only one class can be annotated, the condition can take the form of a drop-down list in which the user selects the different parent classes of interest, which allows to define the set C . acceptable classes defined above. If the type of parent task is multi-class and allows several classes to be annotated on the same region, the condition may be defined by a Boolean formula whose clauses relate to the presence of certain parent classes. Next, the type of annotations for this view is created (step 1105), that is to say the type of associated task (classification, OCR, pose, detection or other). Depending on the type of task, the configuration of the annotation is then defined (step 1106). For example for annotations based on classes, we define the classes of interest: "Screen" or

"Cabinet" in the example of the fiber connection. This step is optional because the classes can be modified after saving the view in memory: the user can for example use the buttons of the action zone 601 of the interface 600 of [fig.6] for example.

[0163] If the view created is a root view, the process goes from step 1102 (Y) to step 1105 directly.

[0164] Once this process is finished, the description of the view is stored in memory (step 1107).

Next, with reference to FIG. 12, the annotation process is described. It begins with the selection of a view ^v j during step 1201. It is then determined during step 1202 whether the selected view is a root view or not.

If it is a root view (Y), a loop is initialized (step 1203) during which all the regions not yet annotated according to the current view are determined (step 1204). We therefore iterate over the root regions of the data in memory: if the region is not annotated by the view ^v j, the latter is selected (step 1205). Otherwise (N) the region is ignored. The iteration is typically done via a query to a database but can be implemented by a counter i which traverses all the root regions. In step 1206, if i has not yet finished iterating over the root regions (N), the loop is incremented (step 1207) or else (Y), if all the regions have been tested, one goes to a step 1208 for storing all the regions filtered in step 1205. This is not not a storage of a new dataset or the creation of a sub-dataset but the memorization of all the regions to be annotated according to the current view, typically according to a cursor returned in response to the request to the database data.

[0167] This step is a prerequisite for displaying the data for the user, for example to prepare the display of the data to be annotated in the area 705 of the interface 700 of [fig.7],

[0168] Returning to step 1202, if the view selected is not a root view (N), then a loop is initialized (step 1209), during which, one determines (step 1210) all the regions of the data annotated by the parent view.

For each region annotated by the parent view (Y), it is determined whether it is annotated by the current view during step 1211. If the region is not annotated by the current view (Y), then determines (step 1212) if the parent view's filter condition is met for the current region. If the condition is respected (Y), then the region is selected (step 1215) to present it for annotation.

The process then continues until all regions have been considered. This is determined in step 1216. If there are still regions to consider (N), the loop is incremented in step 1217 and resumed in step 1210. Otherwise (Y), the process ends with step 1208 already described by storing the regions filtered during step 1215.

[0171] During the tests of steps 1210, 1211 and 1212, if the result is negative (N), the process continues with step 1216 as illustrated by [fig.12].

The data corresponding to the regions to be annotated stored in step 1208 are either displayed for annotation by a user, for example via the interface 705, or supplied to an automatic annotation module.

This annotation (manual or automatic) is described with reference to [fig.13].

For each region ^r stored in step 1208, the user (human or algorithm) is presented with the datum d to which ^r is attached, cropped by ¹ . Formally, at step 1300, the datum crop d, r) is presented to the user for annotation according to the view ^v j selected at 1201. At step 1301, an annotation is received for this datum. It is then determined in step 1302 whether the type of task type linked to ^v j is region-creating. If this is not the case (Y), then we pass to step 1303 of memorizing the annotation. As already indicated, this storage is done in connection, not with the data but with the current view and the region to which the data belongs.

[0175] If, on the other hand, the annotation creates one or more regions (N), denoted for example if it this is a view linked to a detection task, we pass to a step 1304 of storing a relationship between the created regions and the regions from which they come. We then proceed to a step 1305 of storing the annotation in connection with the region created and the current view and this, for each of the regions created. For example, in the context of detection, a class annotated on a bounding box is associated with the corresponding region.

At the end of the process, one can have a database according to [fig.9] which stores annotations in connection with the views, the regions and the conditions. The database also contains the initial data.

As already explained, advantageously, no sub-datasets were created. Thus, one can easily correct, update annotations at one level and propagate changes to child views, which helps to make the annotation process more reliable.

Only visualization can be performed temporarily to allow the user to visualize the data of the sub-datasets and annotate them or to allow an automatic process to take them as inputs and annotate them. This process being carried out data by data, it remains light and does not present any complexity of execution.

The generation of the datasets is then done when training the AI. Either it is done all at once, or on the fly as the training progresses.

[0180] The [fig.14] illustrates a use of annotations produced according to embodiments.

The process is initialized by step 1401 during which the annotations are accessed, for example a database according to FIG. 9. A loop is then initialized (step 1402) during which the different views ^v j (step 1403) and they are applied to the initial data during step 1404.

The result of the filtering is stored in memory during step 1405, with the annotations associated with the view ^v j. Thus, here we make the link between the data and the annotations. We thus begin to build the dataset that will be used by the AI.

It is then checked (step 1406) whether all the views have been considered. If there are still views (N), the loop is incremented (step 1407) and we return to step 1403.

If all the views have been used (Y), then we move on to step 1408 of supplying the dataset to the AI which can then be trained during step 1409 depending on the application.

[0185] In addition, provision can also be made for the data to be also linked to particular applications. In this case, the views will also be applied to a subset of data linked to the application in question.

[0186] [fig.15] is a block diagram of a system 1500 for implementing one or more embodiments of the invention. Methods according to embodiments are computer-implemented. The 1500 system includes a communication bus to which are connected:

- A processing unit 1501, such as a microprocessor, called CPU;

- a random access memory unit 1502, called RAM, for storing executable code of a method according to one embodiment of the invention as well as registers suitable for recording the variables and parameters necessary for the implementation of a method according to embodiments, the memory capacity of which can be extended by an optional RAM connected to an extension port for example;

- a memory unit 1503, called ROM, for storing computer programs intended to implement the embodiments of the invention;

- a network interface unit 1504 connected to a communication network on which the digital data to be processed are transmitted or received. The network interface 1504 can be a single network interface, or composed of a set of different network interfaces (eg, wired and wireless interfaces, or different types of wired or wireless interfaces). Data is written to the network interface for transmission or is read from the network interface for reception under the control of the software application running in the CPU 1501;

- a graphical user interface unit 1505 for receiving inputs from a user or displaying information to a user;

- a 1506 HD rated hard drive

- an I/O 1507 input/output module to receive/send data from/to external systems such as a video source or a screen.

The executable code can be stored either in ROM 1503, or on the hard disk 1506, or on a removable digital medium such as a disk for example. According to a variant, the executable code of the programs can be received by means of a communication network, via the network interface 1504, in order to be stored in one of the storage means of the communication system 1500, such as the disk hard 1506, before being executed.

The central processing unit 1501 is suitable for controlling and directing the execution of the instructions or parts of software code of the program or programs according to the embodiments of the invention, these instructions being stored in one of the aforementioned storage means. After power-up, the processing unit 1501 is capable of executing the instructions of the main RAM 1502 relating to a software application after these instructions have been loaded from the ROM program 1503 or from the hard disk (HD) 1506 for example. Such a software application, when it is executed by the central unit 1501, causes the execution of the steps of a method according to incarnations.

The present invention has been described and illustrated in this detailed description with reference to the appended figures. However, the present invention is not limited to presented embodiments. Other variants, embodiments and combinations of characteristics can be deduced and implemented by those skilled in the art on reading this description and the appended figures.

To meet specific needs, a person skilled in the field of the invention may apply modifications or adaptations.

[0192] In the claims, the term “comprising” does not exclude other elements or other steps. The indefinite article "one" does not exclude the plural. The various characteristics presented and/or claimed can be advantageously combined. Their presence in the description or in different dependent claims does not in fact exclude the possibility of combining them. The reference signs cannot be understood as limiting the scope of the invention.

Claims

Claims

[Claim 1] A method of annotating training data for artificial intelligence comprising the following steps:

- storing, in a database, a set of data to be annotated (901),

- storing, in said database, at least a first description of a first data selection facet in said data set, said first description being associated with a first task to be executed by said artificial intelligence (902) ,

- selecting said first facet in said database (1201),

- applying said first facet to data from said set of data to obtain first filtered data (1300),

- receiving at least a first annotation of said first filtered data (1301), and

- storing said first annotation in the database in association with said first facet (1303, 1304, 1305).

[Claim 2] A method according to claim 1, wherein said database includes a plurality of descriptions of a plurality of data selection facets in said data set and wherein

- said first description includes a hierarchical link to a second description of a second facet of data in the database (1103),

- said first facet is applied to second filtered data obtained by applying said second facet to data of said data set (1215).

[Claim 3] Process according to the preceding claim, in which

- said second facet relates to a plurality of regions in said data set, and

- the first facet is applied to each region covered by the second facet.

[Claim 4] Process according to the preceding claim, in which

- annotations are associated with some of said regions to which said second facet relates as well as with said second facet, and

the first facet is applied to each region carrying an annotation associated with the second facet.

[Claim 5] Process according to one of Claims 2 to 4, in which the description of the first facet comprises a filter condition applied to the annotations associated with said regions as well as with said second facet and in which the first facet is only applied for the regions for which said filter condition is verified (1212).

[Claim 6] Method according to claim 5, in which the said filtering condition is associated with the regions annotated by the said second facet and in which the first facet is applied only to the data resulting from a cropping by these regions (1300) and for which the condition is verified.

[Claim 7] Method according to claim 6, in which, said annotation generates the definition of a region in said set of data, said region is stored in the database in relation to the region having been used to crop the annotated data ( 1304) and wherein said annotation is stored in said database in relation to said first facet and said region (1305).

[Claim 8] Method according to claim 6, in which said annotation does not create a new region and in which said annotation is stored in said database in relation to said first facet as well as the region used to crop the data annotated (1303).

[Claim 9] A method according to any preceding claim, further comprising the step of displaying said first filtered data to a user, said annotation being received from said user.

[Claim 10] Method according to one of Claims 1 to 8, in which the said first filtered data are supplied as input to an artificial intelligence module implementing the said task, the said annotation being received from the said module.

[Claim 11] Machine learning method, for the execution of a task by an artificial intelligence, comprising the following steps:

- accessing (1401) a database comprising a data set and at least one definition of at least one data selection facet in said data set, said one definition further comprising at least one annotation associated with said facet ,

- applying (1404) said data selection facet to said data set to obtain first filtered data,

- storing (1405) said first filtered data in an annotated training data memory,

- associating (1405) said first filtered data with the annotations - performing (1408, 1409) said task by said artificial intelligence.

[Claim 12] Method according to the preceding claim, in which said annotation is generated according to a method according to one of Claims 1 to 10.

[Claim 13] Apparatus (1500) comprising a processing unit (1501) configured to perform steps according to a method according to any preceding claim.