Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/10—Segmentation; Edge detection
  • G06T7/11—Region-based segmentation
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/20—Image preprocessing
  • G06V10/22—Image preprocessing by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
  • G06V10/235—Image preprocessing by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition based on user input or interaction
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/20—Image preprocessing
  • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/94—Hardware or software architectures specially adapted for image or video understanding
  • G06V10/945—User interactive design; Environments; Toolboxes
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20084—Artificial neural networks [ANN]
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
  • G06V2201/03—Recognition of patterns in medical or anatomical images

Definitions

• the invention relates to a 2D or 3D image segmentation system comprising a neural network, an image segmentation method using this system and a method for training the segmentation system, in particular its neural network.
• the invention is intended in particular for the segmentation of medical images, for example originating from magnetic resonance imaging (MRI) or computed tomography (CT) for the identification of organs on each of the section images obtained by FIRM or CT to recreate a 3D model of the organ from these segmentations.
• MRI magnetic resonance imaging
• CT computed tomography
• Image segmentation is a technique for associating each pixel or voxel of a 2D or 3D image with a label representing the object, texture or state, etc. corresponding to said pixel or voxel.
• Image segmentation is particularly used in a medical context to determine organs, tissues, fluids, etc. visible on a 2D or 3D image.
• Such an image is, for example, obtained by MRI and makes it possible to form a 3D model of the targeted organ and, for example, of its environment and of the tumor to be operated on.
• This creation of a 3D model requires the segmentation of the images obtained by FIRM.
• the semi-automatic method allows an interactivity in which the segmentation is both mostly automatic but with a degree of control by a user, who can in particular act on the initialization and on a modification during certain steps.
• the different segmentation methods can implement methods based on deep learning, in particular by using artificial neural networks.
• Deep learning is particularly used for fully automatic processes, but semi-automatic segmentation solutions implementing deep learning have received limited attention compared to the number of automatic solutions.
• Neural networks have greatly improved the performance of automatic segmentation solutions, but semi-automatic segmentations have not benefited from as many improvements due to difficulties in achieving successful neural network integration.
• the invention aims to provide a system and a method for semi-automatic image segmentation, in particular for medical type images, for example sections obtained by MRI.
• the invention also aims to provide a method for training such a segmentation system.
• the invention aims to provide, in at least one embodiment, a semi-automatic segmentation system that can be easily trained by simulating user interactions.
• the invention aims to provide, in at least one embodiment, a segmentation system whose operation and training allow account of previous segmentation results.
• the invention relates to a semi-automatic image segmentation system comprising a neural network configured to, at each iteration, receive input data comprising at least one image to be segmented and to provide output data comprising at least one segmentation mask of each image, characterized in that it further comprises a user interaction loop comprising a user interaction module configured to provide at least one user mask representative of an interaction of a real user and/or of a simulated user on each segmentation mask, said user interaction loop being configured to provide, as neural network input data: each segmentation mask provided by said neural network coming from of a previous iteration of the neural network, called past segmentation mask, and the user mask associated with each past segmentation mask.
• An image segmentation system therefore makes it possible to take into account user interactions both during training and during use for image segmentation, and to associate these interactions with user to a neural network output segmentation mask as a new neural network input.
• the training data is not obtained only by retrieving a large data set but is generated by the segmentation system itself thanks to the user interaction loop.
• the user mask adapts according to the output data of the network and therefore cannot be prepared in advance. Without this loop, as in the systems of automatic segmentation, the system would only be trained from the images to be segmented and therefore could not, when used, take user interactions into account. Using only an input dataset without the loop would therefore not achieve a satisfactory result, regardless of the volume of this input dataset. In particular, using only an input data set without the loop would make it possible to obtain a system which does not allow modification of the result obtained, such modification being necessary in many fields, in particular in the medical field where a user must validate or edit the result before using it in a therapeutic act.
• using the output segmentation mask as an input helps to introduce sequentiality of data processing by taking past segmentation masks into account in defining better future segmentation.
• Using the output segmentation mask and user interaction as input data helps build up an internal sequential memory that keeps track of past segmentations and user interactions to maintain system sequentiality.
• the sequential internal memory keeps all the user interactions and the previous segmentation masks obtained at the output of the neural network, to provide them as new input to the neural network. Segmentation is also accelerated because sequential user interactions have more impact on the outcome.
• the user interaction loop is used both when training the neural network, when testing the segmentation system from test data, and when using the segmentation system for new data.
• interaction by a real user is meant an interaction by a human user who interacts directly with the segmentation mask proposed as output.
• interaction by a simulated user is meant the simulation of an interaction as a human user could do, defined by a user interaction simulation algorithm.
• the user interaction is for example the assignment, to a specific point of the segmented image, of a particular label.
• the user interaction makes it possible in particular to make a correction to the segmentation mask obtained at the output of the neural network, for example by indicating areas of the 2D or 3D image which have not been correctly labelled. Taking the user into account as soon as the system is trained improves the impact of user corrections when using the system.
• the image segmentation system can carry out the initial segmentation itself when it is trained and takes into account the user corrections for an improvement in the segmentation.
• the initial segmentation can be trivial, in particular little or not informative, or calculated by another method of segmentation, typically a fully automatic method.
• the system can advantageously be used for the segmentation of MRI images but can be applied to other types of 2D or 3D images, medical type or not.
• the system can be integrated into a larger system using the results of the segmentation for a particular application, for example for creating a 3D model from the segmented data and displaying the 3D model in augmented reality on a video.
• the position of the interaction is determined according to the center of a region chosen as being to be corrected by the user interaction.
• this step allows the smoothing of the output data by using a graph cutting algorithm (graph cut in English) to improve the relevance of the labels integrated into the segmentation mask.
• the graph cutting algorithm is applicable for binary segmentation and derived algorithms are used for multi-class segmentation.
• the user interaction loop comprises a sequential internal memory in which is sequentially stored each segmentation mask passed and each user mask obtained at each iteration.
• the sequential internal memory comprises a queue of the "first in-first out” or FIFO (for First In-First Out) type, the queue storing at each iteration the segmentation mask and the user mask corresponding to the iteration.
• FIFO First In-First Out
• the invention also relates to a method for training an image segmentation system according to the invention, via at least one data set comprising images to be segmented, characterized in that it comprises, for each image of the set data: at least two iterations in which the neural network processes the input data to provide output data, a first iteration in which the input data includes at least one image to be segmented and a predefined initialization mask, and subsequent iterations in which the input data includes each image to be segmented, each past segmentation mask obtained in the previous iteration, and a user mask representative of an interaction of a real user and/or a user simulated, a step of updating the weights of the neural network from the output data after each iteration or when a predetermined number of iterations have been performed by the neural network.
• a training method according to the invention makes it possible to simulate the user interactions in order to maximize the effectiveness of the training and the relevance of the neural network at the end of the training.
• the training method according to the invention advantageously uses the user interaction loop of the system to allow the use of a user interaction simulation and output data, i.e. the segmentation mask of a previous iteration, as system input data.
• the image segmentation system makes it possible to improve the user interaction simulation based on quality controls of the segmentation obtained, and to take into account the segmentation masks and past user interactions.
• the training thus allows the reduction of the number of false positives and false negatives by simulating what a user would do during the quality control of the segmentation.
• the training sequence formed by the iteration sequence makes it possible to improve the precision of the corrections and thus to increase the quality of the segmentation at each iteration.
• the updating of the weights is for example implemented by an algorithm for minimizing the loss function, the gradient of which is for example calculated by backpropagation.
• the weight update can be implemented at each iteration, in particular each time output data is produced by the neural network, or only at the end of all the iterations so that the weight update does not occur. only performs from the best segmentation obtained.
• the initialization mask is for example a trivial mask, an empty mask, a mask obtained by an automatic pre-segmentation or a mask derived from the ground truth, for example by simulating user interactions from this ground truth.
• the initialization mask is added to the user mask which indicates the errors between the segmentation mask obtained at the output of the neural network and the result desired by the user.
• Each dataset can include one or more images, different or not. Even if two datasets include the same images, the training process will treat two identical images differently thanks to the simulated user interactions that allow modifications in the training.
• the simulated user interactions are obtained from a ground truth.
• the ground truth is formed by predefined labeled data and from which is determined the error between the segmentation mask obtained at the output of the neural network with the desired result.
• the training is thus supervised thanks to previously segmented data, for which the regions of the image have labels assigned to them.
• Ground truth is more commonly called "ground truth" in English.
• the training is generally carried out only from these labeled data which are not sufficient since the user and the segmentation masks of the iteration previous are not taken into account in the training. Furthermore, the training method according to the invention makes it possible to provide different simulated user interactions for the same image, as would be the case with a real user who will never provide the same interactions for the same image.
• the predetermined number of iterations is determined for each image to be segmented and for each data set independently of the predetermined number of iterations determined for the other images to be segmented and the other data sets.
• the number of iterations is predetermined during the processing of a data set or of an image to be segmented but can be modified over time in order to reproduce the behavior of a real user who will not systematically make the same number of corrections.
• a maximum number of iterations can be set to prevent the training from being too long. Setting the maximum number of iterations also avoids making the network dependent on interactions during the test phase. Since the interactions are based on ground truth, it is preferable to target that a minimum of interactions are provided during the training phase while maximizing performance, so that a minimum of interactions are necessary during the training phase. testing phase thereafter.
• the invention also relates to a semi-automatic image segmentation method comprising at each iteration a step of processing input data comprising each image to be segmented by a neural network to provide output data comprising at least one segmentation mask of each image, characterized in that each iteration further comprises: a step of receiving a user mask representative of an interaction of a real user on each segmentation mask, a step of transmitting, as data of input of the neural network at the next iteration, of each segmentation mask provided by said neural network, and of the user mask
• the image segmentation method allows user interactions to be taken into account as many times as necessary until the user considers that the quality of the image segmentation meets his expectations. If the quality is not sufficient, the user interactions and the segmentation mask are passed as input data to provide a new segmentation mask.
• the image segmentation system according to the invention is configured to implement the image segmentation method according to the invention.
• the image segmentation method according to the invention is implemented by an image segmentation system according to the invention.
• the invention also relates to a computer program product for driving an image segmentation system according to the invention, said computer program product comprising program code instructions for execution, when said program product d computer is executed on a computer, steps of the training method according to the invention.
• the invention also relates to a semi-automatic image segmentation computer program product, said computer program product comprising program code instructions for execution, when said computer program product is executed on a computer, steps of the image segmentation method according to the invention.
• the invention also relates to an automatic segmentation system, an automatic segmentation method, an automatic segmentation computer program product, a training method and a training computer program product characterized in combination by all or part of the characteristics mentioned above or below.
• FIG. 1 is a schematic view of a semi-automatic image segmentation system according to one embodiment of the invention.
• FIG. 2 is a schematic view of a semi-automatic image segmentation method according to one embodiment of the invention.
• FIG. 3 is a schematic representation of the steps of a method for training a semi-automatic image segmentation system according to one embodiment of the invention.
• FIG. 1 schematically represents a semi-automatic image segmentation system 10 according to one embodiment of the invention.
• the segmentation system 10 comprises a neural network 12 configured to, at each iteration, receive input data comprising at least one image 14 to be segmented and to provide output data comprising at least one segmentation mask 16 of each image.
• the segmentation mask defines for each part of the image a label making it possible to identify regions of the image, corresponding to a particular class.
• the segmentation can be binary but the segmentation can also be a multi-class segmentation and makes it possible to identify more than two classes of regions.
• a graph cut can make it possible to smooth the output data to obtain the segmentation mask.
• the segmentation system 10 integrates around an existing machine learning model.
• the system uses an existing encoder-decoder neural network architecture comprising Recurrent Neural Network (RNN) modules.
• RNN Recurrent Neural Network
• One embodiment uses for example a ResNet34 encoder described in the publication “He, K., Zhang, X., Ren, S., Sun, J., 2016. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770-778. and a decoder equipped with a pair of standard convolutional layers and a corresponding Long short-term memory (LSTM) recurrent layer at each step of the upsampling path.
• LSTM Long short-term memory
• the system uses focal loss as described in “Lin, T.Y., Goyal, P., Girshick, R.B., He, K., Dollàr, P., 2017 Focal loss for dense object detection. 2017 IEEE International Conference on Computer Vision (ICCV), 2999-3007. and precomputed weights per class at the scale of the dataset.
• the neural network is for example pre-trained on the ImageNet data set and then refined on a data set dedicated to the desired task.
• a dataset can for example be a dataset comprising medical images of the MRI type of female pelvis manually segmented using manual segmentation software such as 3DSlicer and MITK.
• a female pelvis MRI dataset created by the requestor includes ninety-seven MRI series representing three thousand and sixty-six slices in total. This female pelvis MRI dataset is used for training the neural network, with: seventy-seven MRI series representing two thousand four hundred and forty-nine slices for the training set, ten MRI series representing three hundred and eight slices for the validation set, and ten MRI series representing three hundred and nine slices for the test set.
• the female pelvis MRI dataset slices can be pre-processed via normalization, standardization and bias correction, and random data augmentation can be provided, for example: vertical and horizontal flip, intensity shift for brightness, gamma correction for contrast, as well as blur and unsharp mask for sharpness adjustment, etc.
• examples of labels can be the uterus, the bladder, a possible tumor, cavity, background, etc., forming segmentation classes.
• the image segmentation system 10 differs from the prior art in that it further comprises a user interaction loop 18 comprising a user interaction module 20 configured to provide at least one user mask representative of an interaction of a real user and/or a simulated user on each segmentation mask.
• the user interaction loop 18 is configured to provide, as neural network input data: each segmentation mask 16 provided by said neural network coming from a previous iteration of the neural network 12, called segmentation mask past, as represented by the arrow 22, the user mask associated with each past segmentation mask 16 obtained by the user interaction module 20, as represented by the arrow 24.
• the segmentation mask 16 and the user mask are associated via a state module 26 to form a state 28 formed from the meeting of the segmentation mask and the user mask to form new input data.
• the state 28 is fed by an equivalent empty structure 30, a mask obtained by a pre-segmentation obtained automatically or a mask derived from the ground truth when the system is trained.
• State 28 thus brings together all the segmentation masks and past user masks, thus forming an internal sequential memory for the segmentation system.
• the user interaction loop 18 is external to the neural network and does not depend on a specific neural network architecture, although examples of neural networks that can be used are described previously.
• a classic prior art interaction loop without memory takes user interactions into account by providing the neural network as input with the image to be segmented and the user interactions.
• User interactions are represented by N binary masks, with N the number of segmentation class.
• the input of the neural network is thus concatenated in a tensor of size HxLxC, with H the height of the image, L the width of the image and Cbase the number of channels, which is equal to 1+N in a loop d interaction without memory, corresponding to one channel for the image and N channels for the N segmentation classes.
• THE factor 2 corresponds to the presence of N interaction masks and N probability maps corresponding to the intermediate segmentation results.
• FIG. 2 schematically represents a method 40 of semi-automatic image segmentation according to one embodiment of the invention.
• the segmentation method comprises a step 42 of receiving the images to be segmented in order to supply them to the neural network as described in the segmentation system.
• the method comprises at each iteration a step 44 of processing input data comprising each image to be segmented by the neural network to provide output data comprising at least one segmentation mask for each image.
• a quality control step 46 If the quality of the segmentation is considered correct at a quality control step 46, the segmentation process is terminated and a final step 52 provides the segmentation mask associated with the last segmentation.
• each following iteration further comprises a step 48 of receiving a user mask representative of an interaction of a real user on each segmentation mask, and a step 50 of transmitting , as input data of the neural network at the following iteration, of each segmentation mask provided by said neural network, and of the user mask.
• FIG. 3 schematically represents steps of a method for training a semi-automatic image segmentation system according to one embodiment of the invention
• the training method 100 comprises for each image of a data set, a predetermined number of iterations, each iteration being denoted from Ii to Each iteration includes a step 112 of processing input data 114 by the neural network to provide output data 116.
• the input data includes at least one image to be segmented and a predefined initialization mask 110.
• the predefined initialization mask 110 is here for example an empty mask, a trivial mask, a mask obtained by automatic pre-segmentation or a mask derived from ground truth.
• the input data includes each image to be segmented, each past segmentation mask obtained at the previous iteration, and a user mask representative of an interaction of a real user and/or of a simulated user, as symbolized by the dotted arrows which link the output data 116 of one iteration to the input data 114 of the following iteration.
• the user mask is for example obtained by comparing the segmentation mask with the ground truth, i.e. previously labeled images, allowing supervised learning.
• the user mask thus simulates the interaction of a user who knows the desired result of the segmentation.
• the user mask is for example a binary mask per segmentation class, symbolizing a click by the user in the foreground.
• the position of the click is chosen randomly in the largest region of the class, following a probability map in which the maximum is at the center of the region, decreasing away from the center and very reduced outside the region.
• the training therefore makes it possible to fill the memory of D states with realistic values produced by a virtual user.
• the D states are formed by D iterations to fill the internal sequential memory before updating the weights.
• the states are preferably formed by inference via the neural network, i.e. without updating the weights of the neural network.
• the method also includes, after each iteration or when the predetermined number of iterations have been performed by the neural network, a step 120 of updating the weights of the neural network from the output data.
• a step 120 of updating the weights of the neural network is carried out at the last iteration, on the basis of the last output data 116 .
• the predetermined interaction number is determined independently for each class and for each image set.
• the maximum number of interactions is typically three interactions to simulate a typical user, the minimum being zero.
• the probability of adding an additional interaction decreases linearly after each interaction to reach zero when the maximum number of interactions is reached.
• a class is randomly chosen at each epoch in English) of the training method with a dataset, for which no user interaction is generated.
• a percentage of the interactions generated is retained, typically 20%.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Multimedia (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Evolutionary Computation (AREA)
• Software Systems (AREA)
• Artificial Intelligence (AREA)
• Computing Systems (AREA)
• Databases & Information Systems (AREA)
• General Health & Medical Sciences (AREA)
• Medical Informatics (AREA)
• Health & Medical Sciences (AREA)
• Image Analysis (AREA)
• User Interface Of Digital Computer (AREA)
• Image Processing (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=81448340

Family Applications (1)

Country Status (6)

Families Citing this family (1)

• 2022
  • 2022-02-28 FR FR2201763A patent/FR3133099B1/fr active Active
• 2023
  • 2023-02-28 WO PCT/EP2023/055035 patent/WO2023161531A1/fr not_active Ceased
  • 2023-02-28 CA CA3244125A patent/CA3244125A1/fr active Pending
  • 2023-02-28 JP JP2024550563A patent/JP2025507706A/ja active Pending
  • 2023-02-28 CN CN202380024102.4A patent/CN119053990A/zh active Pending
  • 2023-02-28 EP EP23707400.0A patent/EP4487293A1/de active Pending

Also Published As

Similar Documents

Legal Events