EP4566025A1 - Method of tracing an elongate object in image data and image processing system - Google Patents
Method of tracing an elongate object in image data and image processing systemInfo
- Publication number
- EP4566025A1 EP4566025A1 EP23750634.0A EP23750634A EP4566025A1 EP 4566025 A1 EP4566025 A1 EP 4566025A1 EP 23750634 A EP23750634 A EP 23750634A EP 4566025 A1 EP4566025 A1 EP 4566025A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- model
- image data
- volume
- elongate object
- sub
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/60—Rotation of whole images or parts thereof
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10072—Tomographic images
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20004—Adaptive image processing
- G06T2207/20012—Locally adaptive
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20092—Interactive image processing based on input by user
- G06T2207/20101—Interactive definition of point of interest, landmark or seed
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30016—Brain
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30101—Blood vessel; Artery; Vein; Vascular
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30172—Centreline of tubular or elongated structure
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
Definitions
- Embodiments of the invention relate to methods and systems for tracing an elongate object, such as a neurite, in image data representative of a three-dimensional tissue volume. Background Dense mapping of neuronal wires and their synaptic connections for enhanced understanding of computation in the brain is a major goal of high-resolution connectomics.
- Wolrterink et al. "Coronary artery centerline extraction in cardiac CT angiography using a CNN-based orientation classifier", MEDICAL IMAGE ANALYSIS, OXFORD UNIVERSITY PRESS, OXOFRD, GB, vol.
- a method of tracing an elongate object in image data may comprise determining, by an image processing system, positions of a series of points along the elongate object in three-dimensional, 3D, space to trace the elongate object in 3D in the image data.
- the series of points may be determined such that they are located along an approximation of a center line of the elongate object. Determining the positions of the series of points may comprise an iterative process.
- the iterative process may comprise inferring, using an artificial intelligence (AI) model, a steering prediction that is dependent on (and which may be indicative of) a direction along which the elongate object (e.g., a center line or approximation of the center line) extends from a previously determined point along the elongate object (e.g., along the center line or an approximation of the center line), the AI model comprising an input layer operative to receive an AI model input that is based on pixel or voxel values of the image data and an output layer operative to provide, in each iteration of the iterative process, an AI model output that is or indicates the steering prediction.
- AI artificial intelligence
- the iterative process may comprise computing, using at least the steering prediction, a next point of the series of points, the next point following the previously determined point in the series of points.
- the iterative process may comprise iteratively repeating the inferring and computing steps until a termination criterion is fulfilled, to thereby determine the series of points.
- the AI model may comprise a convolutional neural network (CNN) comprising the input layer, a plurality of convolutional layers, a dropout layer or several dropout layers, and the output layer, or a recurrent neural network (RNN), wherein the AI model input that is input to the input layer in an iteration of the iterative process represent a sub-volume of a tissue volume represented by the image data, wherein the AI model input comprises pixel or voxel values representing the sub-volume of the image data or is obtained by multivariate interpolation of pixel or voxel values representing the sub- volume of the image data, wherein an orientation of the sub-volume is dependent on the steering prediction determined in the preceding iteration of the iterative process.
- CNN convolutional neural network
- RNN recurrent neural network
- the steering prediction may be dependent on a second derivative of the elongate object, e.g., of a center line of the elongate object.
- the AI model may be a convolutional neural network (CNN) comprising the input layer, a plurality of convolutional layers, a dropout layer or several dropout layers, and the output layer, or a recurrent neural network (RNN), wherein the AI model input that is input to the input layer in an iteration of the iterative process represent a sub-volume of a tissue volume represented by the image data, the steering prediction being curvature values relative to a reference frame given in relation to the sub-volume such that it yields a position offset for determining in an iterative manner a series of points located on a center line of the elongate object, and in each iteration following an immediately preceding iteration, the respective subvolume that is processed is determined such as to have a rotational orientation that is determined by the direction of extension of the AI model
- the method may comprise outputting a result of the image processing via a graphical user interface or via a data interface. Determining the next point of the series of points may comprise integrating the steering prediction.
- the steering prediction may be dependent on a second derivative of the center line of the elongate object.
- the AI model input that is input to the input layer in an iteration of the iterative process may represent a sub-volume of a tissue volume represented by the image data.
- the sub-volume may be dependent on a preceding steering prediction determined in a preceding iteration of the iterative process and/or the point that has been determined in the preceding iteration of the iterative process.
- An orientation of the sub-volume may be dependent on the preceding steering prediction and/or may be dependent on uncertainties of predictions of the AI model over multiple orientations.
- the orientation of the sub-volume may depend on a direction along which the elongate object extends at or within the sub-volume.
- the orientation of the sub-volume may be approximately aligned with a direction along which the elongate object extends at or within the sub-volume.
- a center of the sub-volume may be shifted relative to the previously determined point by an offset, further optionally wherein the offset is dependent on the steering prediction determined in the preceding iteration.
- the method may comprise multivariate interpolation (such as averaging) the pixel or voxel values of the image data to determine the AI model input.
- Multivariate interpolation may comprise weighted averaging of the pixel or voxel values to generate the AI model input as interpolated pixel or voxel values on a grid that is tilted and/or offset relative to a pixel or voxel grid of the image data.
- Multivariate interpolation may comprise a trilinear projection.
- the method may comprise performing an initialization of the iterative procedure. The initialization may be dependent on whether an initial orientation is available a priori. When the initial orientation is not available, the method may comprise determining the initial orientation for use in an initial iteration of the iterative process.
- Determining the initial orientation may comprise determining uncertainty estimates for the AI model predictions for various orientations and choosing the orientation from among the various orientations for which the uncertainty estimate is minimum.
- the uncertainty estimates may be determined using Monte-Carlo dropout.
- the AI model may comprise a convolutional neural network (CNN).
- the AI model may comprise a recurrent neural network (RNN).
- the AI model may comprise the input layer, a plurality of convolutional layers, a dropout layer or several dropout layers, and the output layer.
- the output layer may be a linear layer.
- the plurality of convolutional layers may comprise one or several strided convolutional layers.
- the AI model may be or may comprise a fully convolutional network.
- the AI model may comprise one or several of pooling layers, attention modules, self- attention modules, atrous convolutions, normalization layers, batch normalization layers, transformers, recurrent layers.
- the AI model may comprise one or several nonlinearities.
- the AI model may comprise exponential linear units (ELU).
- the AI model may comprise rectified linear units (ReLU).
- the AI model may comprise Scaled Exponential Linear units (SELU).
- SELU Scaled Exponential Linear units
- SELU Scaled Exponential Linear units
- the AI model may comprise Gaussian Error Linear Unit (GELU).
- the image data may be representative of a volumetric representation of tissue.
- the image data may comprise a three-dimensional (3D) image or a set of two- dimensional (2D) images.
- the method further may comprise acquiring the image data.
- the image data may be acquired using electron microscopy.
- the method may comprise training the AI model.
- the AI model may be trained using supervised learning. Training the AI model may comprise gradient-based updates of AI model parameters. Training the AI model may comprise training the AI model to comply with a flight policy that causes the iterative process to converge to the elongate object (e.g., along the center line or an approximation of the center line) when starting from an initial location in the image data that is off the center line.
- a convergence distance of the flight policy may determine how rapidly the iterative process converges back to the center line.
- the convergence distance may be a dynamic parameter that is varied during AI model training and/or inference.
- the convergence distance may be set as a function of a distance from a physiological boundary during AI model training.
- the convergence distance may be a monotonously increasing function of the distance from the physiological boundary.
- the method may comprise outputting one or several of a visual representation of the series of points or of the center line; a machine-readable representation of the series of points or of the center line; and a control signal that is dependent on the series of points located along the center line; and uncertainties of the predicted control signal; and a representation of which stop criterion/criteria ended the iterative process.
- the outputting may be performed via a user interface, such as a graphical user interface, or via a data interface.
- a medical imaging method or a medical image processing method comprises receiving medical image data representing a tissue volume and comprising a 3D image or a set of 2D images of tissue comprising neural tissue, performing the image processing method to trace an elongate object in the medical image data, and outputting one or several of a visual representation of the series of points or of the center line; a machine- readable representation of the series of points or of the center line; a control signal that is dependent on the series of points located along the center line; a machine learning (ML) model that has an architecture that depends on the series of points located along the center line.
- a connectomics method comprises performing the method of tracing an elongate object.
- a method of analyzing a bioptic sample may comprise acquiring image data of the bioptic sample, processing the image data using the method of processing image data, and outputting a result of the image data processing via an interface.
- the method of analyzing the bioptic sample may be performed to determine part of a connectome.
- the bioptic sample is a sample that is not being implanted into an animal or human body.
- a method of generating a signal processing logic for controlling a physical asset may comprise performing the method of processing image data, using, by at least one computing system, the output of the elongate object tracing method to select or generate a machine learning, ML, model architecture, train an ML model having the ML model architecture, deploy the trained ML model as the signal processing logic to a control device for execution.
- An image processing system for tracing an elongate object in image data may comprise an interface operative to receive the image data, the image data being biological or medical image data.
- the image processing system may comprise at least one processing device operative to determine positions of a series of points along an elongate object (e.g., along the center line or an approximation of the center line) in three-dimensional, 3D, space to trace the elongate object in 3D in the image data, wherein determining the positions of the series of points may comprise an iterative process.
- the iterative process may comprise inferring, using an artificial intelligence (AI) model, a steering prediction that is dependent on (and which may be indicative of) a direction along which the elongate object extends from a previously determined point along the elongate object, the AI model comprising an input layer operative to receive an AI model input that is based on pixel or voxel values of the image data and an output layer operative to provide, in each iteration of the iterative process, an AI model output that is or indicates the steering prediction.
- the iterative process may comprise computing, using the steering prediction, a next point of the series of points, the next point following the previously determined point in the series of points.
- the iterative process may comprise iteratively repeating the inferring and computing steps until a termination criterion is fulfilled, to thereby determine the series of points.
- the image processing system may be operative such that the AI model may comprise a convolutional neural network (CNN) comprising the input layer, a plurality of convolutional layers, a dropout layer or several dropout layers, and the output layer, or a recurrent neural network (RNN), wherein the AI model input that is input to the input layer in an iteration of the iterative process represent a sub-volume of a tissue volume represented by the image data, wherein the AI model input comprises pixel or voxel values representing the sub-volume of the image data or is obtained by multivariate interpolation of pixel or voxel values representing the sub-volume of the image data, wherein an orientation of the sub-volume is dependent on the steering prediction determined in the preceding iteration of the iterative process.
- CNN convolutional neural network
- RNN recurrent neural
- the steering prediction may be dependent on a second derivative of the elongate object, e.g., of a center line of the elongate object.
- the image processing system may be operative such that the AI model may be a convolutional neural network (CNN) comprising the input layer, a plurality of convolutional layers, a dropout layer or several dropout layers, and the output layer, or a recurrent neural network (RNN), wherein the AI model input that is input to the input layer in an iteration of the iterative process represent a sub-volume of a tissue volume represented by the image data, the steering prediction being curvature values relative to a reference frame given in relation to the sub-volume such that it yields a position offset for determining in an iterative manner a series of points located on a center line of the elongate object, and in each iteration following an immediately preceding iteration, the respective subvolume that is processed is determined such as to have a rotational orientation that
- the image processing system may be operative to perform the method according to an embodiment.
- the image processing system may comprise an image acquisition system operative to capture the image data.
- the image acquisition system may be operative to perform 3D electron microscopy or light microscopy.
- the image processing system may be operative to output a result of the image processing via a graphical user interface or via a data interface.
- the image processing system may be operative such that determining the next point of the series of points may comprise integrating the steering prediction.
- the image processing system may be operative such that the AI model input that is input to the input layer in an iteration of the iterative process may represent a sub-volume of a tissue volume represented by the image data.
- the image processing system may be operative such that the sub-volume may be dependent on a preceding steering prediction determined in a preceding iteration of the iterative process and/or the point that has been determined in the preceding iteration of the iterative process.
- the image processing system may be operative such that an orientation of the sub- volume may be dependent on the preceding steering prediction.
- the image processing system may be operative such that the orientation of the sub- volume may depend on a direction along which the elongate object extends at or within the sub-volume.
- the image processing system may be operative such that the orientation of the sub- volume may be approximately aligned with a direction along which the elongate object extends at or within the sub-volume.
- the image processing system may be operative such that center of the sub-volume may be shifted relative to the previously determined point by an offset.
- the image processing system may be operative such that the offset may be dependent on the steering prediction determined in the preceding iteration.
- the image processing system may be for a multivariate interpolation (such as averaging) the pixel or voxel values of the image data to determine the AI model input.
- the image processing system may be operative such that the multivariate interpolation may comprise weighted averaging of the pixel or voxel values to generate the AI model input as interpolated pixel or voxel values on a grid that is tilted and/or offset relative to a pixel or voxel grid of the image data.
- the image processing system may be operative to perform projections to determine the pixel or voxel values on a grid that is tilted and/or offset relative to a pixel or voxel grid of the image data.
- the image processing system may be operative to perform an initialization of the iterative procedure.
- the image processing system may be operative such that the initialization may be dependent on whether an initial steering prediction is available a priori.
- the image processing system may be operative such that, when the initial orientation is not available, the image processing system determines the initial orientation for use in an initial iteration of the iterative process.
- the image processing system may be operative such that determining the initial orientation may comprise determining uncertainty estimates for various orientations and choosing the orientation from among the various orientations for which the uncertainty estimate is minimum.
- the image processing system may be operative such that the uncertainty estimates may be determined using Monte-Carlo dropout.
- the AI model may comprise a convolutional neural network (CNN).
- the AI model may comprise a recurrent neural network (RNN).
- the AI model may comprise the input layer, a plurality of convolutional layers, a dropout layer or several dropout layers, and the output layer.
- the output layer may be a linear layer.
- the plurality of convolutional layers may comprise one or several strided convolutional layers.
- the AI model may be or may comprise a fully convolutional network.
- the AI model may comprise one or several of pooling layers, attention modules, self- attention modules, atrous convolutions, normalization layers, batch normalization layers, transformers, recurrent layers.
- the AI model may comprise exponential linear units (ELU).
- the AI model may comprise rectified linear units (ReLU).
- the AI model may comprise Scaled Exponential Linear units (SELU).
- the AI model may comprise Gaussian Error Linear Unit (GELU).
- the image data may be representative of a volumetric representation of tissue.
- the image data may comprise a 3D image or a set of 2D images.
- the image processing system may be operative to acquire the image data.
- the image data may be acquired using electron microscopy.
- the image data may be acquired using optical microscopy, e.g., light microscopy.
- the image data may be acquired using positron emission tomography (PET), magnetic resonance (MR) imaging, or another medical imaging modality.
- the elongate object may be a neurite.
- the elongate object may be a blood vessel.
- the elongate object may have a surface that extends around the center line in a locally cylindrical manner.
- the image processing system may be operative to train the AI model.
- the AI model may be trained using supervised learning.
- the image processing system may be operative such that training the AI model may comprise gradient-based updates of AI model parameters.
- the image processing system may be operative such that training the AI model may comprise training the AI model to comply with a flight policy that causes the iterative process to converge to the center line when starting from an initial location in the image data that is off the center line.
- the image processing system may be operative such that a convergence distance of the flight policy may determine how rapidly the iterative process converges back to the center line.
- the image processing system may be operative such that the convergence distance may be a dynamic parameter that is varied during AI model training.
- the image processing system may be operative such that the convergence distance may be set as a function of a distance from a physiological boundary during AI model training.
- the image processing system may be operative such that the convergence distance may be a monotonously increasing function of the distance from the physiological boundary.
- the image processing system may be operative to output one or several of a visual representation of the series of points or of the center line; a machine-readable representation of the series of points or of the center line; and a control signal that is dependent on the series of points located along the center line.
- the image processing system may comprise a user interface, such as a graphical user interface, or a data interface to perform the outputting.
- a training method of training an AI model comprising an input layer operative to receive an AI model input that is based on pixel or voxel values of image data representing a tissue volume and an output layer operative to provide an AI model output that is or indicates a steering prediction, the steering prediction being dependent on (and may be indicative of) a direction along which an elongate object (e.g., along the center line or an approximation of the center line) extends.
- the training method may comprise training the AI model with an objective of predicting the direction along which the center line of an elongate object extends.
- the training method may comprise training the AI model with an objective of outputting a derivative (e.g., a first order, second order, or higher order derivative) of the direction along which the center line of an elongate object extends.
- the training method may be a gradient-based supervised learning method.
- the AI model may be trained using supervised learning.
- the training method may comprise gradient-based updates of AI model parameters.
- the training method may comprise training the AI model to comply with a flight policy that causes the iterative process to converge to the center line when starting from or reaching a location in the image data that is off the center line. A convergence distance of the flight policy may determine how rapidly the iterative process converges back to the center line.
- the convergence distance may be a dynamic parameter that is varied during AI model training and/or inference.
- the convergence distance may be dependent on a distance from an obstacle.
- the obstacle may be a physiological boundary, such as a cell boundary.
- the convergence distance may be set as a function of a distance from a physiological boundary during AI model training.
- the convergence distance may be a monotonously increasing function of the distance from the physiological boundary.
- An AI model input that is input to the input layer may represent a sub-volume of a tissue volume represented by training data.
- the training method may comprise a multivariate interpolation of the pixel or voxel values of the image data to determine the AI model input.
- the multivariate interpolation may comprise weighted averaging of the pixel or voxel values to generate the AI model input as interpolated pixel or voxel values on a grid that is tilted and/or offset relative to a pixel or voxel grid of the image data.
- the AI model may comprise a convolutional neural network (CNN).
- the AI model may comprise a recurrent neural network (RNN).
- RNN recurrent neural network
- the AI model may comprise the input layer, a plurality of convolutional layers, a dropout layer or several dropout layers, and the output layer.
- the output layer may be a linear layer.
- the plurality of convolutional layers may comprise one or several strided convolutional layers.
- the AI model may be or may comprise a fully convolutional network.
- the AI model may comprise one or several of pooling layers, attention modules, self-attention modules, atrous convolutions, normalization layers, batch normalization layers, transformers, recurrent layers.
- the AI model may comprise one or several nonlinearities.
- the AI model may comprise exponential linear units (ELU).
- the AI model may comprise rectified linear units (ReLU).
- the AI model may comprise Scaled Exponential Linear units (SELU).
- the AI model may comprise Gaussian Error Linear Unit (GELU).
- the image data may be representative of a volumetric representation of tissue.
- the image data may comprise a 3D image or a set of 2D images.
- the image data may be electron microscopy (EM) images.
- the image data may be 3D EM images.
- the image data may be X-ray tomography images.
- the image data may be optical microscopy, e.g., light microscopy data.
- the image data may be positron emission tomography (PET), magnetic resonance (MR) imaging, or other medical image data.
- PET positron emission tomography
- MR magnetic resonance
- the elongate object on which the AI model is trained may be a neurite.
- the elongate object on which the AI model is trained may be a blood vessel.
- the elongate object on which the AI model is trained may have a surface that extends around the center line in a locally cylindrical manner.
- a method of tracing an object in image data comprises determining a series of points along the object to trace the object in the image data. Determining a series of points may comprise an iterative process.
- the iterative process may comprise: inferring, using an artificial intelligence (AI) model, a steering prediction, the AI model comprising an input layer operative to receive an AI model input that is based on pixel or voxel values of the image data and an output layer operative to provide, in each iteration of the iterative process, an AI model output that is or indicates the steering prediction; computing, using at least the steering prediction, a next point of the series of points, the next point following the previously determined point in the series of points; and iteratively repeating the inferring and computing steps until a termination criterion is fulfilled.
- the image data may be N-dimensional image data, with N being an integer. N may be at least 2, at least 3, or at least 4.
- the object may be traced in N-D dimensions, where D is an integer equal to or greater than zero and less than N.
- One of the dimensions of the image data may correspond to a time axis.
- Determining the next point of the series of points may comprise integrating the steering prediction.
- Integrating the steering prediction may comprise integrating curvatures in a Bishop reference frame.
- the steering prediction is related to a second derivative of the center line of the object.
- the AI model input that is input to the input layer in an iteration of the iterative process may represent a sub-volume of the image data.
- the sub-volume may be dependent on the position and/or orientation and/or steering prediction determined in one or several preceding iterations of the iterative process.
- An orientation of the sub-volume may be dependent on the preceding steering prediction.
- a center of the sub-volume may be shifted relative to the previously determined point by an offset.
- the offset may be dependent on the steering prediction determined in the preceding iteration.
- the method may further comprise a multivariate interpolation of the pixel or voxel values of the image data to determine the AI model input.
- the multivariate interpolation may comprise weighted averaging of the pixel or voxel values to generate the AI model input as interpolated pixel or voxel values on a grid that is tilted and/or offset relative to a pixel or voxel grid of the image data.
- the multivariate interpolation may comprise a multilinear projection.
- the method may further comprise performing an initialization of the iterative procedure.
- the initialization may be dependent on whether an initial direction is available.
- the method may comprise determining the initial direction for use in an initial iteration of the iterative process. Determining the initial direction may comprise determining uncertainty estimates of predictions for various directions. Determining the initial direction may comprise the direction and steering prediction from among the various directions for which the uncertainty estimate in the corresponding prediction is minimum., The uncertainty estimates may be determined using Monte-Carlo dropout.
- the AI model may comprise a convolutional neural network, CNN, or a recurrent neural network, RNN.
- the AI model may comprise the input layer, a plurality of convolutional layers, a dropout layer or several dropout layers, and the output layer.
- the AI model may comprise one or several of pooling layers, attention modules, self- attention modules, atrous convolutions, normalization layers, batch normalization layers, transformers, recurrent layers.
- the output layer is a linear layer
- the plurality of convolutional layers comprises one or several strided convolutional layers
- the AI model is or comprises a fully convolutional network
- the AI model comprises one or several nonlinearities
- the AI model comprises exponential linear units (ELU);
- the AI model comprises rectified linear units (ReLU);
- the AI model comprises Scaled Exponential Linear units (SELU);
- the AI model comprises Gaussian Error Linear Unit (GELU).
- the object may be an object extending along a time axis of the image data.
- the object may be an object in a video.
- the object may be an object that is to be traced through a series of image frames.
- the object may be a biological, in particular physiological, object.
- the object may be a human or animal neurite.
- the object may be a human or animal blood vessel.
- the object may be an asset of an infrastructure, in particular utility, system.
- the object may be a tube of a fresh water or sewage water system.
- the object may be a tubing of an industrial plant.
- the object may have a surface that extends around a center axis of the object in a locally cylindrical manner.
- the method may further comprise training the AI model.
- Training the AI model may comprise training the AI model to comply with a flight policy that causes the iterative process to converge to the object when starting from or reaching a location in the image data that is off the object.
- a convergence distance of the flight policy may determine how rapidly the iterative process converges back to the object.
- the convergence distance may be a dynamic parameter that may be varied during AI model training.
- the convergence distance may be set as a function of a distance from a boundary represented by the image data during AI model training.
- the convergence distance may be a monotonously increasing function of the distance from the boundary.
- An image processing method may comprise receiving and/or acquiring image data and performing the object tracing method.
- the image processing method may comprise outputting one or several of: a visual representation of the series of points or of the center line; a machine-readable representation of the series of points or of the center line; a control signal that is dependent on the series of points located along the center line; a machine learning, ML, model that has an architecture that depends on the series of points located along the center line.
- the image data may comprise a time series of time-sequential image frames. Tracing the object may comprise tracing a physical object along a time axis in a spatiotemporal coordinate system.
- a control method comprises the object tracing method or image processing method, and further comprises controlling at least one controllable asset based on a result of the object tracing method.
- a processing system that comprises an interface to receive image data and at least one circuit (e.g., a set of processors or other integrated circuits) operative to perform the object tracing method or image processing method.
- a system that comprises at least one controllable asset (e.g., a machine or actuator) and the processing system, wherein the controllable asset is responsive to a control command or control signal generated by the processing system.
- the methods and systems allow neurites to be traced with an accuracy that is comparable to or outperforms conventional automatic or semi-automatic techniques and that approaches that of human annotators.
- the methods and systems are applicable not only to dendrite tracing but can also trace, e.g., axons and spine necks, which are typically harder to trace than dendrites.
- image processing methods and image processing systems operative to perform a local realignment may comprise the following steps executed to perform the local realignment of slices of image data along a slicing direction: determining slice-to-slice shift vectors for a plurality of slices along the slicing direction; identifying valid slice-to-slice shift vectors between slices; using the valid slice-to-slice shift vectors to generate a locally realigned subvolume.
- the locally realigned subvolume may be fed to the AI model input, e.g., to an input layer of a CNN or RNN or to a filter or other preprocessing that preprocesses the realigned subvolume prior to it being processed by the CNN or RNN.
- Determining the slice-to-slice shift vectors may comprise determining auto- and cross- correlation of slices. Determining the slice-to-slice shift vectors and/or their validity may comprise determining a mean and a standard deviation of correlation-weighted shift vectors based on a first threshold criterion. Identifying valid slice-to-slice shift vectors may comprise verifying that the determined auto-correlations fulfill a first criterion (such as having a peak at a desired shift, e.g., at zero shift).
- Identifying valid slice-to-slice shift vectors may comprise verifying that a standard deviation of the shift vector fulfills a third cross-correlation criterion (such as the standard deviation being equal to or less than a fifth threshold).
- Using the valid shift vectors may comprise identifying one or several slices to be realigned even though the slice-to-slice shift vector is not considered valid, and computing the shift vector in this case from valid shift vectors relating to a neighborhood of these one or several slices.
- Using the valid shift vectors may comprise generating a locally realigned and trilinearly interpolated subvolume. Generating the locally realigned and trilinearly interpolated subvolume may comprise performing floor or ceil operations.
- Generating the locally realigned and trilinearly interpolated subvolume may comprise performing a trilinear interpolation as a sequence of linear interpolations along cardinal axes.
- the sequence of linear interpolations may comprise four linear interpolations along a first dimension, two linear interpolations along a second dimension, and one linear interpolation along a third dimension.
- the locally realigned subvolume may be processed to trace an elongate object, such as a blood vessel or neurite, in 3D.
- the local realignment techniques may be used in association with (e.g., as a part of) the methods of tracing an elongate object, without being limited thereto. These local realignment techniques address for referencing slices to a common reference frame in a particularly efficient manner.
- Figure 1 is a block diagram representation of a system according to an embodiment.
- Figure 2 shows an elongate object in 3D image data.
- Figure 3 shows an elongate object in 3D image data.
- Figure 4 is a view of a series of points for explaining operation of systems and methods according to embodiments.
- Figure 5 is a flow chart of a method according to an embodiment.
- Figure 6 is a view of sub-volumes of an imaged tissue volume for explaining operation of systems and methods according to embodiments.
- Figure 7 is a view of voxel grids.
- Figure 8 is a view illustrating determination of an artificial intelligence (AI) model input.
- AI artificial intelligence
- Figure 9 shows an AI model of systems and methods according to embodiments.
- Figure 10 shows an AI model of systems and methods according to embodiments.
- Figure 11 is a flow chart of a method according to an embodiment.
- Figure 12 is a flow chart of a method according to an embodiment.
- Figure 13 shows an AI model of systems and methods according to embodiments and illustrates operation thereof.
- Figure 14 shows a further AI model of systems and methods according to embodiments and illustrates operation thereof.
- Figure 15 is a flow chart of a method according to an embodiment.
- Figure 16 is a view illustrating a flight policy that may be employed during AI model training.
- Figure 17 is a flow chart of a method according to an embodiment.
- Figure 18 is a flow chart of a method according to an embodiment.
- Figure 19 illustrates operation of systems and methods according to embodiments.
- Figure 20 illustrates a local coordinate frame that may be employed by methods and systems according to embodiments.
- Figure 21 shows an error rate of methods and systems according to embodiments as a function of training iterations.
- Figure 22 is a flow chart of a method according to an embodiment.
- Figure 23 is a diagram showing the applications to which embodiments of the invention lend themselves due to their low reconstruction error rate and large distance between reconstruction errors.
- Figure 24 is a diagram showing the improvements attained by embodiments as compared to conventional techniques.
- Figure 25 is a diagram showing the improvements attained by embodiments as compared to conventional techniques.
- Figure 26 is a further diagram showing the improvements attained by embodiments as compared to conventional techniques.
- Figure 27 is a flow chart of a method according to an embodiment.
- Figure 28 is a block diagram representation of a system according to an embodiment.
- the systems and methods disclosed herein are operative to trace an elongate object (such as a neurite, without being limited thereto) in image data.
- the image data may be 3D image data or may be image data in a spatio-temporal coordinate system that has the time axis as one of its axes.
- the image data may be biological or medical image data.
- the image data may include a series of frames that may be taken time-sequentially.
- the techniques disclosed herein are applicable to other image data as well.
- the systems and methods can determine a series of points that are predicted to lie on a center line of the elongate object, to thereby trace the object.
- the systems and methods may be operative to process voxel or pixel values of image data, without requiring prior object recognition in the image data.
- the systems and methods disclosed herein may perform object tracing by (a) selecting a sub-volume of the image data, (b) determining information on a direction along which the elongate object extends in the sub-volume, and (c) processing the direction to determine a further point located along the center axis. These steps may be repeated iteratively.
- the sub-volume that is processed in an iteration may have an orientation that depends on the direction of extension of the object determined in a preceding iteration.
- the direction steers the analysis process, akin to autonomous driving or flight techniques.
- the direction along which the center axis of the elongate object extends is therefore also referred to as “flight direction” herein.
- This direction determines the path along which the techniques disclosed herein operate.
- Changes in the flight direction are determined at least by a steering prediction.
- the steering prediction may define or otherwise relate to a derivative (e.g., a first or second or higher order derivative) of the flight direction.
- steering prediction refers to a quantity that depends on how the elongate object extends from a previously determined point along the elongate object and/or that determines how (in particular with which sub-volume orientation and/or location) the iterative technique disclosed herein is to proceed.
- An orientation of a sub-volume that is analyzed next may be set in dependence on the flight direction and/or steering prediction.
- the determination of the steering prediction may use an AI model.
- the AI model may have an input layer operative to receive voxel values determined from the image data.
- the AI model input may be taken from the image data or, more generally, may be obtained by interpolating adjacent voxels of the image data to accommodate arbitrary positions and orientations of the sub-volume.
- the AI model may have an output layer operative to output the steering prediction.
- the AI model may be an AI model that has been trained not only on positions located along the center line of elongate objects in training image data, but also on positions located off the center line of elongate objects in the training image data. Convergence back to the center line can thereby be ensured also during inference.
- the determination of the new point along the center line may be performed using an integration of the steering prediction.
- the integration may comprise several subsequent integrations, e.g., when the steering prediction relates to a higher order derivative of the center line direction of the elongate object.
- the systems and methods are independent of the affinity prediction of individual voxels and do not require an explicit volume segmentation of the image data.
- the systems and methods are operative to directly solve the problem of following the center line of elongate objects such as neuronal wires.
- the systems and methods facilitate large-scale connectomic reconstruction. Since they are also computationally efficient, resource consumption for acquiring connectomes from animals’ and human brains, in health and disease, for screening of circuit disorders and bioptic analysis of pathological tissue in the clinical setting are facilitated.
- the systems and methods provide an approach for automated axon reconstruction in 3D EM data by training an AI model on mimicking axon flight mode tracing.
- the systems and methods may employ a convolutional neural network (CNN). Details of CNNs are described in, e.g., Y. LeCun et al.1989.
- the systems and methods provide various differences and effects as compared to the CNN-based techniques of M. Januszewski et al. 2018. “High-Precision Automated Reconstruction of Neurons with Flood-Filling Networks.” Nature Methods. https://doi.org/10.1038/s41592-018-0049-4.
- the systems and methods determine the (approximate) center line of elongate objects such as neurites and through iterations skeleton-reconstructs the center line.
- the sub-volume selection (such as sub-volume orientation) and problem formulation ensure the analysis to be aligned to the elongate axis.
- FFN is restricted to integer coordinates
- the techniques disclosed herein afford multivariate interpolation of image data and a continuous problem formulation in terms of a spatially varying coordinate frame, which is more readily applicable to continuous coordinates for input and/or output.
- the AI model is trained on the task of predicting the local continuation of the center line from, e.g., a neurite-centered and -aligned 3D-EM sub- volume. This allows for direct optimization of the error rate, a metric on which conventional automated methods have made little progress and a metric, which exceeds voxel-based metrics in relevance for the extraction of connectomes from 3D-EM volumes that are much larger than typical neurite diameters.
- FIG. 1 is a block diagram of a system 10.
- the system 10 is operative to trace an elongate object (such as a neurite) in 3D image data (such as 3D-EM image data).
- the system comprises an imaging system 20.
- the imaging system 20 may be a 3D EM imaging system.
- the imaging system 20 may comprise a source 21.
- the source 21 may comprise an electron source that may be operative to output a beam of electrons, for example.
- the imaging system 20 may comprise a detector 22.
- the detector 22 may be operative to detect electrons that have been scattered, reflected, or refracted off physiological objects within a sample (such as a biopsy sample).
- the imaging system 20 may comprise an image alignment component 23 for image alignment prior to reconstruction.
- the imaging system 20 may be operative to perform a local realignment.
- the local realignment may be performed by image alignment component 23 or the imaging system 20 may be operative to execute a second, local image realignment component in addition to the image alignment component 23.
- the imaging system 20 (and in particular its local image realignment component) may be operative to perform a local re-alignment and de-alignment for mapping to a global image coordinate system.
- the imaging system 20 may be operative to perform the method of P.H. Li et al., "Automated Reconstruction of a Serial- Section EM Drosophila Brain with Flood-Filling Networks and Local Realignment.” Preprint. https://doi.org/10.1101/605634, in particular the techniques described in association with Fig.3 of this preprint.
- the local realignment may be performed using techniques as specifically described in greater detail hereinbelow.
- the imaging system 20 may comprise an image reconstruction circuit 24.
- the image reconstruction circuit 24 may be operative to compute voxels representing 3D image data of the sample.
- the 3D image data 40 may be provided to image processing system 30.
- other forms of a volumetric tissue representation may be used, such as a series of two-dimensional (2D) images or projections.
- An example for such a representation is X-ray holographic nano-tomography (which may be performed using, e.g., the techniques of A.T. Kuan et al., "Dense neuronal reconstruction through X-ray holographic nano-tomography", Nature Neuroscience volume 23, pages 1637–1643 (2020).
- the image processing system 30 comprises one or several processing circuit(s) 31.
- the processing circuit(s) 31 may include one or several circuits, including integrated circuits such as field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), controller(s), processor(s), superconducting circuit(s), quantum bit (qubit) processing circuits, without being limited thereto.
- the processing circuit(s) 31 may be operative to perform an elongate object tracing to follow an elongate object (such as a neurite, without being limited thereto) through the volumetric representation 40 of tissue.
- an AI model 32 may respectively be fed with AI model input that depends on (and may represent) a sub-volume of the image data 40.
- the AI model 32 is trained to output, responsive to this image data, a direction along which the elongate object extends from the already known point. This direction determines the direction in which the tracing is continued and is therefore also referred to as steering prediction. Based on the steering prediction, a next point (adjacent to the already known point) along the center line is computed. This may be done by integration that may be performed by steering prediction processing 33. The determination of the steering prediction by the AI model and the determination of a new point along the center line by processing of the steering prediction may be repeated to thereby generate a series of points located on the center line. No segmentation of the image data 40 is required for the system or method to operate in the manner disclosed herein.
- Determining the points located along the center line in 3D respectively comprises determining the location of the points within the 3D volume represented by the image data 40. Determining the points includes determining information that allows all three coordinates of the points to be specified in the volume represented by the image data 40. Determining the points may include determining three cartesian coordinates of the points, it being understood that many equivalent formulations may be used to specify the position of a point in 3D.
- a result of the image processing may be output by the image processing system 30.
- the image processing system 30 may comprise a graphical user interface (GUI) to output a graphical representation of the series of points and/or the center line of the elongate object. The GUI may be operative to output a visualization of the center line of the elongate object.
- GUI graphical user interface
- the image processing system 30 may comprise an output interface 34.
- the output interface 34 may be operative to output machine-readable data that depend on a result of the elongate object tracing.
- the machine-readable data may comprise control signals that depend on a result of the elongate object tracing.
- the machine-readable data may comprise a control logic that depends on a result of the elongate object tracing, for example when using the results of neurite tracing for neuroscience-inspired machine learning (ML) model design or training.
- the image processing system 30 may also be operative to control the GUI 30 to allow a user to enter a starting point for the object tracing.
- the image processing system 30 may be operative to control the GUI 30 to allow a user to optionally enter a starting direction for the object tracing.
- Knowledge of the initial direction is optional, as explained in more detail herein.
- An initial direction may be determined automatically, using, e.g., uncertainty estimates obtained from Monte-Carlo dropout.
- elongate object tracing is further illustrated in association with tracing of one elongate object in the following, the systems and methods are operative to trace plural elongate objects.
- the techniques are also applicable to plural elongate objects and/or plural elongate parts of objects and/or plural parts of elongate objects. This may be done in a parallel or sequential manner.
- FIG. 1 shows a volume 40 represented by image data and an elongate object 41 extending therein.
- the systems and methods disclosed herein are operative to trace a center line of the elongate object 41.
- the systems and methods may use steering-prediction dependent selection of sub-volumes, AI model processing of the image data in the sub-volume for updating the steering prediction, and integration of the steering prediction to follow a center line of the elongate object 41.
- the systems and methods may employ these techniques for neurite tracing, without being limited thereto.
- Figure 3 shows a center line 50 of an elongate object.
- the techniques disclosed herein are not only applicable to neurites but are more broadly applicable to other objects such as blood vessels, lumina of the respiratory tract, etc.
- the techniques are also applicable to traces in a spatio-temporal coordinate system (with the physical object tracing an elongate path along the time axis not necessarily being physically elongate).
- the techniques are also applicable to plural elongate objects and/or plural elongate parts of objects and/or plural parts of elongate objects.
- the methods and systems disclosed herein may be used to trace an object (e.g., a center line 50 of an elongate object) that has a locally cylindrical outer shape.
- the term “locally cylindrical” is used to refer to a geometry in which outer walls of the elongate object can be locally approximated by cylinder segments 42, 43. The orientation of the various cylinder segments 42, 43 may be tilted relative to each other in 3D space.
- FIG. 4 illustrates operation of the methods and systems disclosed herein. Processing starts with image data and an initial point 61 located on an elongate object that is to be traced.
- the initial point 61 may be set by a user, also to identify which object is to be traced. Alternatively, automatic object recognition may be employed to identify a starting point 61 on an object.
- An initial flight direction 70 that specifies the direction of the elongate object at the starting point 61 may be received by a user input or may be estimated automatically by the system, using, e.g., prediction uncertainties that specify uncertainties of prediction estimates for different directions (as obtainable from Monte-Carlo dropout, without being limited thereto).
- the systems and methods are operative to - identify a first sub-volume that is located around (but typically has a first center somewhat offset from) the starting point 61, - determine a first steering prediction 71 by processing image information from the first sub-volume, and - processing at least the first steering prediction 71 (e.g., using integration) to determine o a position offset 72 giving the first point 62 subsequent to the starting point 61, o a new sub-volume orientation (not shown) for the subsequent iteration.
- These steps may utilize the starting point 61 and starting direction 70 as input. The process may then be repeated in an iterative manner.
- the systems and methods may be operative to then - identify a second sub-volume that is located around (but typically has a second center somewhat offset from) the first point 62, - determine a second steering prediction by processing image information from the second sub-volume, and - determine a second point 63 subsequent to the first point 62 by processing at least the second steering prediction (e.g., using integration) and determining a position offset 73 and a new sub-volume orientation (not shown).
- the process may be repeated in a recurrent manner to determine a series 60 of points located on the center line 50 by recurrent inference. In this way, additional steering predictions, yielding additional position offsets 74, 75, 79 and additional points 64, 65 of the series 60 may be determined.
- a sub-volume 76 of the volume represented by the image data 40 is determined.
- the sub-volume 76 contains the current point 65 determined in the preceding iteration.
- the sub-volume 76 has an orientation that depends on the preceding steering prediction determined in the preceding iteration.
- Image data for the sub- volume 76 is processed by the AI model to determine a steering prediction 78 for the current iteration of the iterative process.
- the next point and orientation that follow the current point 65 are determined using the steering prediction 78 for the current iteration, e.g., by integration (such as by adding a vector proportional to the momentary flight direction 77 to the current point 65 to determine the next point and by adding a vector proportional to the steering prediction to the momentary flight direction / sub-volume orientation 77 to determine the next flight direction / sub-volume orientation, or alternatively by using more complex extrapolation techniques to extrapolation from the series of previously determined steering predictions).
- analysis of the image data contained in the sub- volume 76 provides the steering prediction 78, which is integrated to determine a next position offset 79 to continue tracing the elongate object.
- the systems and methods disclosed herein may thereby perform processing of sub- volumes of the volume 40 to infer the local direction of the center line 50, which corresponds to the steering prediction.
- Each sub-volume may be centered, in the two directions transverse to the momentary flight direction / sub-volume orientation, around the current point determined in the preceding iteration.
- Each sub-volume may have a center that is offset from the preceding point determined in the preceding iteration, the offset being in the direction parallel to the momentary flight direction.
- the sub-volume 76 may be positioned such that the point 65 determined in the preceding iteration is closer to the end face of the sub-volume 76 that is located from the point 65 in the direction opposite the momentary flight direction 77, than to the other end face of the sub-volume 76 that is located from the point 65 in the direction along the momentary flight direction 77.
- the methods and systems disclosed herein thus employ an approach in which the tracing process is automatically steered along the center line, with a field of view (i.e., sub- volume) that is being analyzed at each step being directed along the estimated tangential of the center line. Continuity of the elongate object is automatically built in the process.
- FIG. 5 is a flow chart of a method 80.
- the method 80 may be performed automatically by the image processing system.
- the method 80 determines a series of points 60 along the center line 50.
- AI model input is determined from the image data and the current point.
- the current point is the starting point 61 that may be set by a user input.
- the current point may be the point 62-65 on the center line determined in the preceding iteration.
- the determination of the AI model input at step 81 may determine the AI model input as image data from a sub-volume 76 of volume 40.
- the sub-volume 76 may contain a current point that may be a result of a preceding iteration.
- the sub-volume 76 may have an orientation that depends on a preceding steering prediction and sub-volume orientation.
- the sub-volume 76 may (and generally is) tilted relative to edges of the volume 40.
- the sub- volume orientation may be determined based on a user input, based on calculated prediction uncertainties over different orientations, as a random orientation, or otherwise.
- the current position and sub-volume orientation is the one determined by the AI model in the preceding iteration of the method 80.
- Determination of the AI model input at step 81 may comprise averaging (e.g., weighted averaging) or other interpolation techniques performed on voxel or pixel values of the image data 40, explained in more detail herein.
- the sub-volume of the image is processed. This may comprise processing the AI model input with the AI model.
- the output layer of the AI model may output an updated steering prediction.
- at least the steering prediction obtained at step 82 is processed to determine a next point along the center line 50 and a next orientation for the sub-volume. This may comprise vector addition or integration of the preceding (and optionally, several previously determined) steering predictions, positions and orientations.
- Termination criteria may be any one or any combination of the following without being limited thereto: intersection of the sub-volume and or next position to be analyzed with a boundary of the image data 40 or a user-defined boundary; threshold criterion for number of iterations or accumulated path length; no identifiable continuation of the elongate object in the sub- volume; intersection of the position or accumulation of path length within an object defined by other means (e.g., intersection of the neurite that is being traced with a previously traced neurite). If the termination criterion is not fulfilled, the method may return to step 81. Otherwise, a result of the elongate object tracing may be output via a GUI or data interface or may be otherwise used at step 85.
- the technique of Figure 5 may be used as part of a more complex process or method. can also be embedded in some higher order logic. For illustration, a method is disclosed that automatically detects missing continuation of an automated reconstruction, starts the technique of Figure 5 responsive to the detection of this missing continuation at that location and run the technique of Figure 5 until it finds another automatically reconstructed component. From that point, the technique disclosed herein (and as explained with reference to Figure 5) may then be run in a backward direction to reconfirm that it ends up at the same start location. Thus, the technique of Figure 5 can be used as a building block for more complex reconstruction methods.
- Figure 6 illustrates the location-dependent selection of a sub-volume for analysis in the systems and methods disclosed herein.
- a sub-volume 91 may have a cuboid shape, without being limited thereto. Four of the edges of the sub-volume 91 may be parallel to the momentary flight direction 71. The four edges parallel to the steering prediction 71 may be chosen constantly or dependent on previous AI model predictions shorter, longer or same length as the other edges of the sub-volume 91.
- the sub-volume 91 may have a size that depends on the object that is to be traced.
- the point 62 may be contained within the sub-volume 91.
- the sub-volume 91 may be positioned such that its center 92 is offset from the point 62 by an offset along the momentary flight direction 73. I.e., the sub-volume may be positioned such that it extends a greater distance along the momentary flight direction 73 than opposite the momentary flight direction, as seen from the point 62.
- the orientation of the sub-volume is variable, depending on the previous flight direction and steering prediction determined in the preceding iteration. For illustration, point 65 along the center line 50 and momentary flight direction 77 determined in an iteration of the recurrent inference process are used to determine which portion of the volume 40 is to be processed by the AI model next to continue tracing of the elongate object.
- a sub-volume 93 may have a cuboid shape, without being limited thereto.
- Four of the edges of the sub-volume 93 may be parallel to the momentary flight direction 77.
- the four edges parallel to the momentary flight direction 77 may be shorter, longer or same length as the other edges of the sub-volume 93 and can either be chosen as a constant or dependent on the AI model predictions from the previous iteration.
- the point 65 may be contained within the sub-volume 93.
- the sub-volume 93 may be positioned such that its center 94 is offset from the point 65 by an offset along the momentary flight direction 77. I.e., the sub-volume may be positioned such that it extends a greater distance along the momentary flight direction 77 than opposite the steering prediction, as seen from the point 65.
- a random rotation 95, 96 around the momentary flight direction 73, 77 may be introduced during training and/or during inference when selecting the sub-volumes 91, 93. This further enhances robustness of the techniques disclosed herein.
- the random rotation 95, 96 may be replaced or combined with methods that make the AI model (approximately) equivariant to this rotation. This has the effect that feeding a rotated input yields an accordingly rotated steering prediction. This may further increase robustness of the techniques disclosed herein.
- edges of the sub-volumes 91, 93 that are parallel to the steering direction may be chosen constantly or dependent on previous AI model predictions shorter, longer or same length as the edges of the sub-volumes 91, 93 that are not parallel to the steering direction. This further enhances robustness of the techniques disclosed herein.
- the dependence of sub-volumes 91, 93 on the preceding steering prediction, preceding flight direction and preceding points has the effect that voxels of the sub-volumes 91, 93 will typically not coincide with voxels of the original image data 40. Interpolation may be used to compute the voxel values within the sub-volumes 91, 93 for arbitrary orientation and position of the sub-volumes 91, 93, as illustrated in Figures 7 and 8.
- Figures 7 and 8 show voxels of the image data 40 (e.g., centers of the voxels) with solid circles.
- Voxels of a sub-volume 91, 93 are shown by open circles and can in general be anisotropic in terms of physical length scales.
- the voxels of the image data 40 define a grid 100.
- the voxels of the sub-volume define a second grid 105, which can be anisotropic.
- Second coordinate axes 106 of the second grid 105 may be rotated in 3D and/or shifted in 3D relative to coordinate axes 105 of the grid 100 on which the voxels of the original image data 40 are defined.
- Interpolation techniques may be used to determine a voxel value (e.g., as a grayscale or color value with various color channels) on a location on the second grid 105, which is then one of the AI model inputs.
- Projection may be used to determine the voxel values on the second grid 105.
- projection such as a trilinear projection
- Other techniques may be used.
- a point on the second grid 105 may be located within a cube defined by several (e.g., eight) voxel centers 103 of the original image data 40. Weighted averaging may be performed to compute the voxel value for the point 106 of the sub-volume 91, 93 from the voxel values of the original image data. For illustration, the voxel value for the point 106 may be computed as a weighted average of the voxel values of the eight voxel centers 103 located around the point on the second grid 105.
- local realignment may be applied.
- local realignment may be applied with respect to (potentially non- integer) x and/or y offsets (which may be obtained from cross-correlation) of 2D image planes (shown by planes of black dots in Figure 7).
- a trilinear interpolation (for generating voxel values at the white dot positions in Figure 7, when voxels at black dot positions are subject to local realignment) can be computed using two bilinear interpolations per white dot position separately determined on floored and DCled z coordinates with potentially different x/y offsets applied to the black dot positions and then combining those two intermediate values from the bilinear interpolations by another linear interpolation along the z axis.
- an AI model that includes convolutional layers has been determined to provide good performance.
- the AI model may be or may comprise a CNN.
- Figures 9 and 10 show AI models that may be used in systems and methods disclosed herein.
- the AI model 110 comprises an input layer 111 operative to receive an AI model input that is based on pixel or voxel values of the image data.
- the AI model input may comprise or may be image data within the sub-volume 91, 93, as determined from the voxel values of the image data 40.
- the voxel values may be converted from integers to floating point representation and normalized before being processed by the AI model.
- the normalization can be applied to limit the AI model input values to a certain range and / or to change statistical properties of the AI model input values, such as to achieve a mean input value of zero and a standard deviation over input values of one.
- the AI model 110 comprises an output layer 112 operative to provide an AI model output that is or indicates the steering prediction 71-75.
- the AI model 110 comprises hidden layers 113.
- the hidden layers 113 may comprise several convolutional layers 114, 115.
- the hidden layers 113 may comprise strided convolutional layers and/or fully connected layers. Strides may be different for different convolutional layers.
- the hidden layers 113 may comprise several convolutional layers 114, a dropout and/or reshaping layer 116, and one or several fully connected layers 115.
- the hidden layers 113 may comprise a linear layer as last layer.
- Various activation functions may be used. For illustration, both exponential linear units (ELU) and rectified linear units (ReLU) have been used successfully in the systems and methods disclosed herein. A wide variety of other activation functions may be used additionally or alternatively.
- ELU exponential linear units
- ReLU rectified linear units
- These activation functions may include any one or any combination of Scaled Exponential Linear Unit (SELU), Gaussian Error Linear Unit (GELU) activation function, without being limited thereto.
- SELU Scaled Exponential Linear Unit
- GELU Gaussian Error Linear Unit
- a wide variety of other artificial neural network layers may be used additionally or alternatively. These layers may include any one or any combination of fully connected layers; pooling layers (such as max-pooling); (self) attention modules; atrous convolutions; (batch) normalization layers; transformers; recurrent layers, such as long-short- term memory (LSTM), gated recurrent unit (GRU), without being limited thereto.
- the AI model may be trained to provide information on the steering prediction in a reference frame that is spatially varying along the elongate curve.
- the AI model may be trained (during a training phase) and may operate during inference to provide information on the steering prediction in a (local) Bishop frame, e.g., by outputting the two Bishop curvatures (see below for more details).
- Figure 11 is a flow chart of a method 120.
- the method 120 may be performed automatically by the image processing system 30.
- the method 120 may be performed to determine a series of points 60 along the center line 50 by recurrent inference.
- a sub-volume is selected for processing.
- the sub-volume may have a position that depends on a previously determined point along the center line (which may be specified by user input in the first iteration or as result of a preceding iteration in the later iterations).
- the sub-volume may have an orientation which depends on previous sub-volume orientations and previously determined steering predictions (which may be specified by user input or determined automatically in the first iteration and which may be taken from the preceding iteration in the later iterations).
- the AI model operates on the image data within the sub-volume. Interpolation of image data voxels may be performed to determine the voxel values in the voxel grid of the respective sub-volume, for arbitrary orientations and/or locations of the sub- volume.
- the AI model may provide, as AI model output, the updated steering prediction.
- the updated steering prediction may be specified relative to a local reference frame, e.g., as curvature values relative to a reference frame given in relation to the sub- volume (such as a Bishop coordinate system). Additional output may be provided by the AI model, such as information on a distance from an obstacle.
- a new point along the center line may be determined based on the steering prediction from step 123. For illustration, integration may be performed to determine the new point along the center line.
- Figure 12 is a flow chart of a method 125.
- the method 125 may be performed automatically by the image processing system 30.
- the method 125 may be performed to determine a series of points 60 along the center line 50 by recurrent inference.
- the method 125 may comprise steps 121-124 explained with reference to Figure 11.
- the method 125 may further comprise the additional step 126 of determining a rotation around the momentary flight direction, which is used for specifying the sub-volume in the subsequent iteration.
- Exemplary rotations 95, 96 are illustrated in Figure 6.
- Figure 13 illustrates an AI model architecture and data processing that may be employed in the methods and systems disclosed herein.
- a sub-volume 131 is analyzed that is centered (in directions perpendicular to the current flight direction) at the current position.
- the AI model 130 may include an input layer operative to receive K ⁇ N ⁇ M voxel values for the image data in the sub-volume 131.
- the number of voxels K and N in the directions transverse to the most recent steering prediction may exceed the number of voxels M parallel to the most recent steering prediction.
- the physical size of a single voxel for the sub-volume may be chosen anisotropically and independently of the physical size of a single voxel in the image data 40.
- the values K and N may be set in dependence on the object that is being traced. For illustration, for an object known or expected to have an elliptical cross-section in a plane transverse to the center axis, it may be useful to set K and N to different values.
- the AI model comprises seven strided convolutional layers conv1-conv7.
- the layers do not all need to be the same.
- three of the convolutional layers conv1-conv3 may have 5 ⁇ 5 ⁇ 3 convolution with stride 2 ⁇ 2 ⁇ 1
- four of the convolutional layers conv4-conv7 may have 3 ⁇ 3 ⁇ 2 convolution with stride 1 ⁇ 1 ⁇ 1.
- the AI model comprises a dropout layer, which may have a dropout rate of 0.5.
- the AI model comprises three fully connected layers f.c.1-f.c.3 and a linear layer.
- the AI model output 132 may be the steering prediction, optionally determined in a local reference frame (e.g., as Bishop curvature).
- the processing includes an integration 133 to determine, based on the AI model output 132, the next point along the center line.
- the sub-volume analyzed in the subsequent iteration may be set without additional random rotation (normal interference path in Figure 13) or with additional random rotation 134.
- Figure 14 shows a further implementation of the AI model 138 that may be used in the methods and systems disclosed herein.
- Parameters 139 of the convolutional layers conv1- conv7 may be modified as compared to the AI model of Figure 13.
- parameters of the fully connected layers f.c.1-3 may be modified as compared to the AI model of Figure 13.
- the parameter values before the vertical bar or after the vertical bar indicated in Figure 14 may be utilized.
- the parameter values 139 and the parameter values 139’ indicated behind the vertical bar in Figure 14 may be used.
- the systems and methods can be used for neurite tracing, without being limited thereto.
- the systems and methods allow neurite reconstruction to be performed from a 3D-EM volume as a center line reconstruction task, in which the neurite is represented by a sequence of visited points.
- the CNN is trained on the task of predicting the local neurite continuation from a neurite-centered and -aligned 3D-EM sub-volume, similar to human annotation flight mode see, e.g., Boergens et al. 2017.
- the input to the CNN consists of a 96 ⁇ 96 ⁇ 16 Voxels (Vx) neurite-centered and –aligned 3D-EM sub-volume 131.
- the sub-volume may cover, e.g.
- the third (Z) dimension corresponds to the current flight direction of the process, while the current position can be made the center of the 4 th Z-plane.
- the field of view is thus asymmetric along the Z direction with more contextual information available in forward than in reverse flight direction. This allows for better steering towards the ‘exit’ within axonal varicosities.
- the EM data may be projected onto axon-aligned planes by means of trilinear interpolation. Note that only by using a 3D EM sub-volume as input it is possible to extract relevant information about the continuation of a neurite from a single input.
- a recurrent neural network that can internally keep track of the sequence of images and thereby estimate neurite continuation as well.
- 7 3D strided convolutional layers were used, followed by a dropout layer (dropout rate: 0.5), three fully connected layers and a last linear layer that estimates the two steering commands and the distance to membrane.
- the two steering values output by the output layer as steering prediction may represent the Bishop curvatures described in more detail below.
- nonlinearity ELU were used (see, e.g., D.A-. Clevert et al. 2015.
- the Bishop frame is a local orthonormal coordinate system spanned by a tangential vector and two normal vectors and obeys a rotation minimizing property disallowing any twists around the tangential vector.
- the neurite aligned projection planes for the AI model’s input may be defined by means of the Bishop normal vectors, while the network’s flight direction may correspond to the tangential vector.
- the evolution of the Bishop frame unit vectors is coupled via the Bishop curvatures, one for each normal vector direction.
- the Bishop curvatures may be the output of the AI model output layer that is used for steering.
- Each of the two Bishop curvatures resembles a signed curvature for the corresponding normal vector direction.
- the signed curvature has also been used as steering command for image-based road following, albeit only for one steering direction corresponding to planar curves (see, e.g., KM. Bojarski et al.2016. “End to End Learning for Self-Driving Cars.” ArXiv E-Prints. https://arxiv.org/pdf/1604.07316.pdf).
- the task of predicting the neurite continuation in the form of Bishop curvatures can be interpreted as fitting a parabola to the neurite’s center line.
- the curvature vector calculated as the sum of Bishop curvatures multiplied by their corresponding Bishop normal vector, determines direction and magnitude of bending of this parabola.
- the Bishop’s curvatures output by the AI model output layer may, thus, be integrated to determine a next one of a series of points along the center line (with the series being a priori unknown and being determined to trace the center line). Training Operation of the trained AI model has been explained above. Systems and methods disclosed herein may be operative to train the AI model for its intended use.
- Figure 15 is a flow chart of a method 140 that includes both AI model training 141 and operation of the trained AI model during inference 142.
- Use of the trained AI model at step 142 may be implemented using any one of the techniques for inferring a direction along which an elongate object extends described herein.
- the training at step 141 may be a supervised training. Sets of annotated 3D-EM images may be used in the training.
- the training may comprise gradient-based updates of AI model parameters. Techniques known to the skilled person may be used for updating model parameters, such as gradient descent techniques.
- the training at step 141 may comprise training the AI model to comply with a flight policy that causes the AI model to converge back to the center line 50 even when starting from or reaching positions off the center line.
- the training step 141 may comprise training the AI model (also) on scenarios in which AI model starts from a position off the center line 50 and / or predictions of the AI model during training may be integrated and fed back as done in inference to potentially also yield positions off the center line.
- a flight policy may be implemented which enforces convergence back to the center line 50, even when the initial sub-volume (i) starts from a location that is offset from the center line and/or (ii) is not aligned with the direction of the elongate object.
- training may also be performed using sub- volumes in which none of the voxel planes of the sub-volume 91, 93 has a center that coincides with a location along the center line 50.
- At least a fraction of the sub-volumes may be deliberately chosen such that the center of, e.g., a 4 th plane (out of 16 planes of the sub- volume for the model in Figure 13) has an offset from the point on the center line determined by the annotated data or in the preceding iteration.
- the offset may be directed transversely to the previously determined extension direction (i.e., transversely to the steering prediction in the Bishop frame).
- the training may be implemented in a manner which uses one or several training parameters that are variable during the training. This allows different policies to be enforced during training. Results of the elongate object model training improve even further when using such training techniques.
- a convergence distance of the flight policy may determine how rapidly the iterative process converges back to the center line.
- the convergence distance may be a dynamic parameter that is varied during AI model training.
- the convergence distance may be set as a function of a distance from a physiological boundary during AI model training.
- the convergence distance may be a monotonously increasing function of the distance from the physiological boundary, thereby implementing a more aggressive return to the center line at positions that are closer to a physiological boundary, such as a cell boundary.
- the flight policy could also be learned directly using reinforcement learning by defining a reward function instead that encourages stable tracing of the elongate object.
- the CNN architecture of Figure 13 can be trained on a neurite following task using TensorFlow (M. Abadi et al.2016. “TensorFlow: A System for Large-Scale Machine Learning.” In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’16), 265–53784).
- the AI model can be trained with a mini-batch size of 128 using RMSProp (see, e.g., T. Tieleman and G. Hinton. 2015. “Neural Networks for Machine Learning Lecture 6a: Overview of Mini-batch Gradient Descent.” Lecture Notes. 2015.
- a volumetric reconstruction of the neurite can be used to determine off-center positions within the neurite.
- the mapping from a particular off position, off direction state to the adapted steering is also referred to as flight policy.
- a greedy flight policy may be used that is only based on local information available to the AI model within the finite field of view, as described in more detail herein.
- the distance within which to converge back to the center line is a free parameter of this flight policy. Different convergence distances may be used for off-center positions within a synaptic bouton. When closer to ’hitting’ the cell membrane, a more aggressive steering with shorter convergence distance is useful to stay within the neurite, while for larger distances to the membrane a gentler steering is affordable and better preserves neurite alignment of the input field of view.
- a dynamic convergence distance may be used that may be set in dependence on (e.g., equal to) the distance to the cell membrane along the flight direction during training.
- This induces the notion of an obstacle, i.e. cell membrane, avoidance and empirically performs better than use of a constant value for the convergence distance during training.
- Figure 16 shows a volume 40 of 3D training data.
- a neurite 151 has been annotated in the 3D training data.
- the AI model is trained using also sub-volumes such as exemplary sub- volume 156 around points such as exemplary point 152, with the exemplary point 152 (that is located in the center of one voxel plane of the sub-volume 156) being offset from the center line of the neurite 151.
- the flight direction 153 may be (significantly) different from the extension direction of the neurite within sub-volume 156 during training.
- One or several training parameters such as a convergence distance that dictates how quickly the process is to converge back to the center line of the neurite, may be dynamically varied.
- the convergence distance may be set in dependence on a distance 155 from a cell membrane 154 which is shown merely schematically.
- the distance 155 may be measured along the momentary flight direction. Performance with regard to Error Rates To assess performance, the image processing systems and methods are evaluated on a validation set and two test sets of linear axonal branches by running recurrent inference starting from both sides.
- the image processing systems and methods were trained on a 3D-EM dataset from mouse primary somatosensory cortex layer 4 acquired using serial block-face scanning electron microscopy (SBEM, (W. Denk and H. Horstmann.2004. “Serial Block-Face Scanning Electron Microscopy to Reconstruct Three-Dimensional Tissue Nanostructure.” Edited by Kristen M. Harris. PLoS Biology 2 (11))).
- SBEM serial block-face scanning electron microscopy
- the dataset has a size of 61.8 x 94.8 x 92.6 ⁇ m 3 and a voxel size of 11.24 x 11.24 x 28 nm 3 .
- Training and validation set axons are a disjoint random subset of axons seeded by means of presynaptically classified segments obtained using the techniques disclosed in B. Staffler et al.2017. “SynEM, Automated Synapse Detection for Connectomics.” ELife 6 (July): e26414. Averaged over all models and training iterations the random inference mode outperforms its normal mode counterpart by 33% (range: 12-52%).
- An AI model that was identified to have particularly good performance on the validation set uses only EM data as input, has an ELU activation function, was trained for 700,000 training iterations and yields, based on heuristic error detection, 16.4 errors/mm.
- the manual inspection of tracings of the inventive technique yields 1 false positive and 0 false negative resets (precision 96%, recall 100%).
- the true error rate is 15.7 errors/mm.
- Application of the selected AI model on a set of 59 soma seeded axons with 6.2 mm path length yields 29.8 errors/mm at 185 resets with 9 false negative resets and 5 false positive resets (precision 95%, recall 95%).
- the true error rate is 30.4 errors/mm.
- SynEM automated synapse detection for connectomics. Elife, 6, e26414. doi:10.7554/eLife.26414) and skeleton traced by annotators.
- segments of the oversegmentation obtained using SegEM with parameters as set for the whole-cell segmentation of the cortex dataset (M. Berning et al. (2015). SegEM: Efficient Image Analysis for High-Resolution Connectomics. Neuron, 87, 1193-1206. doi:10.1016/j.neuron.2015.09.003) may be picked up and combined.
- the volume mask may also be used to iteratively optimize the interpolated skeleton tracing to yield a better centerline approximation.
- the axon-trained AI model (as shown in, e.g., Figure 14 and as explained in more detail herein) may be evaluated on a random subset of 50 spine heads previously attached by human annotators, as well as on the set of spine heads previously used as test set in Motta et al., 2019, loc. cit.
- the AI model of the method and systems disclosed herein may be trained, tested and applied on a subset of a 1.3 x 1.3 x 0.25 mm 3 dataset from a barrel cortex of a 28 days old mouse that may be stained following the protocol by Y. Hua et al. (2015). Large-volume en- bloc staining for electron microscopy-based connectomics. Nature Communications, 6, 7923. doi:10.1038/ncomms8923 with small modifications, sectioned at 35 nm using ATUM (K. J. Hayworth et al. (2006). Automating the collection of ultrathin serial sections for large volume TEM reconstructions.
- a training set may be acquired based on a set of 10 soma seeded axons from Layer 4, for which annotators may pick up segments from the oversegmentation to acquire a volume reconstruction.
- Kimimaro (W. Silversmith et al. (2021). Kimimaro: Skeletonize densely labeled 3D image segmentations. In GitHub repository (v 3.0.0 ed.)) may be used followed by subsampling and B-spline interpolation to extract centerline skeletons from the volume reconstruction. This can yield a training set with a total of 21 mm axon path length. As axon validation set a random 10 out of 20 axons from a Layer 4 seeded bounding box of size (2 ⁇ m) 3 may be traced within a (50 ⁇ m) 3 bounding box yielding 1 mm path length.
- axon test set For the axon test set, another (1.5 ⁇ m) 3 bounding box within layer 4 was densely annotated and a random subset of 5 axons may be traced within the (150 ⁇ m) 3 subvolume Si150L4 (accessible at https://wklink.org/7122) yielding 1.7 mm path length.
- a separate AI model according to methods and systems disclosed herein may be trained on spine attachment. For this training and validation set may be generated from a set of 20 (5 ⁇ m) 3 bounding boxes sampled within layer 4 and annotated for spine heads. A subset of around 1000 spine heads may be volume annotated at 4 x 4 x 35 nm 3 by annotators from the spine head through the spine neck up to the dendritic trunk.
- the methods and systems disclosed herein may be applied to a recently published mm 3 -scale multiSEM dataset with voxel size 4 x 4 x 33 nm 3 (A. Shapson-Coe et al. (2021).
- the training and evaluation of the techniques may focus on a (150 ⁇ m) 3 bounding box containing 6.5 mm of axon path length of the provided ground truth skeleton tracings.
- the published ground truth skeleton tracings in this box may be used to evaluate FFN on soma-seeded axons.
- ground truth skeleton tracings of dense seeded axons may be generated by sampling a bounding box of size (2.5 ⁇ m) 3 within a centered (15 ⁇ m) 3 bounding box, annotating all processes in this bounding box and then sampling a random subset of 5 axons, which may be traced throughout the (150 ⁇ m) 3 bounding box yielding 1.25 mm path length.
- skeleton annotations were already done with high precision along the centerline such that no postprocessing was necessary.
- the volumetric neurite mask needed for training may be generated by means of segment pick-up from the c3 FFN segmentation, cf.
- the methods and systems disclosed herein may be trained or used on any or a combination of artificially generated data mimicking the appearance of elongate objects to be traced during inference X-ray, specifically X-ray holographic nano-tomography, light-microscopy, or magnetic resonance imagery of one or many elongate objects.
- the required amount of training data in path length for the methods and systems disclosed herein to work as described may in general depend among others on the resolution and visual complexity of the data, the variability in morphology and appearance of elongate objects to be traced, the algorithms and artificial neural network architectures employed for training, the number, kinds, and order of image data used for training. Performance compared to Other Image Processing Approaches Performance of the image processing systems and methods disclosed herein was also compared to other techniques.
- connectomic analyses require more reconstruction precision than others.
- connectomic analyses require more reconstruction precision than others.
- Three types of connectomes were used for comparison: (I) the connectome obtained from the fully automated reconstruction in (Motta et al., loc.
- corrections attained by inventive techniques recover the axonal target specificities of inhibitory axons onto apical and smooth dendrites, whereas in the automated state prior to techniques disclosed herein and human corrections this specificity was not detectable. So in fact, when utilizing a connectomic metric of automation qualified by the type of connectomic problems that can be automatically analyzed, error correction using techniques disclosed herein- shifts the automated analysis performance from simpler to more complex connectomic problems.
- the techniques disclosed herein were also applied to datasets from the Automatic Tape- collecting Ultra-Microtome (ATUM)/multiSEM imaging approach that are substantially larger and of different image resolution and contrast.
- ATUM Automatic Tape- collecting Ultra-Microtome
- the inventive techniques also provide enhancements as compared to techniques such as Flood-Filling Networks (M. Januszewski et al. 2018. “High-Precision Automated Reconstruction of Neurons with Flood-Filling Networks.” Nature Methods. https://doi.org/10.1038/s41592-018-0049-4).
- Application of the inventive techniques on automatically detected endings from a Flood-Filling Networks (FFN) segmentation solves 60% of the splits on a random set of axons traced throughout the sub-volume thereby reducing the split rate from 65-70 splits/mm to 28-31 splits/mm, while only introducing few new merge errors (merge rate increased from 3-4 mergers/mm to 5-6 mergers/mm).
- FPN Flood-Filling Networks
- the total runtime on a single node using 32 cores (Intel(R) Xeon(R) Gold 6130 CPU, 2 sockets), a single Tesla V100 GPU (PCIE-16GB) and less than 128GB RAM was 13.6 hours for the reconstruction of around 2.1 meters of axons (including backward validation tracings, a total of 64 million CNN inferences), and hence a reconstruction speed of around 160 mm/h ( ⁇ 1300 steps/s, average step size of 33 nm).
- the image processing systems and methods disclosed herein may be used in association with other reconstruction techniques.
- an automated reconstruction process may be run, - responsive to detecting a missing continuation of an automated reconstruction, the iterative technique disclosed herein may be started at that location and may be run until it finds another automatically reconstructed component. From that point, and - optionally, responsive to finding another automatically reconstructed component, the iterative technique disclosed herein may be run in a backward direction to reconfirm that it ends up at the same start location (for verification purposes), and - the skeleton reconstruction of above defined procedure may be used to error- correct the automated reconstruction, optionally in an iterative manner.
- an automated reconstruction process may be run, - responsive to detecting an incorrect continuation or merge error of an automated reconstruction, the iterative technique disclosed herein may be started at several locations around the error and may be run until it leaves a bounding box around the error location, and - optionally, responsive to leaving the bounding box around the error location, the iterative technique disclosed herein may be run in a backward direction to reconfirm that it ends up at the same start location (for verification purposes), and - the skeleton reconstructions of above defined procedure may be used to error- correct the automated reconstruction, optionally in an iterative manner and further optionally interleaved with iterations of detection and correction of missing continuations.
- Figure 17 is a flow chart of a method.
- the method may be performed automatically by the image processing system.
- an initial 3D image analysis is performed.
- the initial 3D image analysis may be performed for obtaining a 3D connectomic reconstruction.
- Techniques such as those described in A. Motta et al. 2019. “Dense Connectomic Reconstruction in Layer 4 of the Somatosensory Cortex.” Science. Vol 366. Issue 6469. p. eaay3134 or M. Januszewski et al. 2018. “High-Precision Automated Reconstruction of Neurons with Flood-Filling Networks.” Nature Methods. https://doi.org/10.1038/s41592-018-0049-4 without being limited thereto may be used to implement step 161.
- the image processing systems or methods disclosed herein may be used for acting as an error corrector.
- the image processing systems or methods may be employed for neurite (e.g., axon) tracing in cases where other automated techniques are incapable of annotating the image data.
- neurite e.g., axon
- the techniques disclosed herein were applied to all detected spine heads determined using the techniques of A. Motta et al.2019. “Dense Connectomic Reconstruction in Layer 4 of the Somatosensory Cortex.” Science. Vol 366. Issue 6469. p. eaay3134 that previously required human annotations (38% of all detections consuming around 900 working hours of annotators). A precision/recall of attached spine heads of 89%/84% was achieved, approaching a precision of human decision.
- the start positions from the spine head detection that were also provided to human annotators were used.
- the direction prediction of the techniques disclosed herein was then performed.
- the recurrent inference mode of the techniques disclosed herein were applied for the candidate direction with lowest prediction uncertainty and tracing was performed until a dendritic trunk defined by a dendrite mask was reached or a path length threshold was exceeded.
- the precision/recall reported above was achieved without retraining the techniques disclosed herein on spine necks. Even further improvements can be made by training on a dedicated training set from spine necks.
- the stop criterion was based on a bounding box around the merge error – again analogous to human – and tracings were accepted if validation in backward direction yielded the same skeleton reconstruction with some error tolerance.
- Split and merge error resolution was run iteratively until remaining errors dropped below a threshold.
- the final axon reconstruction for this fully automated approach yielded 12 split errors / mm and 16 merge errors / mm on a representative set of 10 axons (1.7 mm), which is comparable to and approaches the error rate of human annotations (5 split errors / mm and 12 merge errors / mm) and outperforms conventional automatic techniques (Motta et al 2019 prior to human annotations: 32 split errors / mm and 14 merge errors / mm).
- Direction Prediction As discussed herein, the image processing systems and methods are operative to trace an elongate object by iteratively determining the local direction (referred to as flight direction) along which an elongate object extends.
- An initial direction (i.e., a direction that is used for selecting a sub-volume for analysis by the AI model in the first iteration) may be specified by user input.
- the image processing system may be operative to control the GUI to enable a respective user input for subsequent use by the elongate object tracing technique disclosed herein.
- the initial direction may be set as a random direction (with the AI model typically guiding the process back to the center line in view of its training).
- Monte-Carlo dropout is used on a number of candidate orientations.
- the implementation of the direction selection from the candidates may be dependent on the stability as estimated by the image processing systems and methods.
- Figure 18 is a flow chart of a method 165.
- the method 165 may be performed automatically by the image processing system or method.
- direction sampling is performed.
- a set of directions which may be equidistant or roughly equidistant on the surface of a unit sphere, may be sampled.
- uncertainty estimates may be computed for the various directions. The computation of the uncertainty estimates may be performed in various ways.
- the uncertainty estimates may be determined for a plurality of directions using a process that comprise running Monte-Carlo dropout, computing covariances of direction estimates (such as Bishop curvatures) per orientation, and determining the uncertainty estimate based on a covariance matrix of the direction estimates.
- one or several directions may be determined based on the uncertainty estimates. Operation & Effects
- the image processing systems and methods address the need for techniques capable of performing automated axon tracing (while lending themselves to other applications as well).
- the image processing systems and methods address axon reconstruction in a manner similar to a flight reconstruction task, yielding a skeleton, by means of a sequence of visited points, rather than a volumetric (voxel) reconstruction.
- the image processing systems and methods provide an automated approach for predicting skeleton reconstructions in the field of connectomics directly from raw EM data.
- the image processing systems and methods provide the possibility for an end to end optimization.
- the image processing systems and methods focus on the essence of neurite reconstruction: following neurites over long distances through the dataset.
- the image processing systems and methods make use of the fact that neurite reconstructions and error metrics on the level of skeletons are more robust than their voxel- based counterparts, which suffer from ambiguities in the affinity of individual voxels, e.g. in the presence of staining or imaging artifacts.
- the image processing systems and methods provide a new approach for neurite tracing, which is skeleton rather than voxel based and show good results.
- Figures 19 and 20 show operation of the image processing systems and methods.
- the image processing systems and methods are operative to trace neurites (e.g., axons) through a 3D volume represented by image data.
- Tracing here means determining a series of points in 3D or determining a continuous curve in 3D that specify the trace of the neurite in 3D.
- a series 171 of sub-volumes 172-274 is analyzed using the techniques disclosed herein.
- a number of planes 175 spaced along a flight direction each include image data that is determined from the original image data using techniques such as interpolation or projection (e.g., trilinear projection). Information such as distance from a membrane may be determined and used in, e.g., AI model training.
- FIG. 20 shows a locally variable coordinate frame 180, such as a Bishop frame, which may be employed in the techniques disclosed herein. Effectively, the image processing systems and methods operate by sequentially analyzing sub-volumes of a larger tissue volume, with the sub-volume representing a field of view as seen when flying along the already traced part of the elongate object center line in 3D.
- the Bishop curvature vector 185 may be an output of the AI model and may be integrated to determine new points along the center line. Techniques for training the AI model are also described in detail.
- the method 200 may be performed by or using an image processing system as disclosed herein.
- 3D EM image acquisition is performed.
- the 3D EM image acquisition may be performed on a bioptic sample.
- the bioptic sample is a sample that is not implanted back into a living animal or human after imaging.
- a trace of at least one neurite is determined.
- the traces of several neurites may be determined concurrently, e.g. by operating several instances of the image processing system in parallel. Additionally this step may comprise a more complex logic for iterative reconstruction and may contain the module 80 from Figure 5 as a submodule.
- a termination criterion is checked. If the termination criterion is not fulfilled, the method returns to step 202.
- the method may be continued until all neurites of interest are traced. In this manner, several neurites may be traced in the volume.
- Neurite tracing may comprise tracing axons, such as determining the axon connectome, and/or dendrites.
- synapse information is computed. This can be done using a synaptic interface classifier known to the skilled person without being limited thereto.
- results may be output. The results may be output via a GUI or data interface, without being limited thereto.
- Figure 23 is a graph 210 which depicts error rates required for various uses in association with connectomics analyses.
- Figure 24 is a graph 220 depicting an evolution of the time required for one mm of annotation, as a function of the techniques available in recent years.
- a further, very pronounced decrease 222 in the annotation consumption rate is provided by the methods and systems of the invention as disclosed herein.
- Figure 25 is a graph 230 that quantifies a rate of axon splits and mergers, reported for randomly seeded axons.
- Results 231 were obtained for a FFN-based reconstruction of human cortex (A. Shapson-Coe et al. (2021). “A connectomic study of a petascale fragment of human cerebral cortex.” bioRxiv, 2021.2005.2029.446289. doi: 10.1101/2021.05.29.446289 (2021)).
- the techniques disclosed herein provide significantly improved results 241 for both segmentation states analyzed, with an increase in merger rate.
- split rate requirements depend on synapse rate along axons: for at least 5 synapses per axon, split rates below 40 per mm and 20 per mm are required for typical mouse and human axons, respectively. Moving automation into these regimes is therefore desirable for more difficult connectomic analyses.
- the methods and systems disclosed herein improved densely seeded axon reconstruction for large-scale data from mouse cortex.
- the results 232 were obtained using a conventional technique (unpublished), as compared to the inventive techniques which provide improved results 242.
- the methods and systems disclosed herein allow the full replacement of human annotation for the employed analyses, providing an improvement over the results 233 obtained using a conventional technique of Motta et al., loc. cit.
- Figure 26 is a further graph 260 that quantifies a rate of axon splits and mergers, reported for randomly seeded axons.
- the techniques were applied to 3D EM data obtained using a state-of-the-art high-throughput 3D EM imaging approach followed by multibeam scanning electron microscope (multiSEM). The data were obtained for both a mouse cortex and a human cortex.
- Data 261, 262, 263 correspond to conventional techniques.
- Data 264 corresponds to data obtained additionally using human annotations.
- Data 271, 272, 273 correspond to data obtained using the methods and systems disclosed herein.
- the methods and systems disclosed herein were seeded at automatically detected endings of axon agglomerates, and used to connect split axonal agglomerates.
- split rates of axons were reduced 7-fold (42.7 to 6.0 per mm axon path length) while only modestly increasing merger rates (3.3 to 4.5 per mm, as seen in Figure 26, top datapoint 262 as compared to bottom datapoint 272).
- Figure 26 shows a quantification of a rate of axon splits and mergers, reported for randomly seeded axons.
- FFN flood-filling networks
- Such error rates can be attained using the techniques disclosed herein (data 272, 273).
- merge errors were detected which extended further than 2.2 ⁇ m from the ground truth and manually verified that this heuristic accurately detects merge errors.
- each merge error was counted as 1/2 and divided by the ground truth path length to yield the merge error rate. This is because each merge error usually connects two neurites and counting them as 1 error per neurite instead of 1/2 would overestimate the total amount of merge errors.
- Figure 27 is a flow chart of a method 240.
- the method 240 may be performed by or using an image processing system as disclosed herein.
- 3D EM image acquisition is performed.
- the 3D EM image acquisition may be performed on a bioptic sample.
- the bioptic sample is a sample that is not implanted back into a living animal or human after imaging.
- image acquisition is performed on a biopsy sample.
- the image acquisition may be a 3D EM image acquisition on a bioptic sample that will be discarded after imaging.
- a trace of at least one neurite is determined. The traces of several neurites may be determined concurrently, e.g.
- results of the tracing are output.
- the results may be output via a GUI (e.g., by outputting a visual representation of parts of a connectome via an optical output device) or data interface, without being limited thereto.
- the image processing systems and methods disclosed herein may be used in the process of generating, training, and/or operating a processing logic (such as a signal processing logic) for controlling at least one physical asset (such as a machine).
- Figure 28 shows a system 250 which comprises an image processing system operative as disclosed herein.
- the image processing system may be operative to determine information on (parts of) a connectome and/or other structural information of the nervous system of a human or other animal.
- the system 250 comprises a physical asset 252 (such as a machine) which is controlled by a controller 251.
- the controller 251 executes a machine learning, ML, model to process signal inputs (which may include sensor inputs from sensor measurements) and generates control signals that affect operation of the physical asset 252.
- the ML model executed by the controller 251 may be generated by a control logic generation module 36 of the processing system 30.
- the control logic generation module 36 of the processing system 30 may use results of the connectome analysis to at least one of: (i) select or generate the ML model with a ML model architecture that depends on the connectome; (ii) train the ML model having the ML model architecture that depends on the connectome; and/or (iii) deploy the trained ML model that depends on the connectome as the signal processing logic to the controller 251 for execution.
- the controller 251 may execute the ML model that depends on the connectome determined by the processing system 30 for controlling the physical asset 252.
- the controller 251 may also feed back information to the processing system 30. For illustration, operation of the controller 251 and/or asset 252 may be monitored during field use.
- the processing system 30 may be caused to repeat and/or modify the object tracing during field use of the controller 251 and/or asset 252. In this way, the control logic may be updated during field use.
- the controller 251 may be coupled to the image acquisition system 20 to control operation thereof. This may be done responsive to the results of the image processing performed by the image processing system 30.
- the tissue was stained following the protocol by Y. Hua et al. 2015. “Large-Volume En-Bloc 599 Staining for Electron Microscopy-Based Connectomics.” Nature Communications 6 (August): 7923 with small modifications and imaged at 4 x 4 nm2 and nominal cutting thickness of 35 nm.
- the image processing systems and methods were also evaluated on a multiSEM dataset with voxel size 4 x 4 x 33 nm 3 (A. Shapson-Coe et al. (2021). “A connectomic study of a petascale fragment of human cerebral cortex.” bioRxiv, 2021.2005.2029.446289.
- Skeleton interpolation Interpolation may be used to obtain a continuous representation and/or center line nodes with a desired step size of neurite branches (e.g. axonal branches) from sparsely placed nodes along the center axis.
- the interpolation may be implemented as a 4th order B-spline interpolation (L. Piegl and W. Tiller.1996.
- the default step size d can be selected to be equal to the smallest length of a voxel. This size may depend on the imaging system and may be, e.g., 11.24 nm for the specific SBEM dataset used in some examples herein.
- the adaptive step size ensures that no voxel up to a radius of half the projection plane size in the projection plane at the current position of the CNN is skipped.
- the same adaptive step size may optionally be used during inference for the inventive technique disclosed herein to determine the step size of the integration step that determines subsequent positions and orientations.
- Bishop frame The Bishop frame (R.L. Bishop.1975. “There Is More than One Way to Frame a Curve.” The American Mathematical Monthly 82 (3): 246–51) consists of three orthonormal vectors, namely the tangential vector to the curve (also termed flight vector) ⁇ , and two vectors and ⁇ 2 .
- the invariance of the curvature vector under certain operations are used in the techniques disclosed herein in association with the AI model.
- online data augmentation may be performed.
- the flight direction can also be flipped, which corresponds to a time reversal symmetry. On the center line this time reversal and tangential vector flip does not affect the Bishop curvatures, while for the flight policy discussed below this symmetry can be broken.
- the Bishop frame has weaker requirements for the curve ⁇ than other frames (such as the Frenet-Serret frame) often used as local coordinate system in differential geometry of parameterized curves.
- the Bishop frame does not require ⁇ ⁇ 0.
- Initial conditions together with the Bishop curvatures uniquely define the center line curve ⁇
- the Bishop frame also exists for parametrized curves in N-dimensional Euclidean space with N ⁇ 3, the same mathematical framework can be used and the techniques disclosed herein are similarly applicable to N-dimensional data.
- parametrized curves in 2- dimensional Euclidean space can be considered a special case of parametrized curves in 3- dimensional Euclidean space, such that also here the described methods are applicable.
- Flight policy A flight policy usable during AI model training (for enforcing convergence back to the center line when applying the AI model to a position off the center line) can be derived by considering (a) the second order Taylor series expansions for the known center line curve ⁇ with corresponding Bishop frame ⁇ , ⁇ 1 , and and curvature vector ⁇ ⁇ ⁇ , and (b) the second order Taylor series expansions for the unknown trajectory converging back to the center line with off-center line position ⁇ and corresponding Bishop frame ⁇ , ⁇ 1 , and and curvature vector ⁇ ⁇ ⁇ :
- convergence back to the on-center line tracing can be formulated as minimization of the future distance ⁇ ⁇ ⁇ defined by:
- the parameter sc may be chosen in dependence on distance of the current off-center line position from an obstacle, such as a membrane (e.g., a cell membrane).
- an obstacle e.g., membrane
- an obstacle e.g., membrane
- convergence to the center line is achieved through iterative application of above steering.
- Direction prediction with Monte-Carlo Dropout The technique disclosed in this section can be used to, e.g., perform direction prediction for obtaining an initial steering prediction.
- equidistant or approximately equidistant orientations ⁇ ⁇ on the surface of the unit sphere can be sampled.
- the uncertainty estimate for a direction can be determined as the square-root of the largest eigenvalue of the covariance matrix divided by the mean curvature, This quantity is similar to a coefficient of variation.
- a weighted average of uncertainties can be determined, e.g. as with weights The weighted averaging provides enhanced stability.
- the orientation with minimal averaged uncertainty may be used as the first orientation candidate.
- a second candidate may be chosen which is at least 110° from the first candidate and has a minimal averaged uncertainty from among the remaining orientations.
- a local realignment may be performed in the method and/or by the imaging system 20.
- the local realignment may be performed by image alignment component 23 or the imaging system 20 may be operative to execute a second, local image realignment component in addition to the image alignment component 23.
- the local realignment techniques will be described, at least in part, in association with 3D EM data. It should be understood that these techniques may also be used in association with other 3D image data or sets of 2D image data.
- the local realignment may comprise the following steps executed to perform a local realignment of slices of image data along a slicing direction: determining slice-to-slice shift vectors for a plurality of slices along the slicing direction; identifying valid slice-to-slice shift vectors between slices; using the valid slice-to-slice shift vectors to generate a locally realigned subvolume.
- the locally realigned subvolume may be fed to the AI model input, e.g., to an input layer of a CNN or RNN or to a filter or other preprocessing that preprocesses the realigned subvolume prior to it being processed by the CNN or RNN.
- Determining the slice-to-slice shift vectors may comprise determining an auto- and cross-correlations of slices. Determining the slice-to-slice shift vectors and/or their validity may comprise determining a mean and a standard deviation of correlation-weighted shift vectors based on a first threshold criterion. Identifying valid slice-to-slice shift vectors may comprise verifying that the determined auto- correlation fulfills a first criterion (such as having a peak at a desired shift, e.g., at zero shift).
- Identifying valid slice-to-slice shift vectors may comprise verifying that a standard deviation of the shift vector fulfills a third cross-correlation criterion (such as the standard deviation being equal to or less than a fifth threshold).
- Using the valid shift vectors may comprise identifying one or several slices to be realigned even though the slice-to-slice shift vector is not considered valid, and computing the shift vector in this case from valid shift vectors relating to a neighborhood of these one or several slices.
- Using the valid shift vectors may comprise generating a locally realigned and trilinearly interpolated subvolume. Generating the locally realigned and trilinearly interpolated subvolume may comprise performing floor or ceil operations.
- Generating the locally realigned and trilinearly interpolated subvolume may comprise performing a trilinear interpolation as a sequence of linear interpolations along cardinal axes.
- the sequence of linear interpolations may comprise four linear interpolations along a first dimension, two linear interpolations along a second dimension, and one linear interpolation along a third dimension.
- the locally realigned subvolume may be processed to trace an elongate object, such as a blood vessel or neurite, in 3D.
- the methods and systems disclosed herein may be operative to apply an alignment method locally on a potentially imperfect global alignment, followed by a neurite reconstruction method receiving the locally realigned 3D subvolume, similar to the techniques of P.H. Li et al., "Automated Reconstruction of a Serial-Section EM Drosophila Brain with Flood-Filling Networks and Local Realignment.” Preprint. https://doi.org/10.1101/605634.
- the reconstructions within the subvolume then need to be dealigned, i.e., projected back into a global reference frame (as discussed in the cited document by P.H. Li et al.).
- the slice-to-slice shift vectors for local realignment are precomputed on a grid for a whole dataset and bilinearly interpolated to yield approximate per-voxel localized alignment, where the size of the grid for precomputed shift vectors is a hyperparameter of the local realignment method.
- locations of global maxima of auto/cross-correlations of slices ( ⁇ ⁇ , ⁇ ⁇ ), ( ⁇ ⁇ , ⁇ ⁇ +1 ) and ( ⁇ ⁇ , ⁇ ⁇ +2 ) may be determined corresponding to pixel-level shift vectors ⁇ ⁇ , ⁇ , ⁇ ⁇ , ⁇ +1 , ⁇ ⁇ , ⁇ +2 .
- the mean and standard deviation of correlation-weighted shift vectors reaching a correlation of at least two-thirds of the maximum correlation relative to the noise level are computed.
- shift vectors were considered if the correlation exceeded ⁇ ⁇ ⁇ , where the peak cross-correlation between slices ⁇ , ⁇ is ⁇ ⁇ , ⁇ , std( ⁇ ⁇ ) std( ⁇ ⁇ ) (i.e., standard error of the correlation expected under the assumption of independently and identically distributed pixel intensity values within each slice).
- ⁇ ⁇ ⁇ the peak cross-correlation between slices ⁇ , ⁇ is ⁇ ⁇ , ⁇ , std( ⁇ ⁇ ) std( ⁇ ⁇ ) (i.e., standard error of the correlation expected under the assumption of independently and identically distributed pixel intensity values within each slice).
- the realignment may be implemented such that, for shift vectors ⁇ ⁇ , ⁇ +1 from the cross- correlation of slices ( ⁇ ⁇ , ⁇ ⁇ +1 ) to be valid, the following must apply: (1) auto-correlation conditions must be true for both slices, (2) the peak cross-correlation normalized with its respective maximum must be ⁇ 0.18, (3) the shift vector standard deviation must be ⁇ 10 pixels.
- the realignment may be implemented such that, for locations, for which ⁇ ⁇ , ⁇ +1 are invalid (in accordance with the above-described criterion for shift vectors to be valid), shift vectors ⁇ ⁇ , ⁇ +2 from ( ⁇ ⁇ , ⁇ ⁇ +2 ) slices are considered if the following criteria are met: (1) auto- correlation conditions (see above) must be true for both slices and (2) the shift vector standard deviation must be ⁇ 11 pixels.
- the realignment may be implemented such that, to compute otherwise invalid shift vectors ⁇ ⁇ , ⁇ +1 , valid shift vectors ⁇ ⁇ , ⁇ +2 and/or ⁇ ⁇ 1, ⁇ +1 may be used.
- Shift vectors ⁇ ⁇ , ⁇ +1 as determined above may be precomputed for every ⁇ -slice on a regular grid with a grid spacing of 512 pixels along in-plane ⁇ ⁇ ⁇ axes.
- bilinear interpolation may be employed.
- the shift vectors of neighboring slices for ⁇ / ⁇ locations ( ⁇ ) ⁇ / ⁇ ⁇ and ( ⁇ ) ⁇ / ⁇ ⁇ may be determined using bilinear interpolation of all relevant slice-to- slice shift vectors ⁇ ⁇ , ⁇ +1 for this subvolume. Then, a cumulative sum may be used to aggregate slice-to-slice shift vectors across ⁇ coordinates of neighboring slices and from the result the aggregated shift vector of slice ⁇ ( ⁇ ) ⁇ / ⁇ ⁇ ⁇ may be subtracted. This yields shift vectors with respect to the current position ⁇ and hence a shift of 0 at slice ⁇ ( ⁇ ) ⁇ / ⁇ ⁇ ⁇ .
- the method and system may make use of the formulation of a trilinear interpolation as a sequence of linear interpolations along cardinal axes: (1) four linear interpolations along a first dimension, (2) two linear interpolations of results from (1) along a second dimension and (3) one linear interpolation of the results from (2) along a third dimension.
- the interpolation of a single voxel can be computed by first linearly interpolating along ⁇ and ⁇ axes independently for floored and DCled ⁇ values respectively with corresponding shifts along ⁇ and ⁇ applied, and then linearly interpolating along the ⁇ axis.
- the methods and systems disclosed herein may be operative to implement the dealignment, comprising first integrating the predicted Bishop curvatures to ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ as before and then applying the aggregated shift vectors from the start ⁇ slice ⁇ ( ⁇ ) ⁇ / ⁇ ⁇ ⁇ , used as reference for the local realignment of the subvolume, to a target ⁇ slice ⁇ ( ⁇ ⁇ ⁇ ⁇ ⁇ ) ⁇ / ⁇ ⁇ ⁇ onto ⁇ ⁇ ⁇ ⁇ ⁇ to yield a position within the global coordinate system.
- Alignment techniques may be used in association with reassembling of 2D images into a consistent 3D image volume with a common global reference frame. This is referred to as alignment.
- alignment In case of errors in any of the preparation, slicing, imaging or reassembling steps, the resulting 3D image dataset will generally not be consistent at all locations.
- the realignment techniques disclosed herein are technically advantages for many imaging techniques disclosed herein. For illustration, to image neural tissue using 3D EM, the neural tissue is physically disassembled (sliced) along one axis, 2D-imaged with an electron microscope, potentially as a mosaic with overlapping fields of view, and virtually reassembled to ideally yield a globally consistent 3D image dataset.
- the realignment techniques disclosed herein may be applied.
- the alignment techniques disclosed herein provide good consistency while avoiding excessive computational runtimes.
- the alignment techniques disclosed herein are beneficial for many of the applications considered herein. While embodiments have been described in detail, various modifications and alterations may be implemented in other embodiments. For illustration, the techniques disclosed herein may be applied to tracing a wide variety of elongate objects.
- the techniques disclosed herein are not limited to tracing elongate objects and/or to tracing in 3D but may be applied to a wide variety of other use cases, such as tracing an object through a series of image frames (e.g., in a video), tracing an object in a spatio-temporal coordinate system, and/or performing image processing for a wide variety of control actions.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Quality & Reliability (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
- Measuring And Recording Apparatus For Diagnosis (AREA)
Abstract
Description
Claims
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| EP22188663 | 2022-08-04 | ||
| PCT/EP2023/071656 WO2024028477A1 (en) | 2022-08-04 | 2023-08-04 | Method of tracing an elongate object in image data and image processing system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| EP4566025A1 true EP4566025A1 (en) | 2025-06-11 |
Family
ID=83232512
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| EP23750634.0A Pending EP4566025A1 (en) | 2022-08-04 | 2023-08-04 | Method of tracing an elongate object in image data and image processing system |
Country Status (6)
| Country | Link |
|---|---|
| EP (1) | EP4566025A1 (en) |
| JP (1) | JP2025528079A (en) |
| KR (1) | KR20250044699A (en) |
| CN (1) | CN119654656A (en) |
| CA (1) | CA3261355A1 (en) |
| WO (1) | WO2024028477A1 (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN121095274B (en) * | 2025-08-26 | 2026-03-03 | 重庆理工大学 | A boundary enhancement model for semi-supervised medical image segmentation |
-
2023
- 2023-08-04 JP JP2025505919A patent/JP2025528079A/en active Pending
- 2023-08-04 CA CA3261355A patent/CA3261355A1/en active Pending
- 2023-08-04 EP EP23750634.0A patent/EP4566025A1/en active Pending
- 2023-08-04 CN CN202380056962.6A patent/CN119654656A/en active Pending
- 2023-08-04 WO PCT/EP2023/071656 patent/WO2024028477A1/en not_active Ceased
- 2023-08-04 KR KR1020257005729A patent/KR20250044699A/en active Pending
Also Published As
| Publication number | Publication date |
|---|---|
| KR20250044699A (en) | 2025-04-01 |
| JP2025528079A (en) | 2025-08-26 |
| CN119654656A (en) | 2025-03-18 |
| CA3261355A1 (en) | 2024-02-08 |
| WO2024028477A1 (en) | 2024-02-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113313234B (en) | Neural network system and method for image segmentation | |
| Januszewski et al. | High-precision automated reconstruction of neurons with flood-filling networks | |
| Blendowski et al. | Weakly-supervised learning of multi-modal features for regularised iterative descent in 3D image registration | |
| CN113269754B (en) | Neural network systems and methods for motion estimation | |
| US10258304B1 (en) | Method and system for accurate boundary delineation of tubular structures in medical images using infinitely recurrent neural networks | |
| Januszewski et al. | Flood-filling networks | |
| Matejek et al. | Biologically-constrained graphs for global connectomics reconstruction | |
| Zubair et al. | Automated brain tumor detection using soft computing-based segmentation technique | |
| Yang et al. | A dense R‐CNN multi‐target instance segmentation model and its application in medical image processing | |
| CN116681685A (en) | Method and device for automatic recognition and segmentation of cell bodies based on two-photon calcium imaging data | |
| Graves et al. | Siamese pyramidal deep learning network for strain estimation in 3D cardiac cine-MR | |
| Liang et al. | Enhanced center coding for cell detection with convolutional neural networks | |
| Mayerich et al. | Deep learning | |
| Su et al. | JANet: A joint attention network for balancing accuracy and speed in left ventricular ultrasound video segmentation | |
| Zhang et al. | Gamed-snake: Gradient-aware adaptive momentum evolution deep snake model for multi-organ segmentation | |
| WO2024028477A1 (en) | Method of tracing an elongate object in image data and image processing system | |
| CN116030463B (en) | Dendritic spine labeling method and tracking method of neuron fluorescence microscopic image | |
| Sun et al. | Caiseg: A clustering-aided interactive network for lesion segmentation in 3d medical imaging | |
| Kar et al. | Assessment of deep learning algorithms for 3D instance segmentation of confocal image datasets | |
| Rohr et al. | Tracking and quantitative analysis of dynamic movements of cells and particles | |
| Wang et al. | AI-enhanced 3D biomedical data analytics for neuronal structure reconstruction | |
| Lu | Breast ultrasound image segmentation based on attention U-Net | |
| Avetisian et al. | Volumetric Medical Image Segmentation with Deep Convolutional Neural Networks. | |
| Lakshmi | Cardiac function review by machine learning approaches | |
| Ghindawi | Advanced Computer Vision Alignment Technique Using Preprocessing Filters and Deep Learning. |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: UNKNOWN |
|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
| PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
| 17P | Request for examination filed |
Effective date: 20250304 |
|
| AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC ME MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 40123256 Country of ref document: HK |
|
| DAV | Request for validation of the european patent (deleted) | ||
| DAX | Request for extension of the european patent (deleted) |