JP2019114262A - Medical image processing apparatus, medical image processing program, learning apparatus and learning program - Google Patents

Abstract

To provide a medical image processing apparatus capable of realizing a registration by a new method.SOLUTION: A medical image processing apparatus includes: an acquisition part configured to receive first image data and second image data; and a generation part configured to generate a prediction displacement for executing registration processing between the first image data and the second image data. The generation part is trained by repeatedly executing the generation of the prediction displacement and a training using the identification part. The identification part is trained so as to distinguish a prescribed displacement and the prediction displacement.SELECTED DRAWING: Figure 4a

Description

  The present embodiment relates to a medical image processing apparatus, a medical image processing program, a learning apparatus, and a learning program.

  Medical image registration is to align two images into a common space. Medical image registration may be performed on volume data representing a medical image, or such volume data that may be processed to obtain a medical image. Volume data can be aligned to a common space.

  Registration may be used, for example, to allow the clinician easy access to information not represented by just one image. In one example, the images may be images acquired at different times, and the information provided by the registration may be tumor progression over time.

  Registration can be performed on images acquired using the same modality of acquisition (monomodality registration) or on images acquired using different modalities of acquisition (multimodality registration) . In the case of multimodality registration, registration of soft tissue images (e.g. magnetic resonance images) and hard tissue images (e.g. computed tomography images) is performed to create a more extensive depiction of the patient's anatomy. There is something to do.

  Registering the images may allow for direct comparison of the images. For example, anatomical features may occupy corresponding positions in each image, where the images have been registered once. Depending on the circumstances, further image processing may be performed on the registered image (eg segmentation, subtraction or image fusion).

  In the registration process in which two images are registered, one of the images is designated as a reference image and the other is designated as a converted image (or floating image) to be converted. be able to. The purpose of the registration process may be to obtain a transformation that relates the coordinate system of the transformed image to the coordinate system of the reference image. A transform may be applied to the transformed image to align the transformed image to a reference image.

  Conventional image registration processes may use the measurement of how well the two images are aligned, and may apply transformations sequentially on one image, with the goal of optimizing the measurement of the alignment. . For example, alignment measurements may comprise similarity measurements. The initial transformation can be applied to the transformed image to obtain values for alignment measurements. The initial transformation can be done sequentially in such a way as to improve the values for the alignment measurements until the values for the alignment measurements are converged.

  Among the registration methods, there are transformations such as rigid transformations by rotation, translation, and scaling.

  In other registration methods, the transformation may include local deformation, and there is also a non-rigid transformation that performs transformation processing.

JP-A-2015-514447

  Generally, for non-rigid registration, regularization is used to constrain the transform space. Such regularization may, for example, constrain the smoothness of the field or the maximum possible absolute magnitude of the maximum possible value of the transformation.

  If regularization is not used, there is a possibility that non-rigid registration may output transformations that do not make physical sense, for example, transformations that represent changes in physically impossible anatomy. is there. For example, when intensity-driven metrics are used, non-rigid registration does not consider how pixels represent biological structure (eg, without keeping adjacent pixels together) It may provide an output that simply matches the brightness of pixels between one image and another.

  That is, according to the conventional registration, operational restrictions may occur when using regularization, and realistic results may be obtained when regularization is not used.

  In view of the above circumstances, the present embodiment aims to provide a medical image processing apparatus, a medical image processing program, a learning apparatus, and a learning program that realize registration by a new method.

  The medical image processing apparatus according to the present embodiment includes an acquisition unit that receives first image data and second image data, and registration processing between the first image data and the second image data. A generation unit configured to generate a predicted displacement to be executed, the generation unit being trained by repeatedly executing generation of the predicted displacement and training using an identification unit; Training to distinguish between the predetermined displacement and the predicted displacement.

The schematic of the apparatus which concerns on embodiment. Flow chart outlining the training method of the regressor. The flowchart which delineates an identifier. 4 is a flowchart depicting an overview of a generator training method according to an embodiment. The flowchart which outlines the training method of the identifier concerning an embodiment. FIG. 6 is a flow chart outlining a trained generator deployment according to an embodiment. FIG. 5 is a schematic diagram depicting the application of a predetermined displacement field to a reference image to synthesize a transformed image. Ground truth displacement field (upper), training using displacement field predicted using a system trained using only mean squared error (MSE) (middle), mean squared error and classifier feedback FIG. 7 is a schematic view of a series of examples of displacement fields (bottom) predicted using the system described above. 4 is a flowchart outlining a training process using multiple classifiers according to an embodiment.

  An image data processing apparatus 10 according to an embodiment is schematically depicted in FIG. In the embodiment of FIG. 1, the apparatus 10 is configured to train a generator for registering medical images and to use the trained generator for registering medical images. ing. In other embodiments, a first device can be used to train a generator, and a second, different device can use a trained generator to register medical images. it can. In further embodiments, any device or combination of devices can be used.

  The image data processing device 10 comprises in this case a computing device 12, such as a personal computer (PC) or a workstation, which comprises a scanner 14, one or more display screens 16 and a computer. It is connected to one or more input devices 18 such as a keyboard, mouse or trackball.

  Scanner 14 may be any scanner configured to perform medical imaging. Such scanner 14 is configured to generate imaging data representative of at least one anatomical region of a patient or other subject. The scanner can be configured to acquire two-dimensional or three-dimensional image data in any imaging modality. For example, the scanner 14 is a magnetic resonance (MR) scanner, a computed tomography (CT) scanner, a cone-beam CT scanner, an X-ray scanner, an ultrasound scanner, a positron emission tomography (PET) A scanner, or a single photon computed computed tomography (SPECT) scanner can be provided. In further embodiments, the scanner may generate any type of image data, but such image data may not be medical imaging data.

  In this embodiment, the imaging data set acquired by the scanner 14 is stored in the data store 20 and then provided to the computing device 12. In an alternative embodiment, the imaging data set is supplied from a remote data store (not shown) that can form part of a medical image archiving and communication system (PACS). Data store 20 or remote data store may comprise any suitable form of memory storage. The imaging data set used for the generation processing and the identification processing executed in the processing circuit 22 may be a set of two-dimensional image data or a set of three-dimensional image data.

  The computing device 12 comprises processing circuitry 22 for data processing, including image data. The processing circuit 22 includes a central processing unit (CPU: central processing unit) and a graphic processing unit (GPU: graphical processing unit).

  The processing circuitry 22 provides processing resources for processing the image data set automatically or semi-automatically. For the sake of simplicity, in the following it will be referred to as medical image processing. However, in practice, the operations described below can be performed on any suitable set of image data representing a medical image. The image data may be processed internally by the processing circuitry 22 without any corresponding image being displayed.

  The processing circuit 22 is configured to use a training circuit 24 (learning device) configured to train a generator to register a medical image, and a trained generator to register a medical image. And a registration circuit 26.

  In this embodiment, the circuits 24, 26 are respectively executed on the CPU and / or the GPU in the manner of a computer program having computer readable instructions executable to carry out the method of the embodiments. In other embodiments, the various circuits may be implemented as one or more application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs).

  The computing device 12 also includes a hard drive, other PC components such as a RAM, a ROM, and a data bus, an operating system such as various device drivers, and a hardware device such as a graphics card. Such components are not shown in FIG. 1 for the sake of clarity.

  FIG. 2 is a flowchart outlining a method of training generator 36.

  The generator 36 is a neural network configured to learn non-rigid transformation predictions to align two predetermined images. Non-rigid transformations are represented by displacement fields (eg, scalar fields, vector fields, tensor fields, etc.). The generator is any function (eg, any depth learning) that provides a fit between the two data sets associated by the transformation, such as minimizing or reducing at least one parameter characterizing the transformation. Law) can be included.

  A plurality of training images are used to train generator 36.

  The generator 36 is trained on ground truth conversion. The ground truth transformation is an already known transformation. The ground truth transformation can also be referred to as a training transformation or a predetermined transformation.

  In fact, it may be difficult to obtain transformations for realistic pairs of images (eg, images of the same anatomy acquired at different times) that can be used as ground truth.

  Thus, in the method of FIG. 2, the ground truth transform is a predetermined transform that is artificially constructed. For example, the ground truth transform can be obtained by sampling a 2D Gaussian profile function or by sampling a sinusoidal function. These ground truth transforms are applied to the training image to obtain a composite transformed image.

  Referring back to the flowchart of FIG. 2, a reference image 30 is received. The reference image 30 is one of the training images for which the generator is to be trained.

  At stage 32 of FIG. 2, a predetermined displacement field is applied to the reference image 30. The predetermined field is an artificially constructed displacement field. For example, a predetermined displacement field may be obtained by sampling a 2D Gaussian function. The predetermined displacement field may be referred to as a synthetic displacement field, for example, because it is constructed artificially rather than being obtained by performing a registration process.

  The result of applying a predetermined displacement field to the reference image 30 is a transformed image 34. Since the transformed image 34 is not acquired from the scanner but is obtained using a (artificially constructed) predetermined displacement field, the transformed image 34 is synthesized with the synthesized transformed image (or synthesized transformed image) I sometimes call.

  The training circuit provides a reference image 30 and a composite transformed image 34 (instead of a predetermined displacement field) generator 36.

  The generator 36 is a neural network represented by a deep neural network (DNN). However, the generator 36 is not limited to the deep neural network, but may be a general neural network. The generator 36 can be initialized with an initial set of weights. Training of the generator 36 may comprise adjusting the weights of the generator 36.

  The generator 36 uses the reference image 30 and the composite transformed image 34 as inputs to its neurallet work. The neural network processes the reference image 30 and the composite transformed image 34. For example, the neural network can extract features from the reference image 30 and the composite transformed image 34 and process such extracted features. The neural network outputs a prediction of displacement field 38 that represents the transformation between reference image 30 and composite transformed image 34.

  The training circuit compares the predicted displacement field 38 to the predetermined displacement field used at stage 32 to synthesize the transformed image 34.

  It is known that the predetermined displacement field is the correct transformation between the reference image 30 and the composed transformed image 34 since it was used to construct the composed transformed image 34. The predetermined displacement field thus provides a ground truth that is compared to the generator's prediction.

  A measurement of the difference between the predicted displacement field 38 and the predetermined displacement field is calculated. In the method of FIG. 2, the measurement of the difference is the mean squared error (MSE). Other versions of the method can use any suitable difference measure.

  If the prediction provided by the generator 36 is good, then the predicted displacement field 38 is very similar to the predetermined displacement field. Conversely, if the prediction provided by the generator 36 is bad, then the predicted displacement field 38 is clearly different from the predetermined displacement field.

  The mean squared error is provided (fed) to the generator 36 as training feedback 40. The training circuit may update the weights of generator 36 in response to training feedback 40.

  The method of FIG. 2 is repeated many times for many different reference images and many predetermined displacement fields. The generator 36 learns from many cases of training feedback 40 to improve the predictions about its displacement field. The generator 36 is trained sequentially to minimize the mean square error between the predicted displacement field and the real displacement field.

  Depending on the circumstances, optimizing the displacement field based only on the mean squared error (eg, as described above with reference to FIG. 2) may lead to poor predicted displacement, It has been discovered. Also, depending on the circumstances, small, slight displacements may be overwhelmed by larger errors elsewhere in the image.

  When training the generator 36 to minimize only the mean squared error, there is no boundary that the generator 36 predicts over the shape of the displacement field. The generator 36 may generate an unrealistic displacement field, which is considered accurate in terms of mean squared error. In some circumstances, the predicted displacement field output by the generator 36 may represent a physically impossible transformation for the anatomy represented in the image attempting to register.

  The solution provided by training the generator 36 to minimize only the mean square error (eg, as depicted in FIG. 2) may be considered to be insufficiently regularized. It is proposed to add a discriminator to the process of training the generator in order to eliminate the insufficient regularization. Such classifiers can be thought of as providing indirect regularization.

  FIG. 3 is a flow chart outlining the process performed by the classifier 56. The discriminator is trained to distinguish the real displacement field 50 (e.g. the predetermined displacement field as described above) and the predicted displacement field 52, e.g. the displacement field predicted by the generator 36 It is done.

  The identifier 56 receives two displacement fields 50 and 52. One of the two displacement fields is a realistic displacement field 50 that represents the transformation between the reference image and the transformed image. For example, the real displacement field 50 may be a predetermined displacement field as described above. The other of the two displacement fields is the displacement field 52 (predicted displacement field 52) predicted by the generator for the same reference image and transformed image as described above.

  The discriminator 56 is not informed as to which of the two displacement fields 50, 52 is real and which is the prediction.

  In the example of FIG. 3, the identifier 56 also receives the difference image 54. The difference image 54 is obtained by subtracting the converted image from the reference image.

  The classifier 56 processes the displacement field and the difference image. In the example of FIG. 3, the identifier 56 is a neural network represented by a deep neural network (DNN). However, the discriminator 56 is not limited to the deep neural network, but may be a general neural network. In addition, the classifier 56 may have a configuration other than a neural network. The identifier 56 receives as inputs the predefined displacement field 50, the predicted displacement field 52 and the difference image 54. The identifier 56 processes the inputs 50, 52, 54. For example, the classifier 56 can extract features from the displacement field and the difference image, and process the extracted features.

  The neural network outputs a determination as to which of the two displacement fields 50, 52 was determined to be real and which was determined to be predictive. Such a determination comprises classification of one of the displacement fields 50, 52 as real and the other of them as prediction. The determination may comprise, for each displacement field 50, 52, the probability or likelihood that the displacement field is real. The determination may also comprise, for each displacement field 50, 52, the probability or likelihood that the displacement field 50, 52 is a prediction.

  4a and 4b are flowcharts outlining the training method of the generator 66 according to an embodiment. The training of the generator 66 is performed by the training circuit 24 of the device of FIG.

  To train the generator 66, the training circuit 24 uses a hostile network, which can be called a deterministic adversary network (DAN). Such a deterministic hostile network consists of two parts. The first part of the deterministic hostile network is the generator 66. Such a generator 66 comprises a first deep neural network. The second part of the deterministic hostile network is then the identifier 76. Such a discriminator 76 comprises a second deep learning network.

  The deep neural network can be a neural network comprising neurons stacked in layers. Neurons stacked in layers can have a non-linear activation function, using the output of one or more previous layers as the input of subsequent layers. Deep neural networks can highly construct non-linear mappings from input space to output space, which can capture the complex relationships of processes or tasks to be modeled.

  In the embodiment of FIGS. 4a and 4b, each of the generator 66 and the identifier 76 comprises a separate convolutional neural network. In other embodiments, any suitable type of deep layer neural network can be used, such as a multilayer perceptron, a convolutional neural network with skip connections, a recurrent neural network, etc. In a further embodiment, the discriminator 76 comprises an algorithm that does not comprise deep learning.

  The generator 66 and the identifier 76 are trained repeatedly in a hostile manner. The training circuit 24 executes the training of the generator 66 and the training of the discriminator 76 alternately on a batch basis (batch-wise basis). Hostile training is performed alternately between the generator training stage and the classifier training stage. The generator training stage is described below with reference to FIG. 4a. The classifier training stage is then described below with reference to FIG. 4b.

  At the generator training stage, the generator 66 is trained (eg, may include the weight updates of the generator 66) and the identifier 76 is held constant.

  The discriminator 76 is trained to discriminate between the predetermined displacement field and the displacement field predicted by the generator. The generator 66 is trained to create a displacement field similar to a predetermined displacement field sufficiently to fool the discriminator 76. By alternately executing the optimization of the generator 66 and the optimization of the discriminator 76, the generator 66 is better at creating a displacement field, and the discriminator 76 is also predicted to be a predetermined displacement field. It is better to distinguish between different displacement fields. By training generator 66 and identifier 76 together in a hostile manner, a better displacement field can be created compared to when generator 66 is trained alone. The displacement field may be less likely to exhibit unrealistic behavior or exhibit discontinuities.

  The training of a deep layer neural network in a hostile manner is described in detail in the following document, and it is assumed that all documents related to the document name are included here. Goodfellow et al, Generative Adversalial Nets, NIPS '14 Proceedings of the 27th International Conference on Neural Information Processing Systems, pages 2672-2680.

  The pre-training process (not shown) can be trained by the training circuit 24 to initialize the weights of the generator 66 and / or the identifier 76. For example, the generator 66 can be pre-trained using the method of FIG. 2, using only the mean square error as the objective function, and the resulting generator 66 can be used as an initial model for hostile training It can be used.

  FIG. 4a depicts a generator training stage, which is part of the hostile training process where the training circuit 24 trains the generator 66, while the discriminator 76 is held constant.

  The generator training stage comprises determining the set of weights for the deep neural network of generator 66, which in this embodiment is a convolutional neural network. The training process uses the identifier 76 to train the generator 66. During training of the generator 66, only the weights of the generator 66 are updated since the weights of the identifier 76 are frozen.

  The training process shown in FIG. 4a begins with the training image, shown in reference image 60. Although only one training image (reference image 60) is shown in the flow chart of FIG. 4a, in practice the training process of FIG. 4a is performed on a large number of training images, eg hundreds or thousands of training images. Ru. The training image is a medical image collected using the scanner 14. Training images may be collected using any suitable imaging modality.

  Reference image 60 is received by training circuit 24 from data store 20. In other embodiments, reference image 60 may be received directly from any suitable data store or from a scanner.

  At stage 62 of FIG. 4a, training circuit 24 applies a predetermined displacement to the reference image. In the present embodiment, the predetermined displacement is a displacement field 70 representing non-rigid transformation. In other embodiments, any displacement can be used. Any suitable format or function of displacement can be used to represent the transformation.

  In the present embodiment, the transformation is de-parameterized. In other embodiments, the transformation may be parameterized. In the embodiment of FIG. 4a, predetermined displacement fields are obtained by sampling a 2D Gaussian profile.

  Although only one synthetic displacement field 70 is shown in FIG. 4b, in practice the training process of FIG. 4a is for a large number of synthetic displacement fields, eg, hundreds or thousands of synthetic displacement fields. To be executed. In an embodiment, the combined displacement is sampled from the 2D Gaussian profile function shown in equation (1) below.

A is a real number uniformly drawn from U (-6.5, 6.5), and N is a real number Gauss is a real number independently drawn from U (-55, 55) u _x and u _{y is} shifted from the center of the image and is a normalization constant as shown in the following equation (2).

The Gaussian profile is sampled independently from the above equation for D _x and D _y to construct a displacement field D = {D _x , D _y } so as to synthesize a transformed image from the reference image.

  In other embodiments, any suitable method of acquisition of predetermined displacement fields 70 can be used. For example, any model can be used to make the predetermined displacement field 70 resemble the displacement field resulting from realistic physical distortion.

  The predetermined displacement field 70 deforms the reference image 60 to obtain the combined transformed image 64.

  The training circuit 24 subtracts the combined transformed image 64 from the reference image 60 to obtain the difference image 74.

  The reference image 60, the transformed image 64, the displacement field 70, and the difference image 74 can be considered together to provide a set of training data for training the generator 66.

  In other embodiments, the reference image 60 is pre-processed using the displacement field 70 to obtain the transformed image 64 and the difference image 74 before starting the training process. In further embodiments, reference images, transformed images, displacement fields and difference images can be obtained in any suitable manner. For example, in some embodiments, predetermined displacement fields 70 can be obtained by applying any suitable registration process to the reference image and the transformed image.

  Training circuit 24 provides reference image 60 and transformed image 64 to generator 66. The generator 66 uses the reference image 60 and the transformed image 64 as inputs to its neural network. Given a reference image 60 and a transformed image 64, the generator 66 predicts the displacement field in a non-parametric formulation to align two predetermined images.

  The neural network of generator 66 outputs the predicted displacement field 72.

  In the present embodiment, the predicted displacement field 72 represents a non-parametric transformation (e.g., a dense warp field. In other embodiments, the predicted displacement field represents a parametric transformation. For example, parametric transformations can comprise spline coefficients to a grid of a single scale or multiple scales of control points to transformations In a further embodiment, displacement can be any suitable representation For example, displacement can be represented as a field or as a parameterized equation.

  Training circuit 24 provides a predetermined displacement field 70 and a predicted displacement field 72 to identifier 76. Training circuit 24 does not indicate which of the provided displacement fields 70, 72 is predetermined and which is predicted. In the present embodiment, training circuit 24 also provides difference image 74 to discriminator 76. In other embodiments, training circuit 24 may provide identifier 76 with a reference image and / or a transformed image. Training circuitry 24 may provide any image or data derived from reference image 60 and / or transformed image 64 to identifier 76. For example, training circuitry 24 may provide distance functions obtained from reference image 60 and transformed image 64 to identifier 76. Training circuit 24 may provide the similarity measure obtained from reference image 60 and transformed image 64 to identifier 76. Furthermore, the training circuit 24 may also provide the discriminator 76 with any form of residual image between the reference image and the transformed image, for example the square of the dot product between the gradients of the reference image and the transformed image. is there.

  The discriminator 76 uses the predetermined displacement field 70 and the predicted displacement field 72 as inputs to its own neural network. The neural network of the discriminator 76 outputs a determination 78 as to which of the displacement fields has been determined to be predetermined and which has been determined to be predicted. The decision 78 comprises or represents the classification of the displacement fields 70, 72 respectively. In this embodiment, the determination comprises the possibility associated with whether the two supplied displacement fields are predetermined displacement fields.

  In this embodiment, the determination may comprise any suitable classification. The classification can comprise binary classification as each displacement field being predetermined or predicted. The classification may comprise a probabilistic classification comprising the probability or likelihood that each image is predetermined or predicted.

  Training circuit 24 provides feedback to generator 66. The feedback comprises a first component based on the output of the generator 66 and a second component based on the output of the identifier 76. The first component is shown as MSE feedback 80 in FIG. 4a. The second component is shown in FIG. 4 a as identifier feedback 82.

  Training circuit 24 adjusts the weight of generator 66 in response to the two components 80, 82 of feedback.

  In this embodiment, the MSE feedback 80 is a value for the objective function, which may be referred to as a loss function or a conventional loss function. The objective function provides a measurement of the difference between the predicted displacement field 72 and the predetermined displacement field 70.

  It is known that the predetermined displacement field 70 accurately represents the conversion between the reference image 60 and the converted image 64, since the predetermined displacement field 70 has been used to construct the converted image 64. There is. Thus, the predetermined displacement field 70 acts as a ground truth, and the predicted displacement field 72 is quantified by comparing the predicted displacement field 72 with the predetermined displacement field 70. .

  In the present embodiment, the objective function aims at the mean squared error. In other embodiments, any objective function can be used, such as mean absolute error or Huber loss. The objective function may be based on displacement field comparisons or direct comparisons of reference and transformed images, respectively.

  In this embodiment, the objective function is a conventional loss function suitable for registration of images of the same modality (monomodality registration), for example the mean squared error between the images. In another embodiment, if the intended use of the generator is to register images of different modalities (multimodality registration), use a loss function appropriate for multimodality registration be able to. For example, calculation of the mean squared error between displacement fields that is independent of modality can be used. In some embodiments, the conventional loss function is based on image residuals appropriate for multi-modality registration, eg, normalized gradient fields.

In this embodiment, the objective function directly calculates the predicted displacement field, which can be denoted D ^pred, and the predetermined displacement field, which can be denoted D ^groundtruth . For example, the calculated value can be written as (D ^pred −D ^groundtruth ).

In other embodiments, the objective function can be calculated in any suitable manner. In some embodiments, a predicted displacement field is applied to the transformed image to obtain a transformed transformed image, which may also be referred to as a corrected transformed image. An objective function is calculated between the reference image and the transformed image transformed. The reference image can be denoted as R. The transformed transformed image can be written FoD ^pred , where F is the transformed image and o is a transformation operation using the predicted displacement field. The objective function can be calculated as (R-FoD ^pred ). Ground truth displacement may not be used if an objective function using the reference image and the transformed image transformed is used. In such cases, ground truth displacement can be provided to the discriminator.

  The classifier feedback 82 is a value for a further function that can be described as a discriminatory loss function. The discriminatory loss function represents the error of the discriminator 76 in detecting which of the provided displacement fields is predetermined and which is predicted. For discriminatory loss, any suitable function can be used. For example, the discriminatory loss function may comprise binary reciprocal entropy.

  The training circuit 24 adjusts the weights of the generator 66 to minimize the mean squared error between the output of the generator and the predetermined displacement field and to maximize the error of the discriminator 76. The weight at generator 66 maximizes the error signal of discriminator 76, since it is actively trained to force discriminator 76 to believe that the predicted displacement field is real. In the meantime, training is adjusted.

  The relative contribution of the MSE feedback 80 and the classifier feedback 82 may be different in different embodiments. In particular, different levels (sometimes called strength) of the classifier feedback 82 can be used. It can be said that the level of discriminator feedback used may affect how well the resulting displacement field is regularized. In the hypothetical scenario that only the classifier was used to train the generator, the displacement field produced by the generator is highly real (eg highly continuous), but the reference image and It may not be related to the converted image. Using a combination of MSE feedback 80 and identifier feedback 82, it may be seen that the displacement field is also regularized while associating the reference image to the transformed image.

  The balance between MSE feedback 80 and identifier feedback 82 may be known by tuning. In some embodiments, tuning is based on manual inspection of predicted transformations. Also, in some embodiments, tuning is based on measurements of one or more features of the displacement field. For example, one or more features can comprise whether the predicted displacement field is locally invertible. Whether it is possible to invert can be measured by calculating the Jacobian determinant of the predicted displacement field.

  The process of FIG. 4a can be repeated for multiple training images to train the weights of generator 66.

  The generator training stage described above with reference to FIG. 4a is performed alternately with the classifier training stage. FIG. 4b is a flowchart outlining the classifier training stage.

  In the discriminator training stage, training circuit 24 trains discriminator 76 using a plurality of predetermined displacement fields and a plurality of corresponding predicted displacement fields. The predicted displacement field is predicted by the generator 66. The weights of generator 66 are fixed to produce the predicted displacement field for the classifier to be trained.

  Returning to FIG. 4b, the training circuit 24 receives a predetermined displacement field 84, a predicted displacement field 85, and a difference image 86. Training circuit 24 provides a predetermined displacement field 84, a predicted displacement field 85, and a difference image 86 to discriminator 76. Although only a single predetermined displacement field 84, predicted displacement field 85, and difference image 86 are depicted in FIG. 4b, the training process of FIG. 4b may, for example, be hundreds or thousands. It can be performed for many displacement field pairs, such as pairs.

  The difference image 86 is the difference between the reference image and the transformed image associated by the predetermined displacement field 84. For example, the transformed image may be made from a reference image using a predetermined displacement field 84, as described above. The predicted displacement field 85 is predicted by the generator 66 from the same reference image and the same transformed image.

  The discriminator 76 uses a predetermined displacement field 84, a predicted displacement field 85, a difference image 86 as input to its neural network. The discriminator 76 determines which of the displacement fields 84, 85 is the predetermined displacement field and which is the predicted displacement field. In the present embodiment, the discriminator 76 generates the possibility of which of the two provided displacement fields 84, 85 relates to the predetermined displacement field.

  The training circuit 24 calculates an error signal of the discriminator 76. The error signal of the discriminator 76 represents the degree of success of discrimination between the predisposed displacement field 84 and the predicted displacement field 85.

  In this embodiment, the error signal used to train the discriminator 76 is the same discriminatory loss function as described above in connection with FIG. 4a. In other embodiments, different functions can be used.

  The training circuit 24 provides the value for the discriminatory loss function to the discriminator 76 as discriminator feedback 88. The training circuit 24 adjusts the weights of the discriminator 76 to minimize the discriminatory loss function.

  A classifier training stage is performed for a plurality of predetermined and predicted displacement fields.

  The generator training stage and the classifier training stage are alternately performed until convergence is reached.

  In this embodiment, the number of training examples used in each generator training stage and the number of training examples used in each classifier training stage are fixed numbers. It has been found that using a fixed number of training examples before switching between generator training and classifier training may result in stable training of the system. In other embodiments, different numbers of training examples can be used at different training stages.

  The number of training examples used may be referred to as the switch rate. In some embodiments, tuning of the switch rate is automated based on monitoring of one or more characteristics of the loss with training. For example, the training circuit 24 can switch from classifier training to generator training (and vice versa) once the loss value for the classifier falls below a predetermined value. In this embodiment, training of the system is halted when the generator begins to over-learn training data. This is measured by having a subset of data as validation data that was not used to train the system. The value of the loss function is calculated periodically on the data for verification as the training proceeds. As the training progresses, training ceases when the value of the loss function no longer decreases.

In summary, the generator 66 is trained using predetermined transformations. The predefined transformations are non-rigid transformations and are described using displacement fields. A classifier 76 is used to train the generator 66. The classifier 76 is a type of deep neural network trained to recognize realistic displacement field features. The discriminator 76 allows the generator 66 to predict displacement fields, which are still accurate (MSE feedback) but better regularized (discriminator feedback), to help train the generator 66. While providing additional feedback.
After training, the discriminator 76 is removed from the system leaving the generator 66. The trained generator 66 can then be used to register new images whose conversion between images is not yet known.

  FIG. 5 is a flow chart outlining the use of trained generator 66.

  The registration circuit 26 receives two medical images 90, 92 to be registered with one another. The two medical images can be referred to as a reference image 90 and a transformed image 92. In some embodiments, the medical images 90, 92 can be images of the same anatomy of the same patient, such as, for example, images of the same anatomy acquired at different times. Also, in some embodiments, the medical images 90, 92 can be images of different subjects. Furthermore, in some embodiments, one of the medical images 90 may comprise or form a portion of an anatomical atlas.

  Registration circuit 26 provides reference image 90 and transformed image 92 as input to generator 66. The neural network of generator 66 is trained to output the predicted displacement field. The process performed by trained generator 66 may be referred to as a registration process.

  The generator 66 outputs a predicted displacement field 94 representing the transformation between the reference image 90 and the transformed image 92. The predicted displacement field can be applied to align the reference image 90 and the transformed image 92.

  Processing circuitry 22 may utilize displacement field 94 and / or the aligned image to perform further processing. The further processing can be any processing where registration of the image is a prerequisite. For example, the further processing may comprise further registration. Further processing may comprise segmentation. The processing may also comprise the detection of at least one anatomical feature in the image. Further processing comprises detection of at least one lesion in the image, for example detection of damage in the image. In some embodiments, detection of anatomical features and / or lesions may comprise segmentation. In other embodiments, detection of anatomical features and / or lesions may comprise detection of their presence. In yet other embodiments, detection of anatomical features and / or lesions may comprise the determination of a position (eg, a single coordinate) relative thereto. In some embodiments, the generator 66 is used to register an image into an atlas to perform atlas-based segmentation or another atlas-based processing.

  The further processing may comprise subtraction or other Poulian arithmetic. Further processing may comprise image fusion, in which features of the aligned image are combined to form a single image.

  In the embodiment described above with reference to FIGS. 4a, 4b, and 5, the deep learning model is used to regress to a non-rigid transformation that aligns the two images.

  Since the neural network of generator 66 is trained to create a displacement field, generator 66 produces the predicted displacement field in one step. Producing the predicted displacement field in one step differs from some known registration methods which use many successive steps of the registration process to perform the registration. By using a trained neural network, registration can be obtained more quickly than using some known registration methods.

  Generator 66 is a single pass generator configured to predict non-rigid displacement fields between images. The trained generator 66 can provide a fast and non-sequential registration method.

  The generator 66 will be learning to create a displacement field over training not only to minimize the mean squared error, but also to overwhelm existing classifiers. Since the generator 66 is trained to mislead the discriminator, it can be expected that the trained generator will output a real displacement field (ie, a displacement field near to real).

  Using a deterministic hostile network to train the generator can be thought of as an additive regularization of the solution displacement field, as well as training. The classifier can be thought of as an additional regularization during generator training. It is similar to the discriminator constraining the displacement field that the generator predicts by penalizing the displacement solution that the generator predicted as fake.

  By training the generator 66 and the identifier 76 in a hostile manner, a better generator 66 is created compared to when the generator 66 is trained alone without using the output from the identifier 76. Can.

  The displacement fields created by the generator 66 may be better regularized. The classifier 76 indirectly regularizes the predicted transformation / displacement field learned by the generator 66. Identifier 76 provides another form of feedback to aid in training generator 66. The identifier 76 takes three inputs. A true displacement field, a predicted displacement field, and a difference image. The identifier 76 generates the possibility as to which of the two supplied fields is the true displacement field. The generator 66 is trained to set the discriminator 76 and additionally to minimize the difference between real and predicted displacement fields.

  In this embodiment, the hostile training of generator 66 is trained using the entire image, and trained generator 66 uses the entire image as its input. In other embodiments, training and / or use of the generator 66 can be performed patchwise with respect to portions of the image instead of at the level of the entire image. In patchwise methods, an image or image volume is divided into sub-images or sub-volumes, which can be called patches. Registration is performed to align individual patches. The initial alignment of the individual patches can be such that pairs of patches will be aligned using that the generator 66 contains the same information. Such initial alignment may, for example, be distortion of the controlled synthesis of the reference image to create a transformed image; a rigid body registration pre-processing step before dividing the image into patches; or if the patches are large enough, Can be achieved through Depending on the circumstances, patch-wise registration may be computationally feasible compared to full image registration. Patch-wise registration may require less computational resources. In patchwise registration methods, full displacement fields can be generated through aggregation of patch level displacement fields.

  In some embodiments, the training of generator 66 is specific to a particular body part and / or a particular imaging modality. For example, in one embodiment, the images used to train generator 66 are all MR images of head slices, and generator 66 is thus trained to register MR images of head slices Ru. In other embodiments, the generator 66 can be trained on images of any body part, for example, on cardiac images or images of any suitable organ. The generator 66 is trained on 2D or 3D images of any modality.

In one example, two generators 66 are trained on MR head image slices.
1. Trained generator using only mean squared error (similar to that described above with reference to FIG. 2)
2. Trained generator using mean squared error and classifier (similar to that described above with reference to FIG. 5)
The generator is trained to output displacement fields given two input images (a reference image and a transformed image).

  FIG. 6 is an illustration of a process for applying predetermined Gaussian displacement fields 104, 106 to an MR head image slice (reference image 100) to synthesize the transformed image 108.

  Arrow 102 illustrates the process of applying displacement fields 104, 106 to reference image 100 to obtain transformed image 108. The plots 104, 106 represent displacement fields for X and Y, respectively. In the plots 104, 106, the color values (indicated by shading in the figure) represent the degree of distortion. The color represents the amplitude of the applied warp field. The color values (indicated by shading) for each pixel in plot 104 represent the amount of X displacement for the corresponding pixel of the image. Also, the color value (shown in gray) for each pixel in plot 106 represents the amount of Y displacement for the corresponding pixel of the image.

  The comparison results of the two trained generators are as follows.

  A generator trained using a combination of mean squared error and classifier achieves higher peak signal to noise ratio than that achieved by a trained generator using only mean squared error It has been proven that.

  7 is a table of images showing the effect of classifier feedback on the training of the generator. FIG. 7 provides the following. Visual demonstration of ground truth displacement fields; displacement fields predicted from generators trained using only mean squared errors; predicted displacement fields from generators trained in deterministic hostile networks, It is. The results are shown for each of the three verification examples (FIG. 1, FIG. 2, FIG. 3).

  The combined ground truth displacement fields are shown at the top of the table. The X and Y displacements are shown for each of the three examples. The degree of displacement is indicated as color (shade). The displacement field shown is that applied to the MR head image slice to produce the image pair to be registered.

  The middle part of the table shows the displacement fields predicted using a system trained using only the mean squared error according to the same method as described above in connection with FIG.

  The lower part of the table is a displacement field predicted using a system trained using mean squared error and classifier feedback according to the same hostile method as described above in connection with FIGS. 4a and 4b. Indicates

  The circles 110 to 117 visually highlight the improvement in the displacement field between the middle (mean squared error only) and the bottom (hostile).

  The displacement field in the middle stage is different from the ground truth displacement field shown in the upper stage. The displacement field in the middle shows a noisy area shown by circles 110, 111, 112, 113.

  The circles 114, 115, 116, 117 in the lower row show the corresponding 110, 111, 112, 113 respectively indicated by the circles in the middle. It can be seen that there is a reduction in the noise around these offset areas. Also, the similarity to the ground truth object shown in the upper row is increasing.

  Using hostile components to extend the training of neural networks that regress to displacement fields between two images may result in improved PSNR statistics. The use of hostile components can also result in more visually pleasing displacement fields.

  The visual results shown in FIG. 7 show that the displacements predicted by the trained model in the DAN framework (bottom) are less spurious noise around the true displacements in both the x and y components of the displacement field It has been demonstrated that it appears to have (spurious clutter).

  The training method described above in connection with FIGS. 4a and 4b uses a single identifier 76. In a further embodiment, multiple classifiers are used to train the generator.

  In some circumstances, it is known that an ensemble of classifiers may provide more accurate predictions than a single classifier. Thus, multiple classifiers may be used for a single generator to provide hostile feedback.

  In an embodiment, multiple classifiers are used with each classifier receiving the predicted displacement.

  FIG. 8 is a flowchart outlining a method according to an embodiment in which multiple classifiers are used to train the generator 124.

  In training, training circuit 24 receives reference image 120 and transformed image 122, which are inputs to generator 124. The generator 124 outputs the predicted displacement 126.

  The predicted displacement 126 is provided to a plurality of classifiers 130a, 130b, ... 130n. Each classifier 130a, 130b,... 130n outputs a separate determination of whether the displacement provided to it is predetermined or predicted.

  In some embodiments, some or all of the classifiers may be a reference image and a transformed image, or some residual image calculated between two such and / or ground, if available. Receive other inputs such as truth displacement.

  In some embodiments, each classifier provides individual hostile feedback. In other embodiments, the classifications from each of the classifiers are aggregated, and hostile feedback is derived from such aggregated classifications.

  Multiple classifiers may differ from one another in various ways. For example, different ones of the multiple classifiers can be trained on different data. Also, different ones of the multiple classifiers can have different structures. Furthermore, different ones of the multiple classifiers can be initialized using different weights.

  Different ones of the multiple classifiers may use different inputs. For example, some classifiers may receive a reference image and a transformed image (or an image or data derived from the reference image and / or the transformed image), but not another.

  In the embodiment described above, the classifiers (or each of the multiple classifiers) comprise a deep learning model. In other embodiments, the discriminator may be used without the deep learning model. Any suitable discriminator can be used as long as the error signal provided by the discriminator is distinguishable with respect to the generator weights.

  In the embodiment described above, the classifier receives the predetermined displacement field and the displacement field predicted by the generator, and which of the received displacement fields is the predetermined displacement field and which is predicted A two-arm discriminator (or each of a plurality of multiple discriminators) configured to output a decision as to whether it is a displacement field. The identifier has two channels, one of which receives the output of the generator and the other of which receives the ground truth displacement.

  In other embodiments, the classifier (or at least one of the multiple classifiers) receives a single displacement field and whether such received displacement field is a predetermined displacement field or predicted by a generator It can be configured to output a determination of whether it is a displaced displacement field. The identifier has a single channel that receives a single set of displacements.

  Depending on the circumstances, it has been found that a two-arm identifier may provide more stable, hostile feedback to the generator as compared to a one-arm identifier.

  In the embodiments described above, the generator is trained on medical images. The generator represents the same anatomical region of the subject, such as an image of the same anatomy acquired with the first medical image data and, for example, acquired at different times or using different imaging modalities. Train about two medical image data. In other embodiments, the first medical image data is representative of an anatomical region of the subject, and the second medical image data is representative of an anatomical region of the subject, or of the subject or a further subject. Represents the corresponding anatomical area of the specimen. In some embodiments, one set of medical image data comprises atlas data.

  In a further embodiment, the generator can be trained on any type of image (which may or may not be a medical image). The generator can be used to register any image type. Images can be collected using any imaging method.

  In other embodiments, a method similar to that described above is used to train the generator to estimate the depth field for a pair of stereo images. In stereo imaging, two images are acquired from different perspectives using two imaging devices. The difference between the two images is a function of the distance from the two imaging devices to the object represented in the images. By processing the distance between points in the two images, a depth field can be created that estimates the distance from the imaging device to the object in the image.

  In some embodiments, the medical image pair is offset from the first and second imaging devices (e.g., the first and second imaging devices) such that the first and second images form a stereo image. And the second camera). The medical image pair is an input to a generator, which is trained to output a depth field for the medical image pair. The generator is trained hostilely with the discriminator so that the predicted depth field is more realistic. A generator training process in which the generator is trained to predict depth fields, and a classifier training process in which the classifier is trained to distinguish between the predetermined depth fields and the depth fields predicted by the generator. It is repeatedly replaced.

  In some embodiments, the second imaging device may be the same device as the first imaging device. For example, the position of the imaging device may be moving between the acquisition of the first image and the acquisition of the second image. In some embodiments, the first imaging device and the second imaging device may comprise different parts of a single device, for example different sensors of a single camera.

  The features described above with respect to generator training and its use to predict displacement fields can also be applied to generator training and its use to predict depth fields for stereo images.

  Certain embodiments provide a method for training a neural network, referred to herein as a generator, to estimate a non-rigid displacement field that aligns two or more images in a hostile manner. The above hostile approach consists of traditional loss function minimization and discriminatory loss maximization.

  The generator can predict either parametric or non-parametric transformations to align two or more images. The hostile training provided by the classifier can be applied at the whole image level. Also, the hostile training provided by the classifier can be applied to patchwise principles.

  The identifier can have a single channel that receives a single set of displacements. The identifier may have two channels, one channel receiving the output of the generator and the other channel receiving the ground truth displacement.

  The classifier may additionally receive an arbitrary distance function / similarity metric between the residual image, the image input to the neural network that regresses to the displacement field.

  Multiple classifiers can be used to provide a hostile / discriminatory loss component for a single predetermined set of displacements.

  The image inputs to the neural network may be of the same modality. The image input to the neural network may be of different modalities.

  Conventional loss functions can be evaluated between predicted and ground truth displacements. Conventional transformation functions can be evaluated between the reference image and the predicted displacement-distorted template image.

  The predicted displacement field can be used for further registration, segmentation or atlas based tasks.

  The generator can estimate the depth field for stereo pairs of images.

  The methods described above can be applied to any suitable human or animal anatomy. The method may be applied to the processing of image data acquired using any suitable imaging procedure type, such as any suitable modality, sequence, acquisition type or processing technique.

  The method has been described above in connection with images, such as, for example, reference images, transformed images, and difference images. The operations described above as being performed on images may actually be performed on a set of image data representing these images. For example, operations may be performed on data comprising a set of pixel or voxel locations and associated luminance. In many cases, operations are performed on image data without the corresponding image being displayed.

  Specific circuits have been described herein. In some embodiments, one or more functions of these circuits may be provided by a single processing resource or other component, or the functions provided by a single circuit may be combined. It may be provided by more than one processing resource or other component. References to a single unit encompass such components, regardless of whether the components providing the functionality of the circuit are remote from one another, and references to the circuits refer to their components. Includes a single component that provides the functionality of the circuit.

  While specific embodiments have been described, these embodiments are presented by way of example only and are not intended to limit the scope of the present invention. In fact, the novel methods and systems described herein may be implemented in various other forms. Moreover, various omissions, substitutions and changes in the form of the methods and systems described herein may be made without departing from the spirit of the invention. The appended claims and their equivalents are intended to cover such forms or variations that fall within the scope of the present invention.

DESCRIPTION OF SYMBOLS 10 ... Image data processing apparatus, 12 ... Calculation apparatus, 14 ... Scanner, 16 ... Display screen, 18 ... Input device, 20 ... Data store, 22 ... Processing circuit, 22 ... Processing apparatus, 24 ... Training circuit, 26 ... Registration Circuit 30 30 Reference image 34 Transformed image 36 Generator 56 56 Identifier 66 66 Generator 76 Identifier 124 124 Generator 130a Identifier 130b Identifier 130

Claims (15)

An acquisition unit for receiving the first image data and the second image data;
A generation unit configured to generate a predicted displacement for performing registration processing between the first image data and the second image data;
Equipped with
The generation unit is trained by repeatedly executing the generation of the predicted displacement and training using the identification unit,
The identification unit is trained to distinguish between a predetermined displacement and the predicted displacement;
Medical image processing apparatus characterized by   The medical image processing apparatus according to claim 1, wherein at least one of the generation unit and the identification unit has a neural network.   The generation unit is characterized in that at least one of registration, subtraction, segmentation, atlas-based processing, image fusion, anatomy detection, and lesion detection is performed based on the predicted displacement. The medical image processing apparatus according to claim 1 or 2.   The medical image processing apparatus according to any one of claims 1 to 3, wherein the first image data and the second image data are two-dimensional image data or three-dimensional image data. A learning device for learning an identification unit and a generation unit,
An acquisition unit for receiving a plurality of training image data sets and a plurality of predetermined displacements corresponding to the plurality of training image data sets;
The generation processing for generating the predicted displacement based on the plurality of training image data sets; and the identification processing for identifying the predicted displacement and the plurality of predetermined displacements as the identification processing. A learning unit that repeatedly executes to train the generation unit and the identification unit;
A learning device characterized by   The learning unit is characterized by maximizing or increasing discriminative loss of the generating unit and the identifying unit, and minimizing or reducing a loss function for registration of the plurality of training image data sets. The learning device according to Item 5. The generation process is
Generating, for each of the plurality of training image data sets, the predicted displacement representing a transformation between the further image data set;
The identification unit
Outputting an identification result as to whether the predicted displacement is the predicted displacement generated by the generation unit or the predetermined displacement;
The learning apparatus according to claim 5 or 6, wherein   8. A learning device according to claim 7, wherein the further image data set is synthesized from the training image data set using one of the predetermined displacements.   The learning device according to any one of claims 5 to 7, wherein the learning unit minimizes or reduces an error of the identification unit in the identification of the predetermined displacement and the predicted displacement. . The identification unit receives the predetermined displacement and the predicted displacement from the generator;
The learning device according to any one of claims 5 to 9, wherein the learning unit trains the identification unit to identify the predetermined displacement and the predicted displacement.   The identification unit executes the identification processing using at least one of residual image data, difference image data, similarity measurement, and a distance function in the identification between the predetermined displacement and the predicted displacement. The learning device according to any one of claims 5 to 10.   The learning apparatus according to any one of claims 5 to 11, wherein the identification unit has a multiple identification unit.   The learning apparatus according to any one of claims 5 to 12, wherein the plurality of training image data sets are two-dimensional image data or three-dimensional image data. On the computer
An acquisition function for acquiring the first image data and the second image data;
A generation function for generating a predicted displacement for performing registration processing between the first image data and the second image data;
To achieve
The generation function is trained by repeatedly executing the generation of the predicted displacement and training using the identification function,
The identification function is trained to distinguish between a predetermined displacement and the predicted displacement;
Medical image processing program characterized by A learning program for learning a generation unit and an identification unit, wherein
On the computer
An acquisition function for acquiring a plurality of training image data sets and a plurality of predetermined displacements corresponding to the plurality of training image data sets;
The generation processing for generating the predicted displacement based on the plurality of training image data sets; and the identification processing for identifying the predicted displacement and the plurality of predetermined displacements as the identification processing. A learning function which is repeatedly executed to train the generation unit and the identification unit;
A learning program characterized by realizing