Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/20—Image preprocessing
  • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/0002—Inspection of images, e.g. flaw detection
  • G06T7/0012—Biomedical image inspection
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/40—Extraction of image or video features
  • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
  • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
  • G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
  • G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
  • G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
  • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
  • G06V2201/03—Recognition of patterns in medical or anatomical images
  • G06V2201/033—Recognition of patterns in medical or anatomical images of skeletal patterns
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
  • G06V2201/03—Recognition of patterns in medical or anatomical images
  • G06V2201/034—Recognition of patterns in medical or anatomical images of medical instruments

Definitions

• Orthopedic surgeries often involve implanting one or more orthopedic prostheses into a patient.
• a surgeon may attach orthopedic prostheses to a scapula and a humerus of a patient.
• a surgeon may attach orthopedic prostheses to a tibia and a talus of a patient. Placing an additional or replacement implant component into a joint that is already fitted with an existing implant component may be referred to as a revision surgery or a surgical revision procedure.
• a computing device may segment image data (e.g., computed tomography (CT) image data) of a joint to separate out individual anatomical objects so that shape and size can be visualized.
• the computing device may present to a doctor or other user a 3D image of the patient anatomy, including specific anatomy such as the scapula, the clavicle, the glenoid, the humerus, and the like.
• CT computed tomography
• the implant may, for example, create noise or artifacts in the CT images that make performing segmentation difficult or make the resulting segmentation unreliable.
• This disclosure describes techniques for segmenting image data representative of an orthopedic joint in which one or more implant components have been placed (e.g., for performing revision procedures).
• the example techniques may utilize U- Net based machine learning models trained based on a weighted loss function that weighs differences between voxel labels in a training sub-volume and outputs of the model based on the locations of the voxels relative to patient anatomy.
• a training dataset that includes a plurality of training images (e.g., training volumes).
• the training images may be divided into training sub-volumes that are used to train the machine learning model (e.g., for efficient memory and processing utilization).
• the techniques of this disclosure may help to overcome or avoid the complications created by the noise and artifacts caused by implants and enable a computing device to produce a useful segmentation of a joint, even when the joint includes an implant component.
• having a useful segmentation of tire joint allows a surgeon to plan a revision surgery that has a higher success rate, as compared to situations in which segmentation is of poorer quality.
• the example techniques may provide for better segmentation of bone and implant, allowing for a practical application of surgery- planning.
• this disclosure describes a method for segmentation of image content, the method comprising: obtaining, by a computing system, medical image data for a patient; applying, by the computing system, a machine learning model having a U- Net architecture to generate labels identifying voxels in the medical image data for a bone and identifying voxels in the medical image data for a prosthesis; and outputting, by the computing system, information that identifies the bone and the prosthesis based on the generated labels.
• FIG. 1 is a block diagram illustrating an example system that may be used to implement the techniques of this disclosure.
• FIG. 2 is a block diagram illustrating example components of a planning system, in accordance with one or more techniques of this disclosure.
• FIGS. 3A and 3B are conceptual diagrams illustrating an example region that includes the bone used for weighting in the loss function, in accordance with one or more techniques of this disclosure.
• FIG. 4 is a conceptual diagram illustrating an example boundary zone of a bone used for weighting the loss function, in accordance with one or more techniques of this disclosure.
• FIG. 5 is a conceptual diagram illustrating an example U-Net convolutional neural network in accordance with one or more techniques of this disclosure.
• FIG. 6 is a flowchart illustrating an example operation in accordance with one or more techniques of this disclosure.
• FIG. 7 is a flowchart illustrating an example operation in accordance with one or more techniques of this disclosure
• FIG. 8 is a flowchart illustrating an example operation in accordance with one or more techniques of this disclosure.
• Orthopedic surgeries often involve implanting one or more orthopedic prostheses into a patient.
• a surgeon may attach orthopedic prostheses to a scapula and a humerus of a patient.
• a surgeon may attach orthopedic prostheses to a tibia and a talus of a patient.
• TKA total knee arthroplasty
• a surgeon may attach orthopedic prostheses to a femur and tibia of a patient.
• a prothesis also called an implant
• the current prosthesis may become less effective.
• the effectiveness of the prosthesis to reduce such as the prothesis reaching the end-of-life.
• a surgeon performs a revision surgery in which the surgeon may replace or adjust the current prosthesis.
• the surgeon requests images of the patient having the prosthesis that can be fed into a planning system, such as an automated planning system, as one example.
• a planning system performs a segmentation procedure on the image data from the images.
• the planning system inputs the image data and generates information that differentiates (e.g., segments) between the prosthesis and patient anatomy (e.g., bone or soft tissue).
• the planning system may output a three-dimensional virtual image in which the prosthesis, respective bones, soft tissue, etc. are presented in different colors so that the surgeon can view the current patient state (e.g., where the prosthesis is relative to a target bone) .
• the planning system uses the segmentation procedure to identify the bones and prosthesis in the image data, so that the planning system can properly color code the bones and prosthesis.
• the prosthesis may cast shadows in the image data.
• the prosthesis may create streaking in the image data. That is, the imaging of the prosthesis may create “noise” in the image data (e.g., such as shadows or streaking). The noise may impact the ability of the planning system to properly segment the bones and the prosthesis.
• the planning system may be unable to properly resolve whether a voxel in the image data should be labeled as belonging to the bone or the prosthesis, and if a bone, which bone type (e.g., scapula or humerus), and if a prosthesis, which prosthesis type (e.g., humeral implant or glenoid implant).
• a bone which bone type (e.g., scapula or humerus)
• a prosthesis which prosthesis type (e.g., humeral implant or glenoid implant).
• a computing system may obtain medical image data for a patient.
• the computing system may apply a machine learning model having a U-Net architecture to generate labels identifying voxels in the medical image data for a bone and identifying voxels in the medical image data for a prosthesis.
• the labels may further identify bone type, such as whether a voxel is for the humerus or the scapula.
• the labels may also identify prothesis type, such as whether a voxel is for the humeral implant or the glenoid implant.
• a computing system may train a neural network having the U-Net architecture to generate the machine learning model.
• a complete dataset used for training a neural network to generate the machine learning model may include a training dataset, a validation dataset, and a test dataset.
• the training dataset includes a plurality of training images (e.g., training volumes).
• the computing system may divide a training image into a plurality of training sub-volumes.
• the computing system may better utilize memory and processing capabilities. That is, training using the training dataset or training volume may be memory and processing intensive.
• the example techniques of dividing into sub-volumes allow for better memory and processing utilization, while generating labels that accurately identify bone and prothesis.
• the computing system may perform a plurality of training iterations.
• the computing system may receive a current training sub-volume (e.g., 192x192x192 voxels of attaining image) having known labels (e.g., ground-truths) identifying voxels in the current training sub-volume for one or more bones and identifying voxels in the current training sub-volume for one or more prostheses.
• the known labels may also indicate bone and prosthesis type (e.g., a voxel corresponds to the humerus or scapula, or a voxel corresponds to the glenoid implant or humeral implant).
• the computing system may apply a machine learning model to the current training sub-volume based on weights to generate a set of labels identifying voxels in the current training sub-volume as the one or more bones and identifying voxels in the current training sub-volume as the one or more prostheses.
• the labels may also include bone or prothesis type.
• a training dataset may include a plurality of training images (e.g., training volumes).
• the computing system may divide attaining image into a plurality of training sub-volumes.
• the computing system may train the neural network to generate the machine learning model by providing the training sub-volumes as inputs to the machine learning model, and comparing the outputs from the machine learning model to the known labels.
• the computing system may perform similar operations for each of the training sub-volumes.
• the computing system may determine a difference between the known labels and the labels generated by the machine learning model.
• the techniques to determine the difference between the known labels and the labels generated by the machine learning model are referred to as applying a loss function.
• the loss function include generalized loss functions like Binary cross-entropy (BCE) and specialized loss functions like Dice loss or focal loss.
• BCE Binary cross-entropy
• Dice loss or focal loss The result of the applying a loss function is a loss value.
• the computing system may generate a loss value from applying a weighted loss function that weights differences between the known labels and the generated set of labels (e.g., labels generated by the machine learning model) based on locations of voxels associated with the known labels and the generated set of labels relative to patient anatomy. For instance, tire computing system may determine one or more sub-loss values, where each sub-loss value is indicative of a difference between the known labels and the generated set of labels for voxels located at a particular location relative to patient anatomy.
• a first sub-loss value may be indicative of a difference between the known labels and the generated set of labels for voxels that form a region that includes the bone (e.g., a region that includes a joint of the bone such as a region that is of clinician interest), and a second sub-loss value may be indicative of a difference between the known labels and the generated set of labels for voxels along a boundary of the bone.
• a third sub-loss value may be indicative of the overall difference between the known labels and the generated set of labels.
• the computing system may weight one or more of the sub-loss values differently to generate the loss value.
• the computing system may update the current weights for the machine learning model based on the loss value. This may complete one iteration of the plurality of training iterations.
• the computing system may then repeat the above example techniques for a next training sub-volume (e.g., the next 192x192x192 voxels of the training image).
• the computing system may repeat this process using training sub-volumes from multiple training images that form the training dataset.
• the completion of the performance of the plurality of training iterations may be considered as a completion of a training epoch.
• the computing system may be configured to perform a plurality of training epochs. For instance, in the first training epoch, the computing system may divide training images into a plurality of training sub-volumes (e.g., 192x192x192 voxels). For a subsequent training epoch, the computing system may re-divide the training images into another plurality of training sub-volumes, and perform the plurality of training iterations on each of the other plurality of training sub-volumes. Performing a plurality of training epochs is provided as an example, and should not be considered limiting.
• the result of the training iterations and/or training iterations with training epochs is the machine learning model.
• the computing system may obtain medical image data (e.g., not training sub-volumes) and apply the machine learning model having the U-Net architecture to generate labels identifying voxels in the medical image data for a bone and identifying voxels in the medical image data for a prosthesis.
• the computing system may then output information that identifies the bone and the prosthesis based on the generated labels.
• the computing system may be able to better segment the prosthesis and the bone. That is, the segmentation procedure, using the example techniques described in this disclosure, may better account for the noise in the image data (e.g., due to the prosthesis).
• the result of the segmentation procedure may be a more accurate segmentation as compared to other techniques that do not utilize the example techniques described in this disclosure.
• the surgeon or an automated planning system may determine a surgical procedure that is more likely to result in successfully addressing patient needs.
• FIG. 1 is a block diagram illustrating an example system 100 that may be used to implement the techniques of this disclosure.
• FIG. 1 illustrates computing system 102, which is an example of one or more computing devices that are configured to perform one or more example techniques described in this disclosure.
• Computing system 102 may include various types of computing devices, such as server computers, personal computers, smartphones, laptop computers, and other types of computing devices.
• computing system 102 includes multiple computing devices that communicate with each other.
• computing system 102 includes only a single computing device.
• Computing system 102 includes processing circuitry 104, storage system 106, a display 108, and a communication interface 110.
• Display 108 is optional, such as in examples where computing system 102 is a server computer.
• processing circuitry 104 examples include one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware or any combinations thereof.
• processing circuitry 104 may be implemented as fixed- function circuits, programmable circuits, or a combination thereof.
• Fixed-function circuits refer to circuits that provide particular functionality and are preset on the operations that can be performed.
• Programmable circuits refer to circuits that can be programmed to perform various tasks and provide flexible functionality in the operations that can be performed. For instance, programmable circuits may execute software or firmware that cause the programmable circuits to operate in the manner defined by instructions of the software or firmware.
• Fixed-function circuits may execute software instructions (e.g., to receive parameters or output parameters), but the types of operations that the fixed-function circuits perform are generally immutable.
• the one or more of the units may be distinct circuit blocks (fixed-function or programmable), and in some examples, the one or more units may be integrated circuits.
• processing circuitry 104 is dispersed among a plurality of computing devices in computing system 102. In some examples, processing circuitry 104 is contained within a single computing device of computing system 102.
• Processing circuitry 104 may include arithmetic logic units (ALUs), elementary function units (EFUs), digital circuits, analog circuits, and/or programmable cores, formed from programmable circuits.
• ALUs arithmetic logic units
• EFUs elementary function units
• storage system 106 may store the object code of the software that processing circuitry 104 receives and executes, or another memory within processing circuitry 104 (not shown) may store such instractions.
• Examples of the software include software designed fbr surgical planning, including image segmentation.
• Storage system 106 may be formed by any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices.
• DRAM dynamic random access memory
• SDRAM synchronous DRAM
• MRAM magnetoresistive RAM
• RRAM resistive RAM
• Examples of display 108 include a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.
• storage system 106 may include multiple separate memory devices, such as multiple disk drives, memory modules, etc., that may be dispersed among multiple computing devices or contained within the same computing device.
• Communication interface 110 allows computing system 102 to communicate with other devices via network 112.
• computing system 102 may output medical images, images of segmentation masks, and other information for display.
• Communication interface 110 may include hardware circuitry that enables computing system 102 to communicate (e.g., wirelessly or using wires) to other computing systems and devices, such as a visualization device 114 and an imaging system 116.
• Network 112 may include various types of communication networks including one or more wide-area networks, such as the Internet, local area networks, and so on. In some examples, network 112 may include wired and/or wireless communication links.
• Visualization device 114 may utilize various visualization techniques to display image content to a surgeon.
• visualization device 114 is a computing device having computer monitor or display screen.
• visualization device 114 may be a computing device that a surgeon uses for planning surgeries.
• visualization device 114 may be a mixed reality (MR) visualization device, virtual reality (VR) visualization device, holographic projector, or other device fbr presenting extended reality (XR) visualizations.
• MR mixed reality
• VR virtual reality
• XR extended reality
• visualization device 114 may be a Microsoft HOLOLENSTM headset, available from Microsoft Corporation, of Redmond, Washington, USA, or a similar device, such as, for example, a similar MR visualization device that includes waveguides.
• the HOLOLENS TM device can be used to present 3D virtual objects via holographic lenses, or waveguides, while permitting a user to view actual objects in a real-world scene, i.e., in a real-world environment, through the holographic lenses.
• Computing system 102 and/or visualization device 114 may utilize visualization tools that are available to utilize patient image data to generate three-dimensional models of bone contours, segmentation masks, or other data to facilitate preoperative planning. These tools may allow surgeons to design and/or select surgical guides and implant components that closely match the patient’s anatomy. These tools can improve surgical outcomes by customizing a surgical plan for each patient.
• An example of such a visualization tool is the BLUEPRINT TM system available from Stryker Corp. The surgeon can use the BLUEPRINT TM system to select, design or modify appropriate implant components, determine how best to position and orient the implant components and how to shape the surface of the bone to receive the components, and design, select or modify surgical guide tool(s) or instruments to carry out the surgical plan.
• the information generated by the BLUEPRINT TM system may be compiled in a preoperative surgical plan for the patient that is stored in a database at an appropriate location, such as storage system 106, where the preoperative surgical plan can be accessed by the surgeon or other care provider, including before and during the actual surgery.
• Imaging system 116 may comprise one or more devices configured to generate medical image data.
• imaging system 116 may include a device for generating CT images.
• imaging system 116 may include a device for generating magnetic resonance imaging (MRI) images.
• imaging system 116 may include one or more computing devices configured to process data from imaging devices in order to generate medical image data.
• the medical image data may include a 3D image of one or more bones of a patient.
• imaging system 116 may include one or more computing devices configured to generate the 3D image based on CT images or MRI images.
• computing system 102 may include one or more computing devices configured to generate the medical image data based on data from devices in imaging system 116.
• Storage system 106 of computing system 102 may store instractions that, when executed by processing circuitry 104, cause computing system 102 to perform various activities.
• storage system 106 may store instructions that, when executed by processing circuitry 104, cause computing system 102 to perform activities associated with a planning system 118.
• this disclosure may simply refer to planning system 118 or components thereof as performing the activities, or may directly describe computing system 102 as performing the activities.
• planning system is used to generically indicate the system that performs example techniques.
• planning system 118 may be executing completely within a device with which a surgeon interfaces.
• planning system 118 may be executing completely in a cloud-based network. That is, in such examples, a device with which a surgeon interfaces may upload information to the cloud that planning system 118 receives. Planning system 118 receives such information, performs example operations, and outputs the results to the device with which the surgeon interfaces.
• planning system 118 may be a combination of cloud-based and executing on the device with which the surgeon interfaces. For instance, surgical planning may be performed on the device with which the surgeon interlaces, but the segmentation may be performed in the cloud.
• the example techniques should not be considered limited to a specific configuration.
• medical image data 126 may include CT images of bones of the patient or 3D images of bones of the patient based on CT images.
• the term “bone” may refer to a whole bone or a bone fragment.
• medical image data 126 may include MRI images of one or more bones of the patient or 3D images based on MRI images of the one or more bones of the patient.
• medical image data 126 may include ultrasound images of one or more bones of the patient.
• Segmentation data 128 may be information indicative of particular patient anatomy or prosthesis. For instance, the patient may have had a prosthesis implanted during a previous surgical procedure. However, due to disease progression, end of life of the prothesis, or other reasons, a revision surgery may be in order as the effectiveness of the prothesis may have reduced. Accordingly, in some examples, medical image data 126 includes images of the bones and images of the current prothesis.
• Segmentation data 128 includes data that indicates which portion of medical image data 126 corresponds to bone or other patient anatomy, and which portion of medical image data 126 corresponds to the prosthesis.
• the data may be various forms such as labels, color, etc. that differentiates voxels in medical image data 126 that correspond to one or more bones or other patient anatomy, and voxels in medical image data 126thatcorrespondto one or more prostheses.
• the data may also label bone type or prosthesis type (e.g., humerus or scapula, or glenoid implant or humeral implant).
• a voxel may represent a 0.6 mm side cube in the image data, but other sizes are possible.
• prostheses add noise to medical image data 126.
• the noise may be a shadow cast by a prosthesis in medical image data, or high intensity (e.g., high luminance) streaking in medical image data 126.
• Such noise impacts the ability of planning system 118 to differentiate between voxels in medical image data 126. That is, the noise may impact the accuracy of segmentation data 128.
• the noise may cause planning system 118 to incorrectly classify a voxel or a group of voxels in medical image data 126 as corresponding to a bone, when the voxel or group of voxels in medical image data 126 correspond to a prosthesis, and vice-versa.
• this disclosure describes example techniques to determine segmentation data 128 more accurately, such as based on applying a machine learning model (e.g., having a U- Net architecture or other architecture) to image data 126 to generate labels identifying voxels in medical image data 126 for a bone and identifying voxels in medical image data 126 for a prosthesis.
• the labels may also identify bone type (e.g., bone name or classification, like humerus, scapula, etc.) and/or prothesis type (e.g., where the prothesis is supposed to be implanted such as glenoid implant, humeral implant, etc.).
• Catalog 122 may include data regarding a plurality of orthopedic prostheses.
• catalog 122 contains data regarding a plurality of orthopedic prostheses available for implantation in the patient.
• the orthopedic prostheses may have different predetermined sizes and predetermined shapes.
• the orthopedic prostheses may include orthopedic prostheses available from one or more medical device suppliers.
• the orthopedic protheses are not patient-specific. In other words, in such examples, shapes and sizes of the orthopedic prostheses are not individually customized to the anatomies of specific patients.
• Catalog 122 is provided as an example, and may not be necessary in all cases.
• Planning system 118 may be configured to assist a surgeon with planning an orthopedic surgery that involves removal of a current orthopedic prosthesis and implantation of an orthopedic prosthesis into a patient, or adjustment of a current orthopedic prosthesis. For instance, as described above, planning system 118 may generate segmentation data 128 in accordance with one or more techniques of this disclosure. For example, planning system 118 may apply a machine learning model having a U-Net architecture to generate labels identifying voxels in the medical image data 126 for a bone and identifying voxels in the medical image data 126 for a prosthesis. Planning system 118 may output information that identifies the bone and the prosthesis based on the generated labels.
• planning system 118 may output, for display on display 108 and/or visualization device 114, image content (e.g., as a 3D virtual image) that identifies the bone and the prosthesis based on the generated labels.
• the bone may be at least one of a humerus or a glenoid.
• the prosthesis may be at least one of a humeral implant or a glenoid implant.
• the labels identifying voxels in the medical image data for the bone may be labels identifying voxels in the image data for the humerus, and labels identifying voxels in the image data for the glenoid.
• the labels identifying voxels in the image data for the prosthesis may be labels identifying voxels in the image data for the humeral implant, and labels identifying voxels in the image data for the glenoid implant.
• FIG. 2 is a block diagram illustrating example components of planning system 118, in accordance with one or more techniques of this disclosure.
• the components of planning system 118 include a machine learning model 200, a prediction unit 202, a training unit 204, and a segmentation unit 206.
• planning system 118 may be implemented using more, fewer, or different components.
• training unit 204 may be omitted in instances where machine learning model 200 has already been trained.
• one or more of the components of planning system 118 are implemented as software modules.
• the components of FIG. 2 are provided as examples and planning system 118 may be implemented in other ways.
• Machine learning model 200 may have a U-Net architecture.
• the example techniques described in this disclosure should not be considered limited to a U- Net architecture, and other example architectures, such as a V-Net architecture, are possible.
• Prediction unit 202 may apply machine learning model 200 to generate labels identifying voxels in the medical image data for one or more bones and identifying voxels in the medical image data for one or more prostheses.
• the medical image data for the patient may include a plurality of input images (e.g., CT images or MRI images, etc.).
• each of the input images may have a width dimension and a height dimension, and each of the input images may correspond to a different depthdimension layer in a plurality of depth-dimension layers.
• the plurality of input images may be conceptualized as a stack of 2D images, where the positions of individual 2D images in the stack in the depth dimension.
• prediction unit 202 may generate the stack of 2D image (e.g., a 3D image) based on a single 2D image.
• prediction unit 202 may provide the input images as input to machine learning model 200.
• Machine learning model 200 may output labels in the voxels of the medical image data that identifies which voxels correspond to the bone (and possibly bone type) and which voxels correspond to the prosthesis (and possibly prothesis type).
• Machine learning model 200 may output additional labels as well, such as labels for soft tissue.
• machine learning model 200 may not be necessary for machine learning model 200 to generate a label for every' single voxel. It may be possible that machine learning model 200 generates a label for a group of voxels.
• prediction unit 202 performs one or more preprocessing operations on the input images prior to providing the input images as input to machine learning model 200.
• prediction unit 202 may normalize voxel sizes of the input images, flip one or more of the input images to represent the same laterality, or perform other operations on the input images.
• prediction unit 202 may crop intensify values of the medical image data to between a maximum intensify value (e.g., 1800 units of the Hounsfield unit scale) and a minimum intensify value (e.g., -500 units of the Hounsfield unit scale). For instance, each voxel may be associated with an intensify' (e.g., luminance) value. The range of the intensify values may be relatively large. By cropping the intensify values to a maximum and minimum intensity value, it may be possible to process the medical image data. For instance, prediction unit 202 may set all intensity values greater than maximum intensity value equal to one, and set all intensity values less than the minimum intensity value equal to zero. Prediction unit 202 may scale all intensity values between the maximum and minimum intensity value to a value between zero and one.
• a maximum intensify value e.g. 1800 units of the Hounsfield unit scale
• a minimum intensify value e.g., -500 units of the Hounsfield unit scale.
• each voxel may be
• the medical image data for the patient may include a 3D input image, which may be based on CT images, MRI images, or another type of image.
• the 3D input image may include a 3D matrix of voxels having luminance (e.g., brightness) attributes.
• prediction unit 202 may provide the 3D input image as input to machine learning model 200.
• Machine learning model 200 may be implemented in one of a variety of ways.
• machine learning model 200 may be a convolutional neural network.
• the convolutional neural network has a U-net or V-net architecture.
• An example U-net architecture is described with respect to FIG. 5, below.
• the convolutional neural network may generate one or more generate labels identifying voxels in the medical image data for a bone and identifying voxels in the medical image data for a prosthesis.
• machine learning model 200 may generate a volume of image data (e.g., 192x192x192 voxels), and prediction unit 202 may assemble each of the volumes to form a more complete 3D image of patient anatomy having one or more prostheses.
• Segmentation unit 206 may receive the 3D medical image data. In some examples, segmentation unit 206 may use the labels to apply different colors to voxels of the 3D medical image data to differentiate bones and prostheses shown in the 3D medical image data.
• machine learning model 200 may provide further granularity regarding the labels.
• the bone may be at least one of a humerus or a glenoid
• the prosthesis may be at least one of a humeral implant or a glenoid implant.
• Machine learning model 200 may generate the labels including labels identifying voxels in the image data for the humerus, labels identifying voxels in the image data for the glenoid, labels identifying voxels in the image data for the humeral implant, and labels identifying voxels in the image data for the glenoid implant.
• Training unit 204 may train machine learning model 200. For instance, training unit 204 may receive a training dataset that includes a plurality of training images (e.g., training volumes). Training unit 204 may divide a training image into a plurality of training sub-volumes. As one example, training unit 204 may receive a set of training images. Each of the training images may be an image of an individual (e.g., not necessarily the patient for whom the revision surgery is being planned) having a prosthesis. Humans may review the training images and manually identify voxels as bone or prosthesis.
• training images e.g., training volumes
• Training unit 204 may divide a training image into a plurality of training sub-volumes.
• training unit 204 may receive a set of training images. Each of the training images may be an image of an individual (e.g., not necessarily the patient for whom the revision surgery is being planned) having a prosthesis. Humans may review the training images and manually identify voxels as bone or prosthesis.
• Training unit 204 may divide (e.g., randomly or pseudo-randomly) a training image into the plurality of training sub-volumes.
• training unit 204 may pseudo-randomly divide the training image such that there is at least one voxel corresponding to a bone (e.g., scapula) in each of the training sub-volumes.
• each training sub-volume is of size 192x192x192 voxels; however, other sized training sub-volumes are possible.
• every training sub-volume may include at least one voxel corresponding to a bone.
• a training sub-volume may include voxels for the prothesis, and no voxels for the bone.
• a training sub-volume may include voxels for the bone, and no voxels for the prosthesis.
• a training sub-volume may include voxels both the prothesis and for the bone. It may be possible for a training sub-volume to include voxels for neither the bone nor the prosthesis.
• all training sub-volumes include bone, and no prosthesis.
• machine learning model 200 may label such voxels as “not bone,” which may indicate prosthesis.
• each training sub-volume has known labels identifying voxels in the training sub-volume for the bone and possibly bone type and identifying voxels in the current training sub-volume for the prosthesis and possibly prosthesis type.
• the known labels identifying voxels in the training sub-volume for the bone and identifying voxels in the current training sub-volume for the prosthesis may be considered as the ground-truths or as the expected output data.
• training unit 204 may perform a plurality of training iterations. For instance, each of the training iterations includes receiving one of the training sub-volumes (e.g., a current training sub-volume), and applying machine learning model 200 to the current training sub-volume to generate a set of labels identifying voxels in the current training sub-volume as the bone and identifying voxels in the current training sub-volume as the prosthesis.
• each of the training iterations includes receiving one of the training sub-volumes (e.g., a current training sub-volume), and applying machine learning model 200 to the current training sub-volume to generate a set of labels identifying voxels in the current training sub-volume as the bone and identifying voxels in the current training sub-volume as the prosthesis.
• Training unit 204 may compare the set the labels generated by machine learning model 200 to the expected output data (e.g., the known labels identifying voxels in the training sub-volume as bone or prosthesis). For example, training unit 204 may apply a loss function to the labels generated by machine learning model 200 and the expected output data. As one example, training unit 204 may compare the known labels and the generated set of labels using Dice loss. For instance, training unit 204 may determine a Sorensen-Dice coefficient, or simply Dice loss, which is a statistic used to gauge the similarities of two sets of labels. Gauging the similarities of two sets of labels may be considered as inverse of determining the difference between sets of labels. That is, it may be possible to the determine the difference between sets of labels based on the Dice loss (e.g., as inverse).
• the expected output data e.g., the known labels identifying voxels in the training sub-volume as bone or prosthesis.
• training unit 204 may apply a loss function to the labels generated by machine learning model 200
• training unit 204 may generate a loss value from applying a weighted loss function (e.g., weighted Dice function) that weights differences between the known labels and the generated set of labels (e.g., output data generated from machine learning model 200).
• the weights for the weighted loss function may be based on the relative location of the voxels that are being compared to patient anatomy.
• Training unit 204 may utilize the loss value to update neural network weights of machine learning model 200. For instance, training unit 204 may use the loss value in a backpropagation process to update the weights of machine learning model 200.
• machine learning model 200 may converge to an accurate estimator of the voxels as bone or prosthesis if different regions, relative to patient anatomy, contribute more or less to the calculation of the neural network weights used by machine learning model 200.
• differences between the labels for the voxels in the training sub-volume and the generated set of labels for the voxels generated by machine learning model 200 (e.g., the output data) that form a region that includes the bone may be weighted more in the loss value calculation than other regions.
• differences between the labels for the voxels in the training sub-volume and the generated set of labels for voxels proximate to a boundary' of the bone may be weighted differently for the loss value calculation as compared to labels for the voxels in the region that includes the bone.
• differences between all of the labels for the voxels in the training sub-volume and corresponding labels in the generated set of labels generated by machine learning model may be weighted differently for the loss value calculation as compared to labels for the voxels in the region that includes the bone.
• the loss value generated using the example techniques described in this disclosure may result in training unit 204 utilizing loss values that can be used to update the neural network weights utilized by machine learning model 200 to generate labels more accurately for voxels as bone or prosthesis.
• some techniques may generate the loss value based on a difference between the generated set of labels and the known labels.
• those techniques may result in less optimal updating of the neural network weights, resulting in a machine learning model that is less accurate at labeling voxels as bone or prosthesis, as compared to one or more example techniques described in this disclosure.
• training unit 204 may generate a first subloss value indicative of a difference between the known labels and the generated set of labels for voxels that form a region that includes a joint of the bone.
• the voxels may be for voxels that form a region of particular interest for the clinical application (e.g., such as tire joint or the expected prosthesis location).
• the region may be predefined around the joint of the bone.
• the region may be specified.
• the bone may be the scapula.
• the region that includes the bone may be a 30 millimeter (mm) radius sphere, centered on the center of a glenoid fossa of the scapula.
• the first sub-loss value may be referred to as “Loss for the sphere around the glenoid” that takes into account the mean Dice for the labels contained in a 30 mm radius sphere, centered on the center of the glenoid.
• Training unit 204 may apply a first weight to the first sub-loss value to generate a first weighted sub-loss value. Training unit 204 may generate the loss value based on the first weighted sub-loss value.
• Training unit 204 may generate a second sub-loss value indicative of a difference between the known labels and the generated set of labels for voxels along a boundary of the bone.
• the boundary of the bone may be approximately 2 mm thick band (e.g., + 1 mm) centered on the border.
• the second sub-loss value may be referred to as “Loss for the boundary zone” that takes into account the mean Dice for the labels contained in approximately a 2 mm thick band centered on the border of the labels.
• Training unit 204 may apply a second weight to the second sub-loss value to generate a second weighted sub-loss value. Training unit 204 may generate the loss value based on the second weighted sub-loss value.
• Training unit 204 may generate a third sub-loss value indicative of a difference between the known labels and the generated set of labels for the voxels.
• the third sub-loss value may be referred to as “Loss for the whole mask” that takes into account the mean Dice for the labels in the whole training sub-volume (e.g., the entire 192x192x192 voxels).
• Training unit 204 may apply a third weight to the third sub-loss value to generate a third weighted sub-loss value.
• Training unit 204 may generate the loss value based on the at least one of the first weighted sub-loss value or the second weighted sub-loss value, and the third weighted sub-loss value.
• the first weight is greater than the second and third weights. Accordingly, the difference in the labels for voxels within the region that includes the bone may impact the overall determination of the neural network weights more than the difference in the labels for voxels along the boundary or difference in the labels for all voxels in the training sub-volume and generated set of labels.
• the first weight equals 0.4
• the second weight equals 0.3
• the third weight equals 0.3
• the above example describes three sub-loss values. However, the example techniques are not so limited. Not all three sub-loss values are needed in every example. Also, there may be more sub-loss value than the examples provided above. In general, training unit 204 may utilize sub-losses from various locations relative to patient anatomy, and weigh each sub-loss with different factors, to determine die loss value, and the above is one example.
• first, second, and third weights are provided merely as an example, and should not be considered limiting. In some examples, it may be possible to update one or more of the first weight, second weight, or third weight during the training of machine learning model 200. As an example, as described in more detail, in some examples, training unit 204 may train machine learning model 200 in training epochs. In one or more examples, the first, second, and/or third weight may be different for different training epochs.
• training unit 204 may receive a current training sub-volume (e.g., 192x192x192 voxels of the training image) having known labels identifying voxels in the current training sub-volume for the bone and identifying voxels in the current training sub-volume for the prosthesis.
• Training unit 204 may apply machine learning model 200 to the current training sub-volume based on neural network weights to generate a set of labels identifying voxels in the current training sub-volume as the bone and identifying voxels in the current training sub-volume as the prosthesis.
• Training unit 204 may generate a loss value from applying a weighted loss function that weights differences between the known labels and the generated set of labels based on locations of voxels associated with the known labels and the generated set of labels relative to patient anatomy (e.g., the region that includes the bone and the boundary' along the bone as examples of locations of voxels relative to patient anatomy). For instance, training unit 204 may determine the plurality' of sub-loss values, weigh the plurality of sub-loss values, and add the result to generate the loss value. Training unit 204 may update the neural network weights based on the loss value.
• next training sub-volume e.g., next 192x192x192 voxel
• Training unit 204 may then apply the machine learning model 200 to the current training sub-volume based on the neural network weights (e.g., as updated from the previous training iteration) to generate a set of labels identifying voxels in the current training sub-volume as the bone and identifying voxels in the current training sub-volume as the prosthesis.
• Training unit 204 may generate a loss value from applying a weighted loss function that weighs differences between the known labels and the generated set of labels based on locations of voxels associated with the known labels and the generated set of labels relative to patient anatomy (e.g., the region that includes the bone and the boundary' along the bone as examples of locations of voxels relative to patient anatomy). Training unit 204 may update the neural network weights based on the loss value. Training unit 204 may repeat these operations for each of the plurality of training sub-volumes.
• the training iterations are described as one example.
• two or more training sub-volumes may be processed in parallel, and such parallel processing may be one iteration.
• the neural network weights generated from each of the parallel processing may be compared, and one of the neural network weights may be selected, or some other processing may be performed to determine neural network weights.
• training unit 204 may perform on the training sub-volumes.
• one set of iterations that training unit 204 performs on the training sub-volumes may be considered as a training epoch.
• training unit 204 may perform operations for a plurality of training epoch, where each epoch includes a plurality of training iterations.
• training unit 204 may divide a training image into a plurality of training sub-volumes.
• Each of the training sub-volumes includes respective voxels of the training image (e.g., 192x192x192 voxels).
• the plurality' of training sub-volumes is a first plurality of training sub-volumes.
• training unit 204 may be configured to perform a first training epoch.
• training unit 204 may be configured to perform of a plurality' of training epochs. For instance, to perform each of the plurality of training epochs, training unit 204 may be configured to re-divide the training images into another plurality of training sub-volumes, and perform the plurality of training iterations on each of the other plurality of training sub-volumes. The re-divided training images may result in different plurality of training sub-volumes due to the random or pseudo-random nature of the dividing of the training image.
• training unit 204 may generate machine learning model 200 that provides accurate labeling of bone and prosthesis. As described above, performing multiple training epochs is not required in every example.
• the above examples are described with glenoid, humerus, and respective prostheses for ease.
• the example techniques may be extended to other bones and prostheses.
• the weighted loss function may be weighted differently, and there may be other locations (e.g., other than region that includes bone and region along a boundary) that are of interest.
• transfer learning may be used fortraining the machine learning model 200.
• a first machine learning model may be trained for a first type of orthopedic prostheses (e.g., a first type of glenoid implant, a first type of tibial implant, a first type of femoral implant, etc.).
• Transfer learning may be used for training a second machine learning model for a second type of orthopedic prostheses (e.g., a second type of glenoid implant, a second type of tibial implant, a second type of femoral implant, etc.).
• Training unit 204 may then train the second machine learning model using training data.
• the second machine learning model may be trained in part using transfer learning from the first machine learning model.
• training unit 204 may perform a forward pass on the machine learning model 200 using a training subvolume, generated from dividing a training image, as input to machine learning model 200. Training unit 204 may then perform a process that compares tire resulting output of machine learning model 200 (e.g., generated set of labels for the training sub-volume) to the corresponding expected output data (e.g., the known labels for the voxels of the training sub-volume). As described above, the comparison may be generating a loss value from applying a weighted loss function that weighs differences between the known labels and the generated set of labels based on locations of voxels associated with the known labels and the generated set of labels relative to patient anatomy.
• tire resulting output of machine learning model 200 e.g., generated set of labels for the training sub-volume
• expected output data e.g., the known labels for the voxels of the training sub-volume
• Training unit 204 may then perform a backpropagation process based on the comparison to adjust parameters of machine learning model 200 (e.g., neural network weights of neurons of machine learning model 200). Training unit 204 may repeat this process with other training sub-volume. For instance, training unit 204 may perform multiple training iterations. In examples where a training iteration is one training epoch, training unit 204 may perform multiple training epochs based on re-dividing the training image to generate another plurality of training sub-volumes, and performing the training iterations on the other plurality of training sub-volumes. Training unit 204 may use some of the training sub-volumes for validation of machine learning model 200.
• parameters of machine learning model 200 e.g., neural network weights of neurons of machine learning model 200.
• Training unit 204 may repeat this process with other training sub-volume. For instance, training unit 204 may perform multiple training iterations. In examples where a training iteration is one training epoch, training unit 204 may perform multiple training ep
• FIGS. 3A and 3B are conceptual diagrams illustrating an example region that includes the bone used for weighting in the loss function, in accordance with one or more techniques of this disclosure.
• training unit 204 may generate a first sub-loss value indicative of a difference between the known labels and the generated set of labels for voxels that form a region that includes the bone.
• FIGS. 3A and 3B illustrate region 300 which is an example of the region used to generate the first sub-loss value.
• region 300 may be a 30 mm radius sphere, centered on the center of the glenoid.
• region 300 may form a region that includes a joint of the bone.
• the voxels may be for voxels that form a region of particular interest for the clinical application (e.g., such as the joint or the expected prosthesis location).
• the region may be predefined around the joint of the bone. In some examples, the region may be specified.
• FIG. 4 is a conceptual diagram illustrating an example boundary' zone of a bone used for weighting the loss function, in accordance with one or more techniques of this disclosure.
• training unit 204 may generate a second sub-loss value indicative of a difference between the known labels and the generated set of labels for voxels along a boundary of the bone.
• FIG. 4 illustrates boundary 400A, which is one example of the boundary of the bone, and boundary 400B, which is another example of the boundary of the bone.
• the voxels along the boundary of the bone may be considered as the voxels between boundary 400A and boundary 400B.
• the boundary of the bone may be approximately 2 mm thick band (e.g., + 1 mm) centered on the border and between the bone and boundary 400A, between the bone and boundary 400B, or between boundary 400A and 400B.
• FIG. 5 is a conceptual diagram illustrating an example U-Net convolutional neural network (CNN) 500 in accordance with one or more techniques of this disclosure.
• medical image data may include a 3D image of one or more bones of the patient, such as the bones of the ankle, knee, shoulder, or other parts of the anatomy of the patient.
• Prediction unit 202 (or another system) may generate the 3D image based on a stack of 2D images (e.g., CT images) arranged along a depth dimension. The location of each pixel in the stack of 2D images may be a voxel.
• the 3D image may be a CT volume scan.
• each of the cuboids may represent a multi- dimensional array. For each cuboid, the number above the cuboid indicates an example number of dimensions of the multi-dimensional array represented by the cuboid.
• a left branch of the “U” shape of U-Net CNN may be referred to as an “encoder.”
• the encoder extracts high level features in a 3D input image.
• a right branch of the “U” shape may be referred to as a “decoder.”
• the decoder reconstructs information at a voxel level.
• An output of U-Net CNN 500 may include labels identifying voxels in the medical image data for a bone and identifying voxels in the medical image data for a prosthesis.
• the encoder is composed of a succession of convolution, batch normalization, and activation (“ReLU,” “PReLU,” and “Leaky ReLU”) layers (represented in FIG. 5 as white filled triangles), and down sampling operations (represented in FIG. 5 as down-forward pointing arrows) at each of levels 502A-502C (collectively, “levels 502”).
• a transition level 504 between the levels 502 of the encoder includes a series of convolution layers.
• the convolutions at each level may use a 3x3x3 filter. In other examples, the number of channels, number of levels, filter size, and stride length may be different.
• the U-Net CNN 500 may include 5 layers with 2 residual units of 2 convolutions each.
• a “residual unit” is a successive application of convolution, normalization, and parametric rectified linear unit for activation function.
• the last layer of the residual unit sums the input to the residual unit with the output of the residual unit.
• each of the 5 layers may have skip connections (e.g., bridges) between them.
• the decoder may be symmetrically built where the down sampling operations are replaced by “Up” convolutions, which may include doubling the size of the feature maps at each level of levels 506A-506C (collectively, “levels 506”).
• U-net CNN 500 includes bridges 508 that enable connections between some high-level features that are extracted to help the reconstruction at the voxel level in the decoder.
• the input arrays for each of levels 506 may be a concatenation of data generated by an up-convolution layer and data derived from a corresponding level of the encoder.
• a final layer of level 506A of the decoder may be a convolution layer (represented in FIG. 5 as a black-filled triangle).
• the output of the decoder may include a plurality of labels identifying voxels in the medical image data for a bone and identifying voxels in the medical image data for a prosthesis
• the layer before the last layer in the decoder may provide confidence values for each of the possible label values.
• tire next-to-last layer may, for each voxel, provide a confidence value that the voxel corresponds to a bone and a confidence value that the voxel corresponds to a prosthesis.
• the last layer of the decoder may determine, for each voxel, which label of the voxel has the highest confidence value and output the determined label. For instance, the last layer may apply a soflmax function to the confidence values.
• the bone may be at least one of a humerus or a glenoid
• the prosthesis may be at least one of a humeral implant or a glenoid implant.
• the labels identifying voxels in the medical image data for the bone may include labels identifying voxels in the image data for the humerus, and labels identifying voxels in the image data for the glenoid.
• the labels identifying voxels in the medical image data for the prosthesis may include labels identifying voxels in the image data for the humeral implant, and labels identifying voxels in the image data for the glenoid implant.
• additional data may be included in the input to level 506C of U-Net CNN 500.
• information regarding biomechanics of the patient may be included in the input to level 506C. Including the information regarding the biomechanics may help U-Net CNN 500 generate labels identify' voxels in the medical image data for a bone and identify voxels in the medical image data for a prosthesis.
• Training unit 204 may train U-Net CNN 500 or similar machine learning models.
• training unit 204 may train U-Net CNN 500 to generate machine learning model 200 in accordance with one or more example techniques described in this disclosure.
• FIG. 6 is a flowchart illustrating an example operation in accordance with one or more techniques of this disclosure.
• Prediction unit 202 of planning system 118 may obtain medical image data 126 for a patient (600).
• prediction unit 202 (or another system) may generate the 3D image based on a stack of 2D images (e.g., CT images) arranged along a depth dimension. The location of each pixel in the stack of 2D images may be a voxel. Thus, the 3D image may be a CT volume scan.
• 2D images e.g., CT images
• Prediction unit 202 may apply machine learning model 200 having a U-Net architecture to generate labels identifying voxels in medical image data 126 for a bone and identifying voxels in the medical image data 126 for a prosthesis (602).
• machine learning model 200 may be configured to provide more granular information.
• the bone may be at least one of a humerus or a glenoid
• the prosthesis may be at least one of a humeral implant or a glenoid implant.
• the labels identifying voxels in the medical image data 126 for the bone may include labels identifying voxels in the image data for the humerus, and labels identifying voxels in the image data for the glenoid.
• the labels identifying voxels in the image data 126 for the prosthesis may include labels identifying voxels in the image data for the humeral implant, and labels identifying voxels in the image data for the glenoid implant.
• Segmentation unit 206 may output information that identifies bone and prosthesis based on the generated labels (604). For example, segmentation unit 206 may receive the generated labels for the voxels, and may output information that a surgeon or other medical professional can view. For instance, segmentation unit 206 may output, for display, image content (e.g., 3D virtual image) that identifies the bone and the prosthesis (e.g., as differently color coded) based on the generated labels.
• image content e.g., 3D virtual image
• FIG. 7 is a flowchart illustrating an example operation in accordance with one or more techniques of this disclosure.
• FIG. 7 illustrates an example operation of training a neural network having the U-Net architecture to generate machine learning model 200.
• training the neural network may include training unit 204 performing a plurality of training iterations. Performing each of the training iterations may include the example techniques illustrated in FIG. 7.
• training unit 204 may receive a current training sub-volume having known labels identifying voxels in the current training sub-volume for the bone and identifying voxels in the current training sub-volume for the prosthesis (700).
• the current training sub-volume may be one of a plurality of training subvolumes that training unit 204 generates by dividing training images into the training subvolumes (e.g., 192x192x192 random or pseudo-random voxels of the training image).
• Training unit 204 may apply the machine learning model 200 to the current training sub-volume based on neural network weights to generate a set of labels identifying voxels in the current training sub-volume as the bone and identifying voxels in the current training sub-volume as the prosthesis (702). That is, training unit 204 maycause machine learning model 200 to generate the set of labels as output data that can be compared to the expected output data (e.g., ground-truths such as the known labels identifying voxels as bone or prosthesis).
• the expected output data e.g., ground-truths such as the known labels identifying voxels as bone or prosthesis.
• Training unit 204 may generate a loss value from applying a weighted loss function that weights differences between the known labels and the generated set of labels based on locations of voxels associated with the known labels and the generated set of labels relative to patient anatomy (704).
• training unit 204 may generate a first sub-loss value indicative of a difference between the known labels and the generated set of labels for voxels that form a region that includes a joint of the bone, or other regions of interest for the clinical application (e.g., a 30 mm sphere centered on the glenoid), apply a first weight to the first sub-loss value to generate a first weighted sub-loss value, and generate the loss value based on the first weighted sub-loss value.
• a first sub-loss value indicative of a difference between the known labels and the generated set of labels for voxels that form a region that includes a joint of the bone, or other regions of interest for the clinical application (e.g., a 30 mm sphere centered on the glenoid)
• training unit 204 may generate a second sub-loss value indicative of a difference between the known labels and the generated set of labels for voxels along a boundary' of the bone (e.g., approximately 2 mm thick band, such as + 1 mm, centered on the border), apply a second weight to the second sub-loss value to generate a second weighted sub-loss value, and generate the loss value based on the second weighted sub-loss value.
• a boundary' of the bone e.g., approximately 2 mm thick band, such as + 1 mm, centered on the border
• training unit 204 may generate a third sub-loss value indicative of a difference between the known labels and the generated set of labels for the voxels, apply a third weight to the third sub-loss value to generate a third weighted sub-loss value, and generate the loss value based on the at least one of the first weighted sub-loss value or the second weighted sub-loss value, and the third weighted sub-loss value.
• the first weight is greater than the second and third weights.
• the first weight equals 0.4
• tire second weight equals 0.3
• the third weight equals 0.3.
• one or more of the first, second, and third weights may be different in different iterations.
• training unit 204 may generate a loss value for each iteration over a plurality of training epochs.
• the first, second, and/or third weights may be different in two or more of the training epochs.
• the vales for the first, second, and/or third weights may a defined rules set for “x” number of epochs, or values that change depending on the loss or metric values.
• Training unit 204 may update the neural network weights based on the loss value (706). Training unit 204 may then determine whether there are more training sub-volumes (708). If there are more training sub-volumes (YES of 708), training unit 204 may set the next training sub-volume as the current training sub-volume (710), and repeat the operations of applying the machine learning model, but with the updated neural network weights, generating a loss value, and updating the neural network weights. If there are no more training sub-volumes (NO of 708), training unit 204 may output machine learning model 200 (712). For instance, the training iterations may be complete, and training unit 204 may store the neural network weights for machine learning model 200. As described above and with FIG.
• each epoch includes one of the plurality of training iterations, if there are no more training epochs, the result may be machine learning model 200 that ready for accurately label voxels in medical image data a bone or prosthesis for revision surgery.
• FIG. 8 is a flowchart illustrating an example operation in accordance with one or more techniques of this disclosure.
• FIG. 8 illustrates an example of outputting a machine learning model 200 after performing a plurality of training epochs.
• training unit 204 may divide a training image into a plurality of training sub-volumes (800).
• the division of the training image may be random or pseudorandom (e.g., at least one voxel is for the bone in the training sub-volumes).
• Each of the training sub-volumes may be 192x192x192 voxels, as an example, but other sizes are possible.
• Training unit 204 may perform training iterations with the training sub-volumes (802). For example, training unit 204 may perform the example techniques of FIG. 7. Training unit 204 may determine whether there are more training epochs (804). The number of training epochs may be predetermined and stored information, or may be determined based on various factors such as the particular bone or prothesis. If there are more training epochs (YES of 804), training unit 204 may re-divide the training image into another plurality of training sub-volumes, and perform training iterations with the other plurality of training sub-volumes. If there are no more training epochs (NO of 804), training unit 204 may output machine learning model 200 (806). For instance, the training iterations may be complete, and training unit 204 may store the neural network weights formachine learning model 200.
• Computer-readable media may include computer-readable storage media, which corresponds to a tangible medium such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol.
• computer-readable media generally may correspond to (1) tangible computer-readable storage media which is non-transitory or (2) a communication medium such as a signal or carrier wave.
• Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instractions, code and/or data structures for implementation of the techniques described in this disclosure.
• a computer program product may include a computer-readable medium.
• such computer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer- readable medium.
• coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infiared, radio, and microwave are included in the definition of medium.
• DSL digital subscriber line
• computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but are instead directed to non-transitory, tangible storage media.
• Disk and disc includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
• processors may be implemented as fixed-function processing circuits, programmable circuits, or combinations thereof, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuitry.
• DSPs digital signal processors
• ASICs application specific integrated circuits
• FPGAs field programmable gate arrays
• Fixed-function circuits refer to circuits that provide particular functionality and are preset on the operations that can be performed.
• Programmable circuits refer to circuits that can programmed to perform various tasks and provide flexible functionality in the operations that can be performed. For instance, programmable circuits may execute instructions specified by software or firmware that cause the programmable circuits to operate in the manner defined by instructions of the software or firmware.
• Fixed-function circuits may execute software instructions (e.g., to receive parameters or output parameters), but the types of operations that the fixed-function circuits perform are generally immutable. Accordingly, the terms “processor'’ and “processing circuity,” as used herein may refer to any of the foregoing structures or any other structure suitable for implementation of the techniques described herein.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Multimedia (AREA)
• Health & Medical Sciences (AREA)
• General Health & Medical Sciences (AREA)
• Artificial Intelligence (AREA)
• Evolutionary Computation (AREA)
• Medical Informatics (AREA)
• Databases & Information Systems (AREA)
• Software Systems (AREA)
• Computing Systems (AREA)
• Life Sciences & Earth Sciences (AREA)
• Biodiversity & Conservation Biology (AREA)
• Biomedical Technology (AREA)
• Molecular Biology (AREA)
• Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
• Radiology & Medical Imaging (AREA)
• Quality & Reliability (AREA)
• Image Analysis (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=90368518

Family Applications (1)

Country Status (2)

Families Citing this family (13)

• 2024
  • 2024-02-12 WO PCT/US2024/015395 patent/WO2024173251A1/en not_active Ceased
  • 2024-02-12 EP EP24713146.9A patent/EP4652575A1/de active Pending

Also Published As

Similar Documents

Legal Events