WO2021211603A1 - Systems and methods for ai-assisted surgery - Google Patents
Systems and methods for ai-assisted surgery Download PDFInfo
- Publication number
- WO2021211603A1 WO2021211603A1 PCT/US2021/027109 US2021027109W WO2021211603A1 WO 2021211603 A1 WO2021211603 A1 WO 2021211603A1 US 2021027109 W US2021027109 W US 2021027109W WO 2021211603 A1 WO2021211603 A1 WO 2021211603A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- labeling
- surgical
- procedure
- module
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 383
- 238000001356 surgical procedure Methods 0.000 title claims abstract description 148
- 238000003384 imaging method Methods 0.000 claims abstract description 37
- 210000003127 knee Anatomy 0.000 claims abstract description 31
- 230000009471 action Effects 0.000 claims description 118
- 238000012549 training Methods 0.000 claims description 109
- 238000012545 processing Methods 0.000 claims description 108
- 210000003484 anatomy Anatomy 0.000 claims description 103
- 238000002372 labelling Methods 0.000 claims description 98
- 230000007170 pathology Effects 0.000 claims description 57
- 238000013473 artificial intelligence Methods 0.000 claims description 55
- 238000011144 upstream manufacturing Methods 0.000 claims description 51
- 230000003416 augmentation Effects 0.000 claims description 35
- 238000010801 machine learning Methods 0.000 claims description 33
- 230000015654 memory Effects 0.000 claims description 33
- 238000013459 approach Methods 0.000 claims description 31
- 239000007943 implant Substances 0.000 claims description 27
- 238000003860 storage Methods 0.000 claims description 24
- 238000013135 deep learning Methods 0.000 claims description 16
- 230000002123 temporal effect Effects 0.000 claims description 14
- 238000013528 artificial neural network Methods 0.000 claims description 12
- 238000002324 minimally invasive surgery Methods 0.000 claims description 10
- 230000003213 activating effect Effects 0.000 claims description 6
- 238000005553 drilling Methods 0.000 claims description 6
- 238000002674 endoscopic surgery Methods 0.000 claims description 5
- 230000000007 visual effect Effects 0.000 claims description 5
- 230000036541 health Effects 0.000 claims description 4
- 230000008569 process Effects 0.000 description 22
- 210000001519 tissue Anatomy 0.000 description 22
- 210000003041 ligament Anatomy 0.000 description 12
- 238000001514 detection method Methods 0.000 description 11
- 230000000694 effects Effects 0.000 description 11
- 238000012360 testing method Methods 0.000 description 11
- 230000008439 repair process Effects 0.000 description 10
- 210000000845 cartilage Anatomy 0.000 description 9
- 238000004891 communication Methods 0.000 description 9
- 238000005259 measurement Methods 0.000 description 9
- 210000002435 tendon Anatomy 0.000 description 8
- 210000000056 organ Anatomy 0.000 description 7
- 241001653121 Glenoides Species 0.000 description 6
- 230000008859 change Effects 0.000 description 6
- 210000004095 humeral head Anatomy 0.000 description 6
- 230000003190 augmentative effect Effects 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 5
- 210000001367 artery Anatomy 0.000 description 4
- 210000000988 bone and bone Anatomy 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000009499 grossing Methods 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 210000003423 ankle Anatomy 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 3
- 238000004140 cleaning Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 230000006837 decompression Effects 0.000 description 3
- 238000000227 grinding Methods 0.000 description 3
- 210000005036 nerve Anatomy 0.000 description 3
- 230000002829 reductive effect Effects 0.000 description 3
- 210000003462 vein Anatomy 0.000 description 3
- 241001260012 Bursa Species 0.000 description 2
- 206010028980 Neoplasm Diseases 0.000 description 2
- 208000024288 Rotator Cuff injury Diseases 0.000 description 2
- 238000002679 ablation Methods 0.000 description 2
- 230000002411 adverse Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 230000000740 bleeding effect Effects 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 230000006378 damage Effects 0.000 description 2
- 238000002405 diagnostic procedure Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000002526 effect on cardiovascular system Effects 0.000 description 2
- 230000002496 gastric effect Effects 0.000 description 2
- 230000000968 intestinal effect Effects 0.000 description 2
- 230000000670 limiting effect Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000007674 radiofrequency ablation Methods 0.000 description 2
- 210000000513 rotator cuff Anatomy 0.000 description 2
- 230000006403 short-term memory Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 239000003826 tablet Substances 0.000 description 2
- 238000010200 validation analysis Methods 0.000 description 2
- 206010008398 Change in sustained attention Diseases 0.000 description 1
- RYGMFSIKBFXOCR-UHFFFAOYSA-N Copper Chemical compound [Cu] RYGMFSIKBFXOCR-UHFFFAOYSA-N 0.000 description 1
- 206010011732 Cyst Diseases 0.000 description 1
- 206010019909 Hernia Diseases 0.000 description 1
- 208000000913 Kidney Calculi Diseases 0.000 description 1
- 206010029148 Nephrolithiasis Diseases 0.000 description 1
- 208000008558 Osteophyte Diseases 0.000 description 1
- 241000287107 Passer Species 0.000 description 1
- 206010046798 Uterine leiomyoma Diseases 0.000 description 1
- 208000027418 Wounds and injury Diseases 0.000 description 1
- 230000001154 acute effect Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 210000001264 anterior cruciate ligament Anatomy 0.000 description 1
- 238000007486 appendectomy Methods 0.000 description 1
- 230000006793 arrhythmia Effects 0.000 description 1
- 206010003119 arrhythmia Diseases 0.000 description 1
- 230000002917 arthritic effect Effects 0.000 description 1
- 238000007681 bariatric surgery Methods 0.000 description 1
- 238000001574 biopsy Methods 0.000 description 1
- 239000008280 blood Substances 0.000 description 1
- 210000004369 blood Anatomy 0.000 description 1
- 230000036772 blood pressure Effects 0.000 description 1
- 201000011510 cancer Diseases 0.000 description 1
- 238000013131 cardiovascular procedure Methods 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 238000002192 cholecystectomy Methods 0.000 description 1
- 210000003109 clavicle Anatomy 0.000 description 1
- 230000001149 cognitive effect Effects 0.000 description 1
- 238000012321 colectomy Methods 0.000 description 1
- 238000000205 computational method Methods 0.000 description 1
- 230000008094 contradictory effect Effects 0.000 description 1
- 208000031513 cyst Diseases 0.000 description 1
- 238000001804 debridement Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000002059 diagnostic imaging Methods 0.000 description 1
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 1
- 208000035475 disorder Diseases 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000007717 exclusion Effects 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000002683 hand surgery Methods 0.000 description 1
- 238000010348 incorporation Methods 0.000 description 1
- 208000014674 injury Diseases 0.000 description 1
- 210000000936 intestine Anatomy 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 201000010260 leiomyoma Diseases 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000005499 meniscus Effects 0.000 description 1
- 238000012978 minimally invasive surgical procedure Methods 0.000 description 1
- 230000000116 mitigating effect Effects 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000000306 recurrent effect Effects 0.000 description 1
- 238000002271 resection Methods 0.000 description 1
- 230000029058 respiratory gaseous exchange Effects 0.000 description 1
- 230000033764 rhythmic process Effects 0.000 description 1
- 238000005096 rolling process Methods 0.000 description 1
- 239000000523 sample Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 210000002832 shoulder Anatomy 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000004513 sizing Methods 0.000 description 1
- 239000010454 slate Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 210000002784 stomach Anatomy 0.000 description 1
- 210000005222 synovial tissue Anatomy 0.000 description 1
- 230000001225 therapeutic effect Effects 0.000 description 1
- 238000002560 therapeutic procedure Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/7715—Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/809—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of classification results, e.g. where the classifiers operate on the same input data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/70—Labelling scene content, e.g. deriving syntactic or semantic representations
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H20/00—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance
- G16H20/40—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H30/00—ICT specially adapted for the handling or processing of medical images
- G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
- G16H40/60—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
- G16H40/67—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices for remote operation
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/361—Image-producing devices, e.g. surgical cameras
- A61B2090/3614—Image-producing devices, e.g. surgical cameras using optical fibre
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/30—Devices for illuminating a surgical field, the devices having an interrelation with other surgical devices or with a surgical procedure
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/04—Inference or reasoning models
- G06N5/046—Forward inferencing; Production systems
- G06N5/047—Pattern matching networks; Rete networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/24—Indexing scheme for image data processing or generation, in general involving graphical user interfaces [GUIs]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10068—Endoscopic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30008—Bone
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/03—Recognition of patterns in medical or anatomical images
- G06V2201/034—Recognition of patterns in medical or anatomical images of medical instruments
Definitions
- Embodiments of the invention relate to systems, devices, and methods to guide diagnostic and surgical procedures, particularly using Artificial Intelligence (AI).
- AI Artificial Intelligence
- Various embodiments of the invention provide computer-implemented medical systems, devices, and methods to guide surgery or other medical procedures. Many embodiments do so by identifying and labeling one or more of anatomical features, pathologies and other features in a surgical field of view in real-time, as well as to provide guidance to an operator are described herein. Any mistake made by an operator in the course of a surgery can be costly. For example, it may be difficult or impossible for an operator to know the exact location of a critical anatomical feature that is hidden from a camera (e.g., a camera used during an arthroscopic or endoscopic surgery), or a part of a pathology that was being removed may be missed from the eyes of an operator or a camera view.
- a camera e.g., a camera used during an arthroscopic or endoscopic surgery
- AI Artificial Intelligence
- computer- implemented medical systems, devices, and methods such as Artificial Intelligence (AI) tools, particularly for guiding medical procedures can be valuable.
- AI tools can have their limitations in accurately and reliably predicting a tool, anatomical structure, or detecting a procedure. In a fast-paced surgical procedure, the AI tool needs to also make predictions with low latency to provide real time assistance to an operator.
- aspects of the present invention provide a pipeline of machine learning algorithms that is versatile and well trained for unique needs of various medical procedures including various minimally invasive procedures such as arthroscopic, endoscopic, laparoscopic, cardioscopic and related procedures.
- Examples of such minimally invasive procedures can include one or more of Arthroscopic procedures (e.g., repair of a tom rotator cuff in the shoulder, ACL surgery of the knee, repair of various non-arthritic hip disorders, repair of damaged cartilage in the ankle or removal of bone spurs in the ankle); Gastro-intestinal (GI) procedures (e.g., biopsy of the intestines, removal of polyps, bariatric surgery, stomach stapling/vertical banded gastroplasty), urological procedures (e.g., removal of kidney stone, bladder repair), gynecological procedures (e.g., a dnc, removal of uterine fibroids) and a laparoscopic procedures (e.g., an appendectomy, cholecystectomy, colectomy, hernia repair, nissen fundoplication.
- GI Gastro-intestinal
- urological procedures e.g., removal of kidney stone, bladder repair
- Various embodiments of the invention provide systems, devices, and methods that can receive information (e.g., image(s), voice, user inputs) during a medical procedure (e.g., a surgery), process the received information to identify features associated with the procedure, and provide recommendations based on the identified features. These features may include anatomical parts or devices, or various steps or outcomes of a procedure. Based on identifying the various features in an operation, the systems, devices, and methods described herein can assist a surgeon in the operation by providing recommendations that can include approach angle of a tool, measurement of an anatomical feature or pathology, an action that should be taken or avoided, to name a few.
- aspects of the present invention also aid surgeons (and related medical personnel) intraoperatively by using the images from the surgical field of view and applying Artificial Intelligence (AI) to provide guidance and assistance to the surgeons and other medical staff.
- AI Artificial Intelligence
- the invention provides systems for guiding an arthroscopic procedure.
- the systems comprise one or more computer processors and one or more non- transitory computer-readable storage media storing instructions that are operable, when executed by the one or more computer processors, to cause the one or more computer processors to perform operations comprising: receiving at least one image captured by an interventional imaging device; identifying one or more image features in the received at least one image using an image recognition algorithm; labeling the identified one or more image features; and displaying the labeled one or more image features in the at least one image to an operator continuously in the course of the arthroscopic procedure.
- the identified one or more image features comprise one or more of an anatomical structure, a surgical tool, a surgical tool element, an operational procedure or an action or a pathology such as tom or injured tissue.
- the labeled one or more image features may be displayed in real time or concurrent to the arthroscopic procedure.
- the arthroscopic procedure is an arthroscopic surgery.
- the image recognition algorithm comprises a hierarchical arrangement of processing modules also referred to as software modules or modules.
- the processing modules comprise a plurality of Artificial Intelligence (AI) modules which, in various embodiments, may correspond to a machine learning algorithm, a deep learning algorithm, or a combination of both.
- the machine learning algorithm comprises an artificial neural network.
- the processing modules comprise at least one dataset which may include at least one training dataset.
- the processing modules comprise an upstream module and a downstream module, where the downstream module is more specialized than the upstream module.
- the upstream module is configured to identify one or more of an anatomy being operated upon by an operational procedure or an action being performed during the arthroscopic procedure.
- the downstream module is configured recognize one or more of an anatomical features of the identified anatomy or a treatment tool feature associated with the operational procedure or the action being performed.
- the processing modules comprise a plurality of upstream modules and a plurality of downstream modules. In some embodiments, at least one of the modules of the plurality of upstream modules is configured to select an individual downstream module from the processing modules for use.
- the operational procedure or the action is identified by one or more of the processing modules based at least partially on identification of a surgical tool used during the procedure, for example an arthroscope vs an endoscope.
- the interventional imaging device is an arthroscope.
- the interventional imaging device is an endoscope.
- the image recognition algorithm is configured to identify one or more of a region of operation or an approach portal of entry of an arthroscopic procedure or other medical procedure.
- the region of operation may correspond to one or more of a shoulder, a knee or a hip.
- At least one module from the processing modules is selected based at least on identification of one or more of the region of operation (e.g., the shoulder) or the approach portal of entry (e.g., a bursal approach for the shoulder, an anterior approach for the knee).
- the operations performed further comprise storing the at least one image in a memory device.
- the operations further comprise discarding the at least one image after the displaying the label elements to optimize memory usage.
- the labeled one or more image features further comprises: a pixel- level masked labeling, a bounding box labeling, a frame-level labeling, or a temporal labeling which may configured to be used for various purposes.
- the pixel-level masked labeling may be used to display the labeled anatomical structure or the surgical tool.
- the bounding box labeling may be used to display one or more of the labeled pathology, the surgical tool, or a foreign object.
- the frame-level labeling may be used to display a labeled anatomical region for example that of the shoulder and the temporal labeling may be used to display the labeled operational procedure or action (e.g., a surgical action such as tissue resection, ablation or suturing).
- a surgical action such as tissue resection, ablation or suturing
- the operations further comprise providing a suggested action which may be based on a number of factors including results or other output from various modules.
- the suggested action may be at least partially based on one more of the following: the labeling of the at least one image, results or other output from the upstream or result or other output from the downstream module.
- the suggested action may have a variety of purposes depending the surgery and situation. For example, the suggested action may be used to assist the operator in the course of the surgery while in other it may for educational or training purposes such as when a surgeon is learning a new procedure and/or during a simulated surgery.
- the suggested action comprises providing a safety warning based at least on identification of a critical anatomical structure (e.g., an artery), a surgical tool, an action, a distance between two or more implants (e.g., if its above or below a critical event) or an adverse event (e.g., bleeding in the surgical field) or biomedical/physiologic data of the patient (e.g., drop in blood p02, blood pressure, respiration rate, irregular heart rhythm (e.g., arrhythmia or other important physiologic parameter of the patient)
- a critical anatomical structure e.g., an artery
- a surgical tool e.g., an action, a distance between two or more implants (e.g., if its above or below a critical event) or an adverse event (e.g., bleeding in the surgical field) or biomedical/physiologic data of the patient (e.g., drop in blood p02, blood pressure, respiration rate, irregular heart rhythm (e.g., arrhythmi
- Various embodiments of the invention may also provide suggested surgical actions based one or more of dimensions, positioning and arrangement of structures including anatomical structures and implants within the surgical field.
- the distance between the two or more implants may create a health risk.
- the actual or allowable distance between two or more positioned implants positioned in the surgical field is different from a predefined distance of the two or more implants making an approach to the surgical field including approaches for position the implants difficult and/or hard to approximate by the human eye particularly when done on screen in two-dimensional view.
- the suggested action may comprises providing a suggested approach angle which may be suggested a drilling angle for placement of an implant or other surgical action.
- Various embodiments of the system may be configured for a number of joint surgeries including for example one or more of shoulder surgery, knee surgery, hip surgery, ankle surgery, hand surgery or elbow surgery.
- one or more of the processing modules described herein may be adapted for each particular site including having data on each particular which may include specialized training data sets.
- the image recognition algorithm is trained using a database.
- the database comprises a plurality of training images comprising one or more surgical procedures, surgical tools, surgical tool elements, anatomical structures, or pathologies.
- a plurality of augmentation techniques is used to improve the training dataset to improve a robustness of the image recognition algorithm.
- the surgical images used for such training including augmented training may be selected from a variety of procedures including one or more minimally invasive procedures such as arthroscopic, endoscopic, laparoscopic and cardioscopic procedures which may correspond to one or more of arthroscopic, bariatric, cardiovascular, intestinal, gynecological, urological surgeries or related procedures.
- the augmentation techniques comprise rotating the training images to improve the robustness against a position or an orientation of a patient during the arthroscopic procedure.
- the augmentation techniques comprise flipping the training images along a vertical axis to improve the robustness against procedures performed on a right or a left side of a patient.
- the augmentation techniques comprise enlarging or cropping the training images to improve the robustness against changes in a depth of view.
- the at least one image is generated from a surgical video stream.
- the surgical video stream is an arthroscopic surgery video stream.
- the surgical video stream may be monocular or stereoscopic.
- embodiment of the system may be adapted to receive and recognize each type of view (monocular and stereoscopic) and switch back and forth between appropriate processing methods when the views are switch.
- Another aspect of the present invention provides computer-implemented methods for guiding an arthroscopic procedure.
- the methods comprise: receiving at least one image captured by an interventional imaging device; identifying one or more image features in the received at least one image using an image recognition algorithm; labeling the identified one or more image features; and displaying the labeled one or more image features in the at least one image to an operator continuously in the course of the arthroscopic procedure.
- the identified one or more image features comprising one or more of an anatomical structure, a surgical tool, a surgical tool element, an operational procedure or action, or a pathology.
- the labeled one or more image features are displayed in real time or concurrent to the arthroscopic procedure.
- the arthroscopic procedure is an arthroscopic surgery.
- the image recognition algorithm comprises a hierarchical arrangement of processing modules.
- the processing modules comprise a plurality of Artificial Intelligence (AI) modules.
- the processing modules comprise at least a machine learning algorithm, a deep learning algorithm, or a combination of both.
- the machine learning algorithm comprises an artificial neural network.
- the processing modules comprise at least one dataset which may include at least one training dataset.
- the processing modules comprise an upstream module and a downstream module, where the downstream module being more specialized than the upstream module.
- the upstream module is configured to identify one or more of an anatomy being operated upon by an operational procedure or an action being performed during the arthroscopic procedure.
- the downstream module is configured to one or more of recognize an anatomical feature of the identified anatomy or recognize a treatment tool feature associated with the operational procedure or the action being performed.
- the processing modules comprise a plurality of upstream modules or a plurality of downstream modules.
- the at least one of the modules of the plurality of upstream modules is configured to select an individual downstream module from the processing modules for use.
- the operational procedure or the action is identified partially based on the identifying the surgical tool.
- the interventional imaging device is an arthroscope. In some embodiments, the interventional imaging device is an endoscope. In some embodiments, the image recognition algorithm is configured to identify one or more of a region of operation or an approach portal of entry of the arthroscopic procedure. In some embodiments, the region of operation is a shoulder. In some embodiments, the region of operation is a knee. In some embodiments, at least one module from the processing modules is selected based at least on the identifying the one or more of the region of operation or the approach portal of entry.
- the operations performed further comprise storing the at least one image in a memory device. In some embodiments, the operations further comprise discarding the at least one image after the displaying the label elements to optimize memory usage.
- the labeled one or more image features further comprises: a pixel-level masked labeling, a bounding box labeling, a frame-level labeling, or a temporal labeling.
- the pixel-level masked labeling is used to display the labeled anatomical structure or the surgical tool.
- the bounding box labeling is used to display the labeled pathology, the surgical tool, or a foreign object.
- the frame-level labeling is used to display a labeled anatomical region.
- the temporal labeling is used to display the labeling the operational procedure or the action.
- the operations further comprise providing a suggested action.
- the suggested action is at least partially based on the labeling of the at least one image.
- the suggested action is at least partially based on the upstream module.
- the suggested action is at least partially based on the downstream module.
- the suggested action is to assist the operator in the course of the surgery.
- the suggested action is provided for educational purposes.
- the suggested action comprises providing a safety warning based at least on identification of a critical anatomical structure, or a distance of two or more implants.
- the distance of the two or more implants creates a health risk. In some embodiments, the distance of the two or more implants is different from a predefined distance of the two or more implants.
- the suggested action comprises providing a suggested approach angle. In some embodiments, the suggested approach angle may comprise a drilling angle.
- the method is configured for a shoulder surgery. In some embodiments, the method is configured for a knee surgery. In some embodiments, the image recognition algorithm is trained using a database. In some embodiments, the database comprises a plurality of training images comprising one or more surgical procedures, surgical tools, surgical tool elements, anatomical structures, or pathologies.
- a plurality of augmentation techniques is used to improve the training dataset to improve a robustness of the image recognition algorithm.
- the augmentation techniques comprise rotating the training images to improve the robustness against a position or an orientation of a patient during the arthroscopic procedure.
- the augmentation techniques comprise flipping the training images along a vertical axis to improve the robustness against procedures performed on a right or a left side of a patient.
- the augmentation techniques comprise enlarging or cropping the training images to improve the robustness against changes in a depth of view.
- the at least one image is generated from a surgical video stream.
- the surgical video stream is an arthroscopic surgery video stream.
- the surgical video stream is monocular.
- the surgical video stream is stereoscopic.
- Another aspect of the present invention provides a method of training an algorithm for guiding an arthroscopic procedure.
- the method comprises: receiving a set of image features based on one or more images relating to the arthroscopic procedure; receiving a training dataset; recognizing one or more of the image features in images of the training dataset; and building an image recognition algorithm based at least partially on the recognition of the one or more image features and the received training dataset.
- the training dataset comprises one or more labeled images relating to the arthroscopic procedure.
- the training dataset comprises one or more labeled images relating to the arthroscopic procedure one or more image features relate to visual properties of one or more of an anatomical structure, a surgical tool, a surgical tool element, an operational procedure or action, or a pathology.
- the image recognition algorithm is configured to identify and label the one or more image features in an unlabeled image relating to the arthroscopic procedure.
- the labeled one or more image features are displayed in real time or concurrent to the arthroscopic procedure.
- the image recognition algorithm comprises a hierarchical arrangement of processing modules.
- the processing modules comprise a plurality of individual image processing modules.
- the plurality of individual image processing modules comprises a first module for identifying the arthroscopic procedure in place, a second module for recognizing and labeling one or more surgical tools and surgical tool elements, a third module for recognizing and labeling one or more anatomical structures, or a combination thereof.
- the processing modules comprise a plurality of Artificial Intelligence (AI) modules.
- the processing modules comprise at least a machine learning algorithm, a deep learning algorithm, or a combination of both.
- the machine learning algorithm comprises an artificial neural network.
- the processing modules comprise at least one dataset. In some embodiments, the processing modules comprise at least one training dataset.
- the processing modules comprise an upstream module and a downstream module, where the downstream module being more specialized than the upstream module.
- the upstream module is configured to identify one or more of an anatomy being operated upon by an operational procedure or an action being performed during the arthroscopic procedure.
- the downstream module is configured to one or more of recognize an anatomical feature of the identified anatomy or recognize a treatment tool feature associated with the operational procedure or the action being performed.
- the processing modules comprise a plurality of upstream modules or a plurality of downstream modules. In some embodiments, at least one of the modules of the plurality of upstream modules is configured to select an individual downstream module from the processing modules for use.
- identifying one or more image features in the at least one image further comprises selecting one or more processing modules from a plurality of processing modules, where the selection is at least partially based on the region and/or the portal of the arthroscopic surgery.
- the operational procedure or the action is identified partially based on the identifying the surgical tool.
- the unlabeled image is captured by an interventional imaging device.
- the interventional imaging device is an endoscope.
- the unlabeled image is generated from a surgical video stream.
- the surgical video stream is an arthroscopic surgery video stream where the endoscopic surgery is an arthroscopic surgery.
- the surgical video stream is monocular. In some embodiments, the surgical video stream is stereoscopic.
- the image recognition algorithm is configured to identify one or more of a region of operation or an approach portal of entry of the arthroscopic procedure.
- the region of operation is a shoulder.
- the region of operation is a knee.
- at least one module from the processing modules is selected based at least on the identifying the one or more of the region of operation or the approach portal of entry.
- the image recognition algorithm stores the labeled image in a memory device. In some embodiments, the image recognition algorithm discards the labeled image to minimize memory usage.
- the labeled image comprises: a pixel-level masked labeling, a bounding box labeling, a frame-level labeling, or a temporal labeling.
- the pixel- level masked labeling is used to display the labeled anatomical structure or the surgical tool.
- the bounding box labeling is used to display the labeled pathology, the surgical tool, or a foreign object.
- the frame-level labeling is used to display a labeled anatomical region.
- the temporal labeling is used to display the labeling the operational procedure or action.
- the training dataset is configured for a shoulder surgery.
- the image recognition algorithm is trained for a shoulder surgery using the training dataset configured for a shoulder surgery.
- the training dataset is configured for a knee surgery.
- the image recognition algorithm is trained for a knee surgery using the training dataset configured for a knee surgery.
- the training dataset comprises a plurality of training images comprising one or more surgical procedures, surgical tools, surgical tool elements, anatomical structures, or pathologies.
- a plurality of augmentation techniques is used to improve the training dataset to improve a robustness of the image recognition algorithm.
- the augmentation techniques comprise rotating the training images to improve the robustness against a position or an orientation of a patient during the arthroscopic procedure.
- the augmentation techniques comprise flipping the training images along a vertical axis to improve the robustness against procedures performed on a right or a left side of a patient.
- the augmentation techniques comprise enlarging or cropping the training images to improve the robustness against changes in a depth of view.
- the system comprises one or more computer processors and one or more non-transitory computer-readable storage media storing instructions that are operable, when executed by the one or more computer processors, to cause the one or more computer processors to perform operations comprising: (a) receiving at least one image captured by an interventional imaging device; (b) identify one or more image features of a region of treatment or a portal of entry in the region based on at least one upstream module; (c) activating a first downstream module to identify one or more image features of an anatomical structure, or a pathology based at least partially on the identified one or more image features in step (b); (d) activating a second downstream module to identify one or more image features of a surgical tool, a surgical tool element, an operational procedure or action relating to the arthroscopic surgery based at least partially on the identified one or more image features in step (b); (e
- the at least one upstream module comprises a first trained image processing algorithm. In some embodiments, the at least one upstream module comprises a first trained image processing algorithm first downstream module comprises a second trained image processing algorithm. In some embodiments, the at least one upstream module comprises a first trained image processing algorithm first downstream module comprises a second trained image processing algorithm second downstream module comprises a third trained image processing algorithm. In some embodiments, the steps (c) and (d) are independent from one another. In some embodiments, the first, second, or third trained image processing algorithms comprise at least a machine learning algorithm, a deep learning algorithm, or a combination of both. In one or more embodiments, the machine learning algorithm includes an artificial neural network.
- the machine learning or the deep learning algorithms are trained using at least one training dataset.
- the training dataset is configured for a shoulder surgery.
- the training dataset is configured for a knee surgery.
- the training dataset comprises a plurality of training images comprising one or more surgical procedures, surgical tools, surgical tool elements, anatomical structures, or pathologies.
- a plurality of augmentation techniques is used to improve the training dataset to improve a robustness of the image recognition algorithm.
- the augmentation techniques comprise rotating the training images to improve the robustness against a position or an orientation of a patient during the arthroscopic procedure.
- the augmentation techniques comprise flipping the training images along a vertical axis to improve the robustness against procedures performed on a right or a left side of a patient.
- the augmentation techniques comprise enlarging or cropping the training images to improve the robustness against changes in a depth of view.
- the first, second, or third trained image processing algorithms store the displayed images with labeled features in a memory device. In some embodiments, the first, second, or third trained image processing algorithms discard the displayed images with labeled features to minimize memory usage.
- the arthroscopic procedure is an arthroscopic surgery.
- the at least one image is generated from a surgical video stream.
- the surgical video stream is an endoscopic surgery video stream.
- the surgical video stream is monocular.
- the surgical video stream is stereoscopic.
- Another aspect of the present invention provides a non-transitory computer readable medium comprising machine executable code that, upon execution by one or more computer processors, implements any of the methods above or elsewhere herein.
- Another aspect of the present invention provides a system comprising one or more computer processors and computer memory coupled thereto.
- the computer memory comprises machine executable code that, upon execution by the one or more computer processors, implements any of the methods above or elsewhere herein.
- Embodiments of the systems and methods of the invention are particularly useful in aiding the surgeon during surgery such as arthroscopic surgier by providing real time display of anatomical structures, pathologies, pathology repair and selected measurements within the surgical field.
- Such display reduces errors due to misidentified tissue structures and measurements in turn leading to improved surgical and patient outcomes. It also does so by reducing the cognitive load on the surgeon allowing them to focus on the procedure and at the time alerting them to potentially adverse actions, conditions or results during the course of the surgery that they may otherwise miss due to focus or hyper focus on critical tasks.
- Such alerts prevent or reduce potential adverse events during surgery by alerting the surgeon before they happen and/or contemporaneously to their occurrence to allow the surgeon to take appropriate action for their prevention or mitigation.
- alerts lead to both reduced morbidity and mortality as well as improved acute surgical outcomes and long-term outcomes for the patient as the surgery is performed with no or fewer errors and with more precision and accuracy in the surgical actions performed (e.g., accuracy of implant placement or removal of damaged or diseased, tissue while preserving healthy tissue).
- FIG. 1 shows a flow chart of an example of hierarchical arrangement of modules in a system for AI-assisted surgery, according to some embodiments.
- FIG. 2A shows an image of an arthroscopic surgical procedure, according to some embodiments.
- FIG. 2B shows an example of labeling features in an arthroscopic surgical procedure, according to some embodiments.
- FIG. 3A shows an example of labeling a tool, an action, or a procedure in an image captured by an arthroscope, according to some embodiments.
- FIG. 3B shows another example of labeling a tool, an action, or a procedure in an image captured by an arthroscope, according to some embodiments.
- FIG. 3C shows an example of recognizing an arbitrary tool in a field of surgery, according to some embodiments.
- FIG. 4 shows a flowchart of an example of training an image recognition algorithm, according to some embodiments.
- FIG. 5 shows a computer system that is programmed or otherwise configured to implement methods provided herein, according to some embodiments.
- FIG. 6 shows an example of an AI pipeline used in a simulated knee surgery, according to some embodiments.
- FIG. 7A shows an example of recognizing and labeling anatomical structures, according to some embodiments.
- FIG. 7B shows an example of recognizing and labeling a pathology, according to some embodiments.
- FIGs. 8A shows an example of an image of an arthroscopic surgical procedure, according to some embodiments.
- FIG. 8B shows an example of an image of an arthroscopic surgical procedure labeled using image recognition algorithms, according to some embodiments.
- FIGs. 8C-8L show examples of anatomical structures in an image labeled by subject matter experts, according to some embodiments.
- FIGs. 8M-8V show examples of anatomical structures in an image labeled by AI algorithms, according to some embodiments.
- FIG. 9 shows an example of an action recognized by the system for AI-assisted surgery, according to some embodiments.
- FIG. 10 shows an example flowchart of the process of identifying a surgical procedure, according to some embodiments.
- FIG. 11A shows another example of the process of identifying a surgical procedure, according to some embodiments.
- FIG. 11B shows an example of an output from tool detection, anatomy detection, and activity detection models across time, according to some embodiments.
- FIG. 12 shows a schematic diagram of elements in an AI assisted surgery system, according to some embodiments.
- Various embodiments of the invention provide icomputer-implemented medical systems, devices, and methods for assisting surgeons in an intraoperative setting using AI.
- the systems, devices, and methods disclosed herein may improve upon existing methods of surgical assistance by providing an improved classification (e.g., real-time) of various elements involved in a surgical operation (e.g., surgical tools, anatomical features, anatomical pathology/injury features, operation procedures).
- an improved classification e.g., real-time
- elements involved in a surgical operation e.g., surgical tools, anatomical features, anatomical pathology/injury features, operation procedures.
- One or more embodiments of the systems, devices, and methods provided herein may achieve this objective through the use AI methods (e.g., machine learning, deep learning) to build a classifier which improves a real-time classification of elements involved in a surgical operation.
- the classifier model may improve real-time characterization of various elements involved in an operation which may lead to higher operation success rate for example due to fewer errors due to misidentified anatomical structures.
- the classifier model may provide an operator (e.g., surgeon, operating room nurse, surgical technician) with information for making more accurate and timely decisions (e.g., labeling critical anatomical features in real-time).
- the computer-implemented medical systems, devices, and methods as disclosed herein may improve upon existing methods of clinical decision support systems by leveraging parameters related to various elements in the context of a surgical operation to generate accurate real-time decisions continuously.
- a surgical operation involves various elements such as the patient’s body with various parts and anatomical complexity, plurality of tools and devices, actions based on the surgical procedure as well as actions based on a non-routine event that may occur in the surgery.
- the systems, devices, and methods disclosed herein can operate continuously with the changing environment of the surgical operation to make classifications and suggestions based on a plurality of AI modules organized to make decisions in a hierarchical manner.
- the classifier described herein can classify anatomical features (e.g., shoulder, knee, organ, tissue, or pathology) while a view of the endoscopic camera may change during the operation.
- the systems, devices, and method described here can recognize surgical tools as they appear in the field of view.
- the systems, devices, and methods as disclosed herein may be used to classify various elements involved in an operation.
- the classifier disclosed herein can identify and label anatomical structures (e.g., anatomical parts, organs, tissues), surgical tools, or a procedure being performed in an operation.
- the invention may help with recognizing critical structures (e.g., nerves, arteries, veins, bone, cartilage, ligaments) or pathologies (e.g., a tissue that needs to be removed).
- the critical structures may be visible or hidden in the field of view.
- the systems, devices, and methods described herein can identify and mark the critical structures (e.g., color markings on the video stream).
- Systems according to many embodiments may comprise a plurality of surgical AI modules organized to make decisions in a hierarchical manner.
- the surgical AI assist modules disclosed herein may analyze video feeds of the surgical field of view and render decisions in a substantially continuous manner. These modules can be connected by a messaging framework which can process the flow of video data, and a plurality of outputs (e.g., decisions).
- the AI systems may react to changes in the surgical field of view.
- the methods and systems of the disclosed invention may be applied to various surgical operations including different anatomical regions and organs.
- the surgical AI assist can work when surgeons change views, i.e., access the surgical field from different approach portals.
- the term “if 1 is optionally construed to mean “when” or “upon” or “in response to determining” or “in accordance with a determination” or “in response to detecting,” that a stated condition precedent is true, depending on the context.
- the phrase “if it is determined [that a stated condition precedent is true]” or “if [a stated condition precedent is true]” or “when [a stated condition precedent is true]” is optionally construed to mean “upon determining” or “in response to determining” or “in accordance with a determination” or “upon detecting” or “in response to detecting” that the stated condition precedent is true, depending on the context.
- the term “about” or “approximately” means an acceptable error for a particular value as determined by one of ordinary skill in the art, which depends in part on how the value is measured or determined. In certain embodiments, the term “about” or “approximately” means within 1, 2, 3, or 4 standard deviations. In certain embodiments, the term “about” or “approximately” means within 30%, 25%, 20%, 15%, 10%, 9%, 8%, 7%, 6%, 5%, 4%, 3%, 2%, 1%, 0.5%, 0.1%, or 0.05% of a given value or range.
- the terms “comprises,” “comprising,” or any other variation thereof, are intended to cover a nonexclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
- the terms “subject” and “patient” are used interchangeably.
- the terms “subject” and “subjects” refers to a human being. In certain embodiments, the subject is going through a surgical operation.
- the subject is 0 to 6 months old, 6 to 12 months old, 1 to 5 years old, 5 to 10 years old, 10 to 15 years old, 15 to 20 years old, 20 to 25 years old, 25 to 30 years old, 30 to 35 years old, 35 to 40 years old, 40 to 45 years old, 45 to 50 years old, 50 to 55 years old, 55 to 60 years old, 60 to 65 years old, 65 to 70 years old, 70 to 75 years old, 75 to 80 years old, 80 to 85 years old, 85 to 90 years old, 90 to 95 years old or 95 to 100.
- surgical AI or “surgical AI module”, as used herein, generally refer to a system, device, or method that uses Artificial Intelligence algorithms to assist before, during, and/or after a surgical operation.
- a surgical AI module can be defined as a combination of input data, machine learning or deep learning algorithms, training datasets, or other datasets.
- machine learning may generally refer to computer algorithms that can improve automatically overtime. Any description herein of machine learning can be applied to Artificial Intelligence, and vice versa, or any combination thereof.
- the terms “continuous,” “continuously” or any other variation thereof, generally refer to a substantially uninterrupted process or a process with time delay that is acceptable in the context of the process.
- Video stream or “video feed”, as used herein, refer to data generated by a digital camera. Video feed may be a sequence of static or moving pictures.
- region may generally refer to anatomical features of the human body.
- a region may be larger than an organ and may comprise an organ.
- An organ may comprise one or more tissue types and structures.
- a Tissue may refer to a group of cells structurally joined to complete a common function.
- a structure can refer to a part of a tissue.
- a structure may refer to one or more parts of one or more tissues joined together to create an anatomical feature.
- surgical field of view refers to the extent of visibility captured by an interventional imaging device.
- Field of view may refer to the extent of visual data captured by a digital camera that is observable by human eye.
- decision may refer to outputs from a machine learning or AI algorithm.
- a decision may comprise labeling, classification, prediction, etc.
- interventional imaging device generally refers to an imaging device used for medical purposes.
- the interventional imaging device may refer to an imaging device that is used in a surgical operation.
- the surgical operation in some embodiments, may be a simulation of an operation.
- operator refers to a medical professional involved in a surgical operation.
- An operator can be a surgeon, an operating room nurse, a surgical technician.
- One aspect of the invention provides a system for guiding an arthroscopic procedure.
- the system may comprise one or more computer processors and one or more non-transitory computer- readable storage media storing instructions that are operable, when executed by the one or more computer processors, to cause the one or more computer processors to perform operations comprising: receiving at least one image captured by an interventional imaging device; identifying one or more image features in the received at least one image using an image recognition algorithm; labeling the identified one or more image features, wherein the identified one or more image features comprising one or more of an anatomical structure, a surgical tool, a surgical tool element, an operational procedure or an action, or a pathology; and displaying the labeled one or more image features in the at least one image to an operator continuously in the course of the arthroscopic procedure.
- the operational procedure or the action may be identified partially based on identifying the surgical tool.
- an interventional imaging device is an endoscope.
- an interventional imaging device is an arthroscope.
- the operations may further comprise storing the image in a memory device. In some embodiments, the operations further comprise discarding the image after displaying the labeled image features, to optimize a memory usage.
- the arthroscopic procedure may be an arthroscopic surgery (or arthroscopy).
- An arthroscopic surgery also known as arthroscopy or keyhole surgery can be a minimally invasive surgical procedure (e.g., it requires only a small incision) on a joint.
- the arthroscopy may comprise either a diagnostic or therapeutic procedure.
- Therapeutic aspects or types of arthroscopy may also comprise surgical repairs, such as, for example, debridement, or cleaning, of a joint to remove bits of tom cartilage, ligament reconstruction, or synovectomy (removal of the joint lining).
- the arthroscopy can be performed using an arthroscope.
- the arthroscope may be inserted into a subject’s body to perform the arthroscopic procedure in or around the joint through a small incision.
- the arthroscope may be an endoscope.
- the arthroscope may comprise a fiberoptic scope.
- the arthroscope may be flexible or rigid.
- the arthroscope may comprise a camera (e.g., a digital camera), a light source, a lens to create the image of the field of view, or a mechanism to carry the image to a sensor.
- the sensor may comprise a global shutter (e.g., a CCD sensor) or a rolling shutter (e.g., a CMOS sensor).
- the images captured by the arthroscope may be displayed on a display (e.g., a monitor). An operator (e.g., a surgeon) may use the displayed images (e.g., video feed from the arthroscope) to perform the surgery.
- Arthroscopic surgeries may use small incisions through which tools and endoscopes may access a joint to diagnose or repair a pathology. Due to the minimally invasive nature of the procedure, patients may experience less pain, heal faster than conventional surgeries or experience less bleeding. However, arthroscopic surgeries can be more technically demanding than conventional surgeries (e.g., an open surgery). In arthroscopy, an operator (e.g., a surgeon) may operate with limited field of view, using small tools and with a restricted range of movement. Since the field of view may restricted, the surgeon may, for example, have to pan the scope to cover a target anatomy.
- Nonlimiting example of challenges associated with an arthroscopy may comprise tracking a target location, such as, for example, an intended repair location, a recognition of a pathology (e.g., cancerous tissue) , critical structures (e.g., nerves, arteries, veins, bone, cartilage, ligaments), etc.; visual orientation with respect to an anatomical structure; intraoperatively measuring of a dimension (e.g., sizing a rotator cuff tear); keeping track of bones or tissue boundaries for placing an implant when using a graft; a field of view that may not be sufficient for an operator to place an anchor in a predefined location, where the procedure may require a larger field of view; or correlating preoperative diagnostic imaging with intraoperative field of view to, for example recognize a critical sight.
- a target location such as, for example, an intended repair location, a recognition of a pathology (e.g., cancerous tissue) , critical structures (e.g., nerves, arteries, veins, bone, cartilage,
- the critical sight may comprise a pathology (e.g., a tumor or a cyst), or a predefined implant or repair site.
- a pathology e.g., a tumor or a cyst
- the systems and methods provided herein may be configured for a shoulder surgery. In some embodiments, the systems and methods provided herein may be configured for a knee surgery.
- an image captured by imaging device may be received by a computer system.
- the computer system may comprise an image recognition or algorithm or another related algorithm.
- the image recognition algorithm may identify one or more features in the image received from the arthroscope.
- the image recognition algorithm may be configured to identify one or more of a region of operation or an approach portal of entry (or a portal) of the arthroscopic procedure (e.g., an arthroscopic surgery).
- the region of operation is a shoulder.
- the region of operation is a knee.
- the image recognition algorithm may comprise a processing module.
- the image recognition algorithm may comprise an arrangement of processing modules.
- the arrangement of processing modules may be hierarchical.
- a hierarchical arrangement of processing modules may comprise a first processing module that may be upstream of a second processing module and/or downstream of a third processing module.
- the image recognition algorithm may comprise at least two processing modules an upstream module and a downstream module.
- the downstream module may be more specialized (e.g., configured to identify features associated with a particular anatomical structure, pathology, tissue type, procedure, etc.) than the upstream module.
- a module may be a generic module.
- a specialized module may comprise a portal recognition module, an anatomy recognition module, modules associated with specific anatomical structures (e.g., a shoulder module, a knee module), a pathology recognition module, modules associated with a specific pathology (e.g., cancer, defects in the cartilages, rotator cuff tears, dislocated / tom labrum, tom ACL, anterior cruciate ligament, tom meniscus, tom bicep tendon, inflamed synovial tissue, or femoral acetabular impingement (FAI)) or a pathology measurement module.
- a generic module may comprise a tool recognition module, an action recognition module (e.g., drilling, grinding, cleaning, etc.).
- the processing module may comprise a plurality of upstream modules, a plurality of downstream modules, or a combination of both.
- at least one of the plurality of upstream modules may be configured to select at least one downstream module from the plurality of downstream modules of the processing modules to further process an image.
- identifying one or more features identified in the image may further comprise selecting one or more processing modules from the plurality of AI modules, wherein the selection is at least partially based on an output from at least one upstream module.
- a processing module is selected based at least on the identifying the one or more of the region of operation or the approach portal of entry (or portal).
- an upstream module may be configured to identify one or more of an anatomy being operated upon by an operational procedure or an action being performed during the arthroscopic procedure.
- a downstream module may be configured to recognize one or more of an anatomical feature of the identified anatomy associated with the operational procedure or the action being performed.
- a downstream module may be configured to recognize a treatment tool feature associated with the operational procedure or the action being performed.
- the processing modules may comprise an Artificial Intelligence (AI) module.
- AI Artificial Intelligence
- the processing modules may comprise a plurality of AI modules.
- the processing modules or the plurality of AI modules may comprise at least a machine learning algorithm, a deep learning algorithm, or a combination of both.
- the machine learning algorithm may comprise a trained machine learning algorithm.
- the machine learning algorithm may comprise an artificial neural network.
- the processing modules or an AI in the processing modules may comprise at least one dataset.
- the processing modules or an AI in the processing modules may comprise at least a training dataset.
- FIG. 1 is a flow chart that shows an example of a hierarchical arrangement 100 of modules.
- a first module may comprise a region recognition module 110 to determine a field of surgery.
- a region recognition module may receive an image 101 (e.g., a frame from a video feed) from an imaging device (e.g., an arthroscope).
- the region recognition module 110 may not recognize a field of surgery in the image.
- the region recognition module 110 may then stop transfer of the image to other modules and/or discard the frame.
- the region recognition module 110 may recognize a field of surgery in the image.
- the region recognition module 110 may then send the image to other modules downstream of the region recognition module 110 comprising a generic module, a specialized module, or both.
- modules downstream of region recognition module 110 may comprise a portal recognition module 111, a tool recognition module 120, or both.
- the image may be sent to two or more modules substantially simultaneously (e.g., in parallel).
- the image may be sent to two or more modules consecutively (e.g., in series).
- the image may be sent to a tool recognition module 120, an action recognition module 121, or a procedure recognition module 122 in order mentioned herein.
- the tool recognition module 120 may recognize a tool in the image. Subsequently, the module 121 may determine an action based at least in part on the recognized tool.
- the module 121 may then send the image to a procedure recognition module, which then determines a surgical procedure being performed based at least in part on the determined action and/or the recognized tool.
- the region recognition module 110 may also send the image to the portal recognition module 111 in parallel to the tool recognition module 120.
- the portal recognition module 111 may then determine a portal of entry associated with the region recognized by region recognition module 110. Based at least in part on the portal of entry recognized by portal recognition module 111, an image may be sent to one or more specialized modules which form a specialized or customized pipeline.
- Such a specialized pipeline may be specialized to a particular anatomical region or site, (e.g., the shoulder, knee or hip) and may comprise specialized modules (e.g., AI trained for a specific anatomical structure or pathology).
- One or more modules in the specialized pipeline may be activated upon recognition of region of entry (e.g., a bursal region) by the portal recognition module 111.
- labeling an image feature further comprises: a pixel-level masked labeling, a bounding box labeling, a frame-level labeling, or a temporal labeling.
- the pixel-level masked labeling is used to display the labeled anatomical structure or the surgical tool.
- the bounding box labeling is used to display the labeled pathology, the surgical tool, or a foreign object.
- the frame-level labeling is used to display a labeled anatomical region.
- temporal labeling is used to display the labeling of the operational procedure or the recognized action.
- one or more images may be sent to one or more modules to recognize and/or label one or more features on the image(s). Examples of labeled images are shown in FIGs.7A-7B.
- An anatomy recognition module 130 shown in FIG. 1 may receive one or more images from the region recognition module 110 and portal recognition module 111 and may recognize and label anatomical structures (as shown in Fig. 7A). Different color masks 701a, 701b, 701c, 70 Id, along with descriptive labels 702, 703 may be used to differentiate the recognized anatomical structures.
- a pathology recognition module 131 shown in FIG.
- region recognition module 130, anatomy recognition module and any subsequent recognition or measurement module are specialized to a particular anatomical site (e.g., the shoulder or knee) be virtue of information identified or otherwise obtained by portal recognition module 111 or a similar module. Because of this specialization, less memory, programming and processing capability/r are required of a computer system performing analysis and processing of images received from an imaging device used during the surgery.
- FIG. 7B shows an example of a pathology which has been identified and labeled using a colored mask 70 Id as well as a descriptive label 705.
- a tear has been detected and labeled in the labrum (FIG. 7B).
- a pathology measurement module 132 shown in FIG. 1 may receive the image from the pathology recognition module 131 and may provide one or more measurements for the pathology identified and labeled.
- FIG. 2A shows an image of an arthroscopic surgical procedure.
- the image was then provided as an input to the system described herein.
- the image was processed using an image recognition model configured to analyze shoulder surgery images.
- the image recognition model may comprise specialized modules.
- the specialized module may be used to analyze intraarticular region (FIG. 2A-2B).
- FIG. 2B shows an example of labeling features in the image provided in FIG. 2A using the methods and systems described herein.
- Features 210 (bicep tendon (Bicep_Tndn)), 220 (Labrum), 230 (Glenoid), 240 (coracoid process(Cora-pro)), 250 (middle glenohumeral ligament (MGH-Liga)), and 260 (subscapularis (Subscap)) may be provided labels, such as with color masks and a legend LG (which also can provide color or pattern guide for other anatomical features that may be labeled and/or masked, such as the humeral head (Humrl Hd), supra articular process (Supra artclr), descending portion of coracoid (Cora- dee), inner glenohumeral ligament (IGH-Liga), to name a few) as shown in FIG. 2B.
- LG which also can provide color or pattern guide for other anatomical features that may be labeled and/or masked, such as the humeral head (Humrl Hd), supra articul
- FIG. 8A-8V show examples of identifying various features in a surgical procedure using the systems and methods described herein.
- An image from the surgical field of view e.g., a frame of a video stream
- the image recognition model may produce predictions for different classes of anatomical structures (e.g., humeral head, glenoid, subscapularis, bicep tendon, ligaments, tendons, etc.) that can be recognized in the input image (FIGs.
- anatomical structures e.g., humeral head, glenoid, subscapularis, bicep tendon, ligaments, tendons, etc.
- the predictions are combined into an output image provided with a prediction mask (FIG. 8B).
- the output image may contain different color masks to differentiate different classes of anatomical structures identified.
- the classification process described herein may be applied to consecutive frames received from a video camera.
- the labeled output image may then be overlaid on the video stream.
- the overlay may be performed in real-time or in substantially close to real-time.
- FIG. 3A shows an example of labeling a tool, an action, or a procedure in an image captured by an arthroscope.
- the image recognition algorithm recognized and labeled a tool 310 (e.g., a radiofrequency ablation tool). Based on the recognized tool 310, a tool identify label 320 may be provided for the type of tool identified (e.g., a radiofrequency ablation tool), an action recognition module may identify the action being performed was and provide an activity label 330 (e.g., showing a cleaning being performed).
- a procedure recognition module may determine the procedure performed and provide a procedural label 340, for example showing a coronoid decompression procedure being performed.
- FIG. 3B shows another example of labeling a tool, an action, or a procedure in an image captured by an arthroscope.
- the image recognition algorithm recognized and labeled an arthroscopic burrs tool at label 350.
- a tool may comprise a surgical probe, a shaver, a burr grinder, a drill tool, an implant, a drill guides, a radio frequency or other ablation tool, an anchor, a graspers, suture passers, or a scissors.
- grinding was determined to be the action being performed at a label 360 using an action recognition module.
- a procedure recognition module determined the procedure to be a coronoid decompression procedure at a label 370.
- An arthroscopic surgery may comprise actions being performed on areas comprising a shoulder, a knee or a hip.
- a shoulder surgery may be performed in an intraarticular region and/or the bursal region.
- Anatomical structures that may be recognized in the intraarticular region may comprise humeral head, labrum, glenoid, supraspinatus, bicep tendon, glenoid humeral ligaments, subscapularis.
- Anatomical structures that may be recognized in the bursal region may comprise acromian, clavicle, humeral head, bursal, or rotator cuff.
- a surgical procedure may be initiated from one or more portals of entry into the region of operation (e.g., intraarticular region, bursal region, a portal of entry adjacent to a knee).
- a view of the surgical field of view may therefore vary based at least on the portal or an angle of entry used in an operation.
- FIG. 3C shows an example of recognizing an arbitrary tool 380 in a field of surgery.
- the operations further comprise providing a suggested action to an operator (e.g., a surgeon).
- the operator may be performing the procedure (e.g., arthroscopic surgery).
- the operator may be a person other than the surgeon.
- the operator may operate the imaging device (e.g., an arthroscope).
- the suggested action is to assist an operator in the course of the surgery (e.g., an arthroscope).
- a surgeon may require a measurement of a tissue or pathology, and the methods and systems described herein may provide the measurement to the surgeon intraoperatively.
- the suggested action may comprise a safety warning to allow the operator to know of a potential safety issue.
- the system may recognize a distance of one or more implants being implanted in a patient to be different from a predefined safe distance.
- the distance may be a distance between two or more implants from one another.
- the distance may be a distance of one or more implants from an anatomical structure, an anatomical feature, or a pathology.
- a suggested action may comprise avoiding a critical anatomical feature such as, for example, a vein, an artery, nerves, bone, cartilage, or ligaments.
- the suggested action comprising a safety warning is based at least on an identified anatomical feature, a recognized tool, an identified action or a combination thereof.
- a tool may be recognized (e.g., a burr grinder) that may potentially damage a tissue (e.g., a cartilage).
- a burr grinder e.g., a burr grinder in close proximity of a tissue recognized, for example, as cartilage
- the system may generate a safety warning.
- a suggested action may comprise suggesting an approach angle for a device.
- the approach angle may be an approach angle of a drilling tool.
- the suggested action is provided for educational purposes.
- a video stream of a medical procedure e.g., arthroscopy
- the methods and systems described herein may be used to label features overlaid on the video stream (e.g., images or frames of the video stream).
- the suggested actions may also be overlaid on the video stream for educational purposes.
- the suggested action is at least partially based on the labeling (e.g., labeling of the features) of the image.
- the suggested action may be at least in part based on an output from one of the plurality of modules (e.g., upstream modules, or downstream modules).
- the suggested action is at least partially based on the output from at least one of the modules of the plurality of upstream modules.
- the suggested action is at least partially based on the output from at least one of the modules of the plurality of downstream modules.
- the image may be generated from a surgical video stream.
- the surgical video stream is an arthroscopic surgery video stream.
- the surgical video stream is monocular or stereoscopic.
- the labeling of features on an image may be performed at a speed similar to a rate of acquiring images from an imaging device.
- the arthroscope may generate consecutive images (e.g., a video feed) at a rate of at least about 10 frames per second (fps).
- the image recognition algorithm is trained using a database (e.g., a training dataset).
- the database may comprise a plurality of training images.
- the plurality of training images may comprise one or more surgical procedures, surgical tools, surgical tool elements, anatomical structures, or pathologies.
- a training dataset may be generated using an image editing technique.
- An image editing technique may comprise augmenting an image of a region, a portal, an anatomical structure or an anatomical feature with an image of a surgical tool. The augmented image may then be used to train the image recognition algorithm to recognize the tool within a context of the region, the portal, the anatomical structure or the anatomical feature.
- the image editing, or augmentation techniques may comprise rotating a training image to improve a robustness of the image recognition algorithm against a position or an orientation of a patient, for example, during the arthroscopic surgery.
- the image editing, or augmentation techniques may comprise enlarging or cropping the training images to improve the robustness against changes in a depth of view.
- the image editing, or augmentation techniques may comprise flipping a training image along a vertical axis to improve the robustness against procedures performed on a right or a left side of a patient.
- the surgical images used for such training including image editing and augmented training may be selected from a variety of procedures including one or more minimally invasive procedures such as arthroscopic, endoscopic, laparoscopic and cardioscopic procedures which may correspond to one or more of arthroscopic, bariatric, cardiovascular, intestinal, gynecological, urological surgeries or related procedures.
- minimally invasive procedures such as arthroscopic, endoscopic, laparoscopic and cardioscopic procedures which may correspond to one or more of arthroscopic, bariatric, cardiovascular, intestinal, gynecological, urological surgeries or related procedures.
- FIG. 4 is a flowchart of an example of training an image recognition algorithm.
- An AI training method 400 may comprise a dataset 410.
- the dataset 410 may comprise images of a surgical tool, an anatomical structure, an anatomical feature, a surgical tool element, an image acquired from a video feed of an arthroscope, a portal of a surgery, a region of a surgery, etc.
- the dataset may further comprise an imaged that has been edited or augmented using the methods described hereinbefore.
- the images in the dataset 410 may be separated into at least a test dataset 420 and a training dataset 430.
- the dataset 410 may be divided into a plurality of test datasets and/or a plurality of training datasets.
- a training dataset may be used to train an image recognition algorithm.
- a plurality of labeled images may be provided to the image recognition algorithm to train an image recognition algorithm comprising a supervised learning algorithm (e.g., a supervised machine learning algorithm, or a supervised deep learning algorithm).
- Unlabeled images may be used to build and train an image recognition algorithm comprising an unsupervised learning algorithm (e.g., an unsupervised machine learning algorithm, or an unsupervised deep learning algorithm).
- a trained model may be tested using a test dataset (or a validation dataset).
- a test dataset may comprise unlabeled images (e.g., labeled images where a label is removed for testing a trained model).
- the trained image recognition algorithm may be applied to the test dataset and the predictions may be compared with actual labels associated with the data (e.g., images) that were removed to generate the test dataset in a testing model predictions step 460.
- a model training step 440 and a testing model predictions step 460 may be repeated with different training datasets and/or test datasets until a predefined outcome is met.
- the predefined outcome may be an error rate.
- the error rate may be defined as one more of an accuracy, a specificity, or a sensitivity or a combination thereof.
- the tested model 450 may then be used to make a prediction 470 for labeling features in an image from an imaging device (e.g., an arthroscope) being used in the course of a medical procedure (e.g., arthroscopy).
- the prediction may comprise a plurality of predictions 480 comprising a region of a surgery, a portal of the surgery, an anatomy, a pathology, a tool, an action being performed, a procedure being performed, etc.
- FIG. 6 An example of an AI pipeline, as described herein, used in a simulated knee surgery as described herein is provided in FIG. 6.
- a video feed of the surgery may be provided on a first screen 601.
- a second screen may provide an AI inference of the surgical field 602 comprising anatomical masks as described hereinbefore.
- the AI feed may be overlaid on the video feed of the surgery in real-time.
- the AI feed and the video feed of the surgery may be provided on one screen simultaneously.
- the computer-implemented method may comprise receiving at least one image captured by an interventional imaging device; identifying one or more image features in the received at least one image using an image recognition algorithm; labeling the identified one or more image features, wherein the identified one or more image features; and displaying the labeled one or more image features in the at least one image to an operator continuously in the course of the arthroscopic procedure.
- the labeled one or more image features may be displayed in real time or concurrent to the arthroscopic procedure.
- the identified one or more image features may comprise one or more of an anatomical structure, a surgical tool, a surgical tool element, an operational procedure or action, or a pathology.
- the method of training an algorithm may comprise receiving a set of image features based on one or more images relating to the arthroscopic procedure; receiving a training dataset, wherein the training dataset comprises one or more labeled images relating to the arthroscopic procedure; recognizing one or more of the image features in images of the training dataset; building an image recognition algorithm based at least partially on the recognition of the one or more image features and the received training dataset.
- the one or more image features may relate to relate to visual properties of one or more of an anatomical structure, a surgical tool, a surgical tool element, an operational procedure or action, or a pathology.
- the image recognition algorithm may be configured to identify and label the one or more image features in an unlabeled image relating to the arthroscopic procedure.
- the image recognition algorithm may be trained using training data.
- Training data may comprise images, where subject matter experts trace outlines of various anatomical structures, pathologies, tools, etc. The process of training is similar to the example shown in FIG.8A-8V, as described elsewhere herein. These subject matter experts labeled images may be also used to train the algorithm to, for example, recognize a given tool performing a given action.
- a combination of tool recognition, anatomical recognition, and an action recognition may be used to predict a surgical procedure being performed. For example, first an anatomical feature or region (e.g., bursa) may be recognized, which may then trigger other recognition modules to recognize a tool or an action being performed.
- 3B shows an example, where a region of operation is recognized as being bursa, then a burr tool identified with label 350 and a grinding action identified with label 360 may be recognized. The combination of these predictions may lead to recognizing the procedure as the coracoid decompression identified with label 370.
- a tool may not be present in the image.
- the system may still recognize an activity at least partially based on anatomical features being recognized, a previous image analyzed, or a combination thereof.
- the previous image processed by the system may have comprised recognizing a tool, an action, a surgical procedure, or combinations thereof.
- FIG. 9 shows an example of an action (e.g., clearing) identified with a label 920 being recognized by the system, while a tool is not present in the image as shown by a label 910 (showing NA or non-applicability).
- a surgical procedure identified with a label 930 may not be assigned to an image (FIG. 9).
- the operator may swap tools during the course of a surgical procedure. In some embodiments, the swapping of the tools may take several seconds or minutes.
- the system and method described herein can retain a memory of the recognized tool.
- the tool may be obscured or not recognizable (e.g., by movement of camera, orientation of the tool may render the tool unrecognizable).
- an AI architecture may be used.
- the AI architecture may comprise a neural network architecture.
- the neural network architecture may comprise a long short term memory (LSTM).
- FIG. 10 shows an example flowchart of the process of identifying a surgical procedure, as described herein.
- Image frames with annotations 1001 may be received and segmented into one or more segments using one or more classifier models.
- the classifier models may comprise a tool recognition model 1002, an anatomy detection model 1003, an activity detection model 1004, or a feature learning model 1005.
- the outputs from the one or more classifiers may be combined using a long short term memory (LSTM) 1006.
- LSTM is an artificial recurrent neural network (RNN) classifier that may be used to predict based on image recognition at one moment compared to what has been recognized prior.
- RNN artificial recurrent neural network
- LSTM may be used to generate a memory of a context of the images being processed, as described herein.
- the context of the images is then used to predict a stage of the surgery comprising a surgical procedure.
- a rule-based decision to combine the classified segments into one image may then be processed to identify/predict a surgical procedure 1007 being
- Another aspect of the invention provides a system for implementing a hierarchical pipeline for guiding an arthroscopic surgery.
- the system may comprise one or more computer processors and one or more non-transitory computer-readable storage media storing instructions that are operable, when executed by the one or more computer processors, to cause the one or more computer processors to perform operations.
- the operations may comprise (a) receiving at least one image captured by an interventional imaging device; (b) identify one or more image features of a region of treatment or a portal of entry in the region based on at least one upstream module; (c) ) activating a first downstream module to identify one or more image features of an anatomical structure, or a pathology based at least partially on the identified one or more image features in step (b); (d) activating a second downstream module to identify one or more image features of a surgical tool, a surgical tool element, an operational procedure or action relating to the arthroscopic surgery based at least partially on the identified one or more image features in step (b); (e) labeling the identified one or more image features; and displaying the labeled one or more image features in the at least one image continuously to an operator in the course of the arthroscopic surgery.
- the at least one upstream module may comprise a first trained image processing algorithm.
- the downstream module may comprise a second trained image processing algorithm.
- the second downstream module may comprise a
- FIG. 11A shows another example of the process of identifying a surgical procedure, as described herein.
- Image frames with annotations 1101 may be received and segmented into one or more segments using one or more classifier models.
- the classifier models may comprise a tool recognition model 1102, an anatomy detection model 1103, and/or an activity detection model 1104.
- the outputs from the one or more classifiers may be processed and combined using a rule-based decision algorithm 1106.
- a predicted surgical procedure 1107 may be provided as a label overlaid on the surgical video stream.
- FIG. 1 IB shows an example of an output from each of the components across time (e.g., duration of a procedure), described herein. The output shown in FIG.
- the 1 IB may comprise a movement of the recognized tool, anatomy, and/or a change in the activity.
- the movement of a tool may be due to the movement of the camera or a change on the procedure, where a tool is swapped.
- the movement in the anatomy may be due the structural changes in the anatomy due, for example, to the procedure or due to the movement of the camera.
- the operator may pause or change an activity during the procedure and resume the action after the pause.
- An example of an output from the tool recognition model 1102 (e.g., a movement of a recognized tool) is shown in graph 1112; an example of an output from the anatomy detection model 1103 (e.g., a movement of a recognized anatomy) is shown in graph 1113; and an example of an output from the activity detection model 1104 (e.g., a pause or change in the activity or the surgical procedure) is shown in graph 1114.
- the outputs from the tool recognition model 1102, the anatomy detection model 1103, and/or the activity detection model 1104 are combined (e.g., summed) as shown in graphl 115.
- the sum of the outputs is averaged over time to generate an averaged sum as shown in graph 1116.
- the average sum is processed using a smoothing algorithm to generate a smooth average sum.
- the smoothing algorithm may comprise a gaussian smoothing function.
- the smoothing algorithm may comprise a convolutional neural network (e.g., convolutional neural network) with its activity shown in graph 1117.
- FIG. 12 shows a schematic diagram of exemplary elements in AI assisted surgery methods and systems, described herein.
- the systems and methods may comprise elements comprising model development 1201, model repository 1202, and providing (e.g., visualizing) a label or mask 1203.
- Developing AI/machine learning models or algorithms used to process a surgical image may comprise training and/or validation steps 1210.
- Test data 1211 may be used to train and validate AI/machine learning models.
- a feature identified/recognized in an image obtained from a surgery e.g., a frame of a surgical video
- Computer systems may comprise elements comprising model development 1201, model repository 1202, and providing (e.g., visualizing) a label or mask 1203.
- Developing AI/machine learning models or algorithms used to process a surgical image may comprise training and/or validation steps 1210.
- Test data 1211 may be used
- FIG. 5 shows a computer system 501 that is programmed or otherwise configured to perform one or more functions or operations of methods of the present invention.
- the computer system 501 can regulate various aspects of the present invention, such as, for example, of receiving an image from an interventional imaging device, identifying features in the image using an image recognition algorithm, overlaying the features on a video feed on a display device, make recommendations or suggestion to an operator based on the identified features in the image.
- the computer system 501 can be an electronic device of a user or a computer system that is remotely located with respect to the electronic device.
- the electronic device can be a mobile electronic device.
- the computer system 501 includes a central processing unit (CPU, also “processor” and “computer processor” herein) 505, which can be a single core or multi core processor, or a plurality of processors for parallel processing.
- the computer system 501 also includes memory or memory location 510 (e.g., random-access memory, read-only memory, flash memory), electronic storage unit 515 (e.g., hard disk), communication interface 520 (e.g., network adapter) for communicating with one or more other systems, and peripheral devices 525, such as cache, other memory, data storage and/or electronic display adapters.
- the memory 510, storage unit 515, interface 520 and peripheral devices 525 are in communication with the CPU 505 through a communication bus (solid lines), such as a motherboard.
- the storage unit 515 can be a data storage unit (or data repository) for storing data.
- the computer system 501 can be operatively coupled to a computer network (“network”) 530 with the aid of the communication interface 520.
- the network 530 can be the Internet, an internet and/or extranet, or an intranet and/or extranet that is in communication with the Internet.
- the network 530 in some embodiments is a telecommunication and/or data network.
- the network 530 can include one or more computer servers, which can enable distributed computing, such as cloud computing.
- the network 530 in some embodiments with the aid of the computer system 501, can implement a peer-to-peer network, which may enable devices coupled to the computer system 501 to behave as a client or a server.
- the CPU 505 can execute a sequence of machine -readable instructions, which can be embodied in a program or software.
- the instructions may be stored in a memory location, such as the memory 510.
- the instructions can be directed to the CPU 505, which can subsequently program or otherwise configure the CPU 505 to implement methods of the present invention. Examples of operations performed by the CPU 505 can include fetch, decode, execute, and writeback.
- the CPU 505 can be part of a circuit, such as an integrated circuit.
- a circuit such as an integrated circuit.
- One or more other components of the system 501 can be included in the circuit.
- the circuit is an application specific integrated circuit (ASIC).
- the storage unit 515 can store files, such as drivers, libraries and saved programs.
- the storage unit 515 can store user data, e.g., user preferences and user programs.
- the computer system 501 in some embodiments can include one or more additional data storage units that are external to the computer system 501, such as located on a remote server that is in communication with the computer system 501 through an intranet or the Internet.
- the computer system 501 can communicate with one or more remote computer systems through the network 530.
- the computer system 501 can communicate with a remote computer system of a user (e.g., a portable computer, a tablet, a smart display device, a smart tv, etc.).
- remote computer systems include personal computers (e.g., portable PC), slate or tablet PC’s (e.g., Apple® iPad, Samsung® Galaxy Tab), telephones, Smart phones (e.g., Apple® iPhone, Android- enabled device, Blackberry®), or personal digital assistants.
- the user can access the computer system 501 via the network 530.
- Methods as described herein can be implemented by way of machine (e.g., computer processor) executable code stored on an electronic storage location of the computer system 501, such as, for example, on the memory 510 or electronic storage unit 515.
- the machine executable or machine-readable code can be provided in the form of software.
- the code can be executed by the processor 505.
- the code can be retrieved from the storage unit 515 and stored on the memory 510 for ready access by the processor 505.
- the electronic storage unit 515 can be precluded, and machine-executable instructions are stored on memory 510.
- the code can be pre-compiled and configured for use with a machine having a processer adapted to execute the code or can be compiled during runtime.
- the code can be supplied in a programming language that can be selected to enable the code to execute in a pre-compiled or as- compiled fashion.
- aspects of the systems and methods provided herein can be embodied in programming.
- Various aspects of the technology may be thought of as “products” or “articles of manufacture” typically in the form of machine (or processor) executable code and/or associated data that is carried on or embodied in a type of machine readable medium.
- Machine- executable code can be stored on an electronic storage unit, such as memory (e.g., read-only memory, random-access memory, flash memory) or a hard disk.
- “Storage” type media can include any or all of the tangible memory of the computers, processors or the like, or associated modules thereof, such as various semiconductor memories, tape drives, disk drives and the like, which may provide non-transitory storage at any time for the software programming. All or portions of the software may at times be communicated through the Internet or various other telecommunication networks. Such communications, for example, may enable loading of the software from one computer or processor into another, for example, from a management server or host computer into the computer platform of an application server.
- another type of media that may bear the software elements includes optical, electrical and electromagnetic waves, such as used across physical interfaces between local devices, through wired and optical landline networks and over various air-links.
- a machine readable medium such as computer-executable code
- a tangible storage medium such as computer-executable code
- Non-volatile storage media include, for example, optical or magnetic disks, such as any of the storage devices in any computer(s) or the like, such as may be used to implement the databases, etc. shown in the drawings.
- Volatile storage media include dynamic memory, such as main memory of such a computer platform.
- Tangible transmission media include coaxial cables; copper wire and fiber optics, including the wires that comprise a bus within a computer system.
- Carrier-wave transmission media may take the form of electric or electromagnetic signals, or acoustic or light waves such as those generated during radio frequency (RF) and infrared (IR) data communications.
- RF radio frequency
- IR infrared
- Common forms of computer-readable media therefore include for example: a floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD or DVD-ROM, any other optical medium, punch cards paper tape, any other physical storage medium with patterns of holes, a RAM, a ROM, a PROM and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave transporting data or instructions, cables or links transporting such a carrier wave, or any other medium from which a computer may read programming code and/or data.
- the computer system 501 can include or be in communication with an electronic display 535 that comprises a user interface (UI) 540 for providing, for example, an overlay of the identified features on a video feed from an arthroscope or to provide a recommendation to an operator in the course of a surgery.
- UI user interface
- Examples of UFs include, without limitation, a graphical user interface (GUI) and web-based user interface.
- Methods and systems of the present invention can be implemented by way of one or more algorithms.
- An algorithm can be implemented by way of software upon execution by the central processing unit 505.
- the algorithm can, for example, receiving an image from an interventional imaging device, identifying a feature in the image using an image recognition algorithm, overlaying the features on a video feed on a display device, make recommendations or suggestion to an operator based on the identified feature in the image.
- elements, characteristics, or acts from one embodiment can be readily recombined or substituted with one or more elements, characteristics or acts from other embodiments to form numerous additional embodiments within the scope of the invention.
- elements that are shown or described as being combined with other elements can, in various embodiments, exist as standalone elements.
- embodiments of the invention specifically contemplate the exclusion of an element, act, or characteristic, etc. when that element, act or characteristic is positively recited.
- the scope of the present invention is not limited to the specifics of the described embodiments but is instead limited solely by the appended claims.
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Theoretical Computer Science (AREA)
- Medical Informatics (AREA)
- General Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Multimedia (AREA)
- Databases & Information Systems (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Public Health (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Epidemiology (AREA)
- Primary Health Care (AREA)
- Biomedical Technology (AREA)
- Radiology & Medical Imaging (AREA)
- Surgery (AREA)
- Urology & Nephrology (AREA)
- Quality & Reliability (AREA)
- Computational Linguistics (AREA)
- Business, Economics & Management (AREA)
- General Business, Economics & Management (AREA)
- Data Mining & Analysis (AREA)
- Pathology (AREA)
- Image Analysis (AREA)
Abstract
Various embodiments of the invention provide systems and methods to assist or guide an arthroscopic surgery or other surgical procedure e.g., surgery of the shoulder, knee or hip. The method comprises steps of receiving an image from an interventional imaging device, identifying a feature in the image using an image recognition algorithm, overlaying the features on a video feed on a display device and making recommendations or suggestions to an operator based on the identified feature in the image.
Description
SYSTEMS AND METHODS FOR AI-ASSISTED SURGERY
CROSS-REFERENCE TO REUATED APPUICATIONS [0001] This PCT application claims priority to Indian Provisional Patent Application No. 20204105990, filed April 13, 2020, and U.S. Provisional Application No. 63/030,695, filed May 27, 2020, and 63/143,367, filed January 29, 2021, the contents of all of which are fully incorporated herein by reference for all purposes.
BACKGROUND
[0002] Field of the Invention: Embodiments of the invention relate to systems, devices, and methods to guide diagnostic and surgical procedures, particularly using Artificial Intelligence (AI).
[0003] In recent years, Artificial Intelligence has begun to be developed to be used to process images to recognize features of a human face as well as different anatomical structures in a human body. These AI tools can be used to automatically recognize an anatomical feature to assist an operator during a medical procedure. Computational methods such as machine learning and deep learning algorithms can be used for image or language processing to gather and process information generated in a medical procedure. The hope is to use AI algorithms that can then be used to predict or improve the outcome of the surgery or to guide educating new physicians through a virtual or educational procedure. Current AI-assisted surgical systems and methods are still less than ideal in many respects to be used to, for example, guide a surgical procedure. Accordingly, improved AI-assisted surgical systems and methods are desired.
SUMMARY
[0004] Various embodiments of the invention provide computer-implemented medical systems, devices, and methods to guide surgery or other medical procedures. Many embodiments do so by identifying and labeling one or more of anatomical features, pathologies and other features in a surgical field of view in real-time, as well as to provide guidance to an operator are described herein. Any mistake made by an operator in the course of a surgery can be costly. For example, it may be difficult or impossible for an operator to know the exact location of a critical anatomical feature that is hidden from a camera (e.g., a camera used during an arthroscopic or endoscopic surgery), or a part of a pathology that was being removed may be missed from the eyes of an operator or a camera view. Therefore, computer- implemented medical systems, devices, and methods such as Artificial Intelligence (AI) tools, particularly for guiding medical procedures can be valuable. These AI tools can have their limitations in accurately and reliably predicting a tool, anatomical structure, or detecting a procedure. In a fast-paced surgical procedure, the AI tool needs to also make predictions with low latency to provide real time assistance to an operator.
[0005] Recognized herein is the need for fast, accurate and reliable AI tools to assist an operator in real time during the course of a surgical operation to improve an outcome of the surgery. Accordingly, aspects of the present invention provide a pipeline of machine learning algorithms that is versatile and
well trained for unique needs of various medical procedures including various minimally invasive procedures such as arthroscopic, endoscopic, laparoscopic, cardioscopic and related procedures. Examples of such minimally invasive procedures can include one or more of Arthroscopic procedures (e.g., repair of a tom rotator cuff in the shoulder, ACL surgery of the knee, repair of various non-arthritic hip disorders, repair of damaged cartilage in the ankle or removal of bone spurs in the ankle); Gastro-intestinal (GI) procedures (e.g., biopsy of the intestines, removal of polyps, bariatric surgery, stomach stapling/vertical banded gastroplasty), urological procedures (e.g., removal of kidney stone, bladder repair), gynecological procedures (e.g., a dnc, removal of uterine fibroids) and a laparoscopic procedures (e.g., an appendectomy, cholecystectomy, colectomy, hernia repair, nissen fundoplication.
[0006] Various embodiments of the invention provide systems, devices, and methods that can receive information (e.g., image(s), voice, user inputs) during a medical procedure (e.g., a surgery), process the received information to identify features associated with the procedure, and provide recommendations based on the identified features. These features may include anatomical parts or devices, or various steps or outcomes of a procedure. Based on identifying the various features in an operation, the systems, devices, and methods described herein can assist a surgeon in the operation by providing recommendations that can include approach angle of a tool, measurement of an anatomical feature or pathology, an action that should be taken or avoided, to name a few.
[0007] Aspects of the present invention also aid surgeons (and related medical personnel) intraoperatively by using the images from the surgical field of view and applying Artificial Intelligence (AI) to provide guidance and assistance to the surgeons and other medical staff. We refer to AI modules/algorithms used intraoperatively as Surgical AI.
[0008] In a first aspect, the invention provides systems for guiding an arthroscopic procedure. In some embodiments, the systems comprise one or more computer processors and one or more non- transitory computer-readable storage media storing instructions that are operable, when executed by the one or more computer processors, to cause the one or more computer processors to perform operations comprising: receiving at least one image captured by an interventional imaging device; identifying one or more image features in the received at least one image using an image recognition algorithm; labeling the identified one or more image features; and displaying the labeled one or more image features in the at least one image to an operator continuously in the course of the arthroscopic procedure. In some embodiments, the identified one or more image features comprise one or more of an anatomical structure, a surgical tool, a surgical tool element, an operational procedure or an action or a pathology such as tom or injured tissue. Application of embodiments of the system to the guidance of other medical procedures including minimally invasive procedures such as endoscopic, laparoscopic, and interventional cardiovascular procedures is also contemplated.
[0009] In various embodiments, the labeled one or more image features may be displayed in real time or concurrent to the arthroscopic procedure. In some embodiments, the arthroscopic procedure is an
arthroscopic surgery. In some embodiments, the image recognition algorithm comprises a hierarchical arrangement of processing modules also referred to as software modules or modules.
[0010] In some embodiments, the processing modules comprise a plurality of Artificial Intelligence (AI) modules which, in various embodiments, may correspond to a machine learning algorithm, a deep learning algorithm, or a combination of both. In some embodiments, the machine learning algorithm comprises an artificial neural network. Also in various embodiments, the processing modules comprise at least one dataset which may include at least one training dataset.
[0011] In various embodiments, the processing modules comprise an upstream module and a downstream module, where the downstream module is more specialized than the upstream module. In some embodiments, the upstream module is configured to identify one or more of an anatomy being operated upon by an operational procedure or an action being performed during the arthroscopic procedure. In some embodiments, the downstream module is configured recognize one or more of an anatomical features of the identified anatomy or a treatment tool feature associated with the operational procedure or the action being performed. In some embodiments, the processing modules comprise a plurality of upstream modules and a plurality of downstream modules. In some embodiments, at least one of the modules of the plurality of upstream modules is configured to select an individual downstream module from the processing modules for use.
[0012] In some embodiments, the operational procedure or the action is identified by one or more of the processing modules based at least partially on identification of a surgical tool used during the procedure, for example an arthroscope vs an endoscope. In some embodiments, the interventional imaging device is an arthroscope. In some embodiments, the interventional imaging device is an endoscope. In some embodiments, the image recognition algorithm is configured to identify one or more of a region of operation or an approach portal of entry of an arthroscopic procedure or other medical procedure. In various embodiments, the region of operation may correspond to one or more of a shoulder, a knee or a hip.
[0013] In some embodiments, at least one module from the processing modules is selected based at least on identification of one or more of the region of operation (e.g., the shoulder) or the approach portal of entry (e.g., a bursal approach for the shoulder, an anterior approach for the knee). In some embodiments, the operations performed further comprise storing the at least one image in a memory device. In some embodiments, the operations further comprise discarding the at least one image after the displaying the label elements to optimize memory usage.
[0014] In various embodiments, the labeled one or more image features further comprises: a pixel- level masked labeling, a bounding box labeling, a frame-level labeling, or a temporal labeling which may configured to be used for various purposes. For example, in some embodiment, the pixel-level masked labeling may be used to display the labeled anatomical structure or the surgical tool. Also in one or more embodiments, the bounding box labeling may be used to display one or more of the labeled pathology, the surgical tool, or a foreign object. Further according to one or more embodiments, the frame-level labeling
may be used to display a labeled anatomical region for example that of the shoulder and the temporal labeling may be used to display the labeled operational procedure or action (e.g., a surgical action such as tissue resection, ablation or suturing).
[0015] In various embodiments, the operations further comprise providing a suggested action which may be based on a number of factors including results or other output from various modules. For example, in one or more embodiments, the suggested action may be at least partially based on one more of the following: the labeling of the at least one image, results or other output from the upstream or result or other output from the downstream module. Also, in various embodiments the suggested action may have a variety of purposes depending the surgery and situation. For example, the suggested action may be used to assist the operator in the course of the surgery while in other it may for educational or training purposes such as when a surgeon is learning a new procedure and/or during a simulated surgery. In additional embodiments, the suggested action comprises providing a safety warning based at least on identification of a critical anatomical structure (e.g., an artery), a surgical tool, an action, a distance between two or more implants (e.g., if its above or below a critical event) or an adverse event (e.g., bleeding in the surgical field) or biomedical/physiologic data of the patient (e.g., drop in blood p02, blood pressure, respiration rate, irregular heart rhythm (e.g., arrhythmia or other important physiologic parameter of the patient)
[0016] Various embodiments of the invention may also provide suggested surgical actions based one or more of dimensions, positioning and arrangement of structures including anatomical structures and implants within the surgical field. For example, in some surgical situations in which embodiments of the invention are used, the distance between the two or more implants may create a health risk. In particular the actual or allowable distance between two or more positioned implants positioned in the surgical field is different from a predefined distance of the two or more implants making an approach to the surgical field including approaches for position the implants difficult and/or hard to approximate by the human eye particularly when done on screen in two-dimensional view. Accordingly, in these and related embodiments, the suggested action may comprises providing a suggested approach angle which may be suggested a drilling angle for placement of an implant or other surgical action.
[0017] Various embodiments of the system may be configured for a number of joint surgeries including for example one or more of shoulder surgery, knee surgery, hip surgery, ankle surgery, hand surgery or elbow surgery. In these and related embodiments one or more of the processing modules described herein may be adapted for each particular site including having data on each particular which may include specialized training data sets.
[0018] In some embodiments, the image recognition algorithm is trained using a database. In some embodiments, the database comprises a plurality of training images comprising one or more surgical procedures, surgical tools, surgical tool elements, anatomical structures, or pathologies. In some embodiments, a plurality of augmentation techniques is used to improve the training dataset to improve a robustness of the image recognition algorithm. The surgical images used for such training including
augmented training may be selected from a variety of procedures including one or more minimally invasive procedures such as arthroscopic, endoscopic, laparoscopic and cardioscopic procedures which may correspond to one or more of arthroscopic, bariatric, cardiovascular, intestinal, gynecological, urological surgeries or related procedures.
[0019] In some embodiments, the augmentation techniques comprise rotating the training images to improve the robustness against a position or an orientation of a patient during the arthroscopic procedure. In some embodiments, the augmentation techniques comprise flipping the training images along a vertical axis to improve the robustness against procedures performed on a right or a left side of a patient. In some embodiments, the augmentation techniques comprise enlarging or cropping the training images to improve the robustness against changes in a depth of view.
[0020] In some embodiments, the at least one image is generated from a surgical video stream. In some embodiments, the surgical video stream is an arthroscopic surgery video stream. In various embodiments, the surgical video stream may be monocular or stereoscopic. In these and related embodiments, embodiment of the system may be adapted to receive and recognize each type of view (monocular and stereoscopic) and switch back and forth between appropriate processing methods when the views are switch.
[0021] Another aspect of the present invention provides computer-implemented methods for guiding an arthroscopic procedure. In some embodiments, the methods comprise: receiving at least one image captured by an interventional imaging device; identifying one or more image features in the received at least one image using an image recognition algorithm; labeling the identified one or more image features; and displaying the labeled one or more image features in the at least one image to an operator continuously in the course of the arthroscopic procedure. In some embodiments, the identified one or more image features comprising one or more of an anatomical structure, a surgical tool, a surgical tool element, an operational procedure or action, or a pathology. In some embodiments, the labeled one or more image features are displayed in real time or concurrent to the arthroscopic procedure.
[0022] In some embodiments, the arthroscopic procedure is an arthroscopic surgery. In some embodiments, the image recognition algorithm comprises a hierarchical arrangement of processing modules. In some embodiments, the processing modules comprise a plurality of Artificial Intelligence (AI) modules. In some embodiments, the processing modules comprise at least a machine learning algorithm, a deep learning algorithm, or a combination of both. In some embodiments, the machine learning algorithm comprises an artificial neural network.
[0023] In some embodiments, the processing modules comprise at least one dataset which may include at least one training dataset. In some embodiments, the processing modules comprise an upstream module and a downstream module, where the downstream module being more specialized than the upstream module. In some embodiments, the upstream module is configured to identify one or more of an anatomy being operated upon by an operational procedure or an action being performed during the arthroscopic procedure. In some embodiments, the downstream module is configured to one or more of
recognize an anatomical feature of the identified anatomy or recognize a treatment tool feature associated with the operational procedure or the action being performed. In some embodiments, the processing modules comprise a plurality of upstream modules or a plurality of downstream modules. In some embodiments, the at least one of the modules of the plurality of upstream modules is configured to select an individual downstream module from the processing modules for use. In some embodiments, the operational procedure or the action is identified partially based on the identifying the surgical tool.
[0024] In some embodiments, the interventional imaging device is an arthroscope. In some embodiments, the interventional imaging device is an endoscope. In some embodiments, the image recognition algorithm is configured to identify one or more of a region of operation or an approach portal of entry of the arthroscopic procedure. In some embodiments, the region of operation is a shoulder. In some embodiments, the region of operation is a knee. In some embodiments, at least one module from the processing modules is selected based at least on the identifying the one or more of the region of operation or the approach portal of entry.
[0025] In some embodiments, the operations performed further comprise storing the at least one image in a memory device. In some embodiments, the operations further comprise discarding the at least one image after the displaying the label elements to optimize memory usage.
[0026] In some embodiments, the labeled one or more image features further comprises: a pixel-level masked labeling, a bounding box labeling, a frame-level labeling, or a temporal labeling. In some embodiments, the pixel-level masked labeling is used to display the labeled anatomical structure or the surgical tool. In some embodiments, the bounding box labeling is used to display the labeled pathology, the surgical tool, or a foreign object. In some embodiments, the frame-level labeling is used to display a labeled anatomical region. In some embodiments, the temporal labeling is used to display the labeling the operational procedure or the action.
[0027] In some embodiments, the operations further comprise providing a suggested action. In some embodiments, the suggested action is at least partially based on the labeling of the at least one image. In some embodiments, the suggested action is at least partially based on the upstream module. In some embodiments, the suggested action is at least partially based on the downstream module. In some embodiments, the suggested action is to assist the operator in the course of the surgery. In some embodiments, the suggested action is provided for educational purposes. In some embodiments, the suggested action comprises providing a safety warning based at least on identification of a critical anatomical structure, or a distance of two or more implants.
[0028] In some embodiments, the distance of the two or more implants creates a health risk. In some embodiments, the distance of the two or more implants is different from a predefined distance of the two or more implants. In some embodiments, the suggested action comprises providing a suggested approach angle. In some embodiments, the suggested approach angle may comprise a drilling angle.
[0029] In some embodiments, the method is configured for a shoulder surgery. In some embodiments, the method is configured for a knee surgery. In some embodiments, the image recognition
algorithm is trained using a database. In some embodiments, the database comprises a plurality of training images comprising one or more surgical procedures, surgical tools, surgical tool elements, anatomical structures, or pathologies.
[0030] In some embodiments, a plurality of augmentation techniques is used to improve the training dataset to improve a robustness of the image recognition algorithm. In some embodiments, the augmentation techniques comprise rotating the training images to improve the robustness against a position or an orientation of a patient during the arthroscopic procedure. In some embodiments, the augmentation techniques comprise flipping the training images along a vertical axis to improve the robustness against procedures performed on a right or a left side of a patient. In some embodiments, the augmentation techniques comprise enlarging or cropping the training images to improve the robustness against changes in a depth of view.
[0031] In some embodiments, the at least one image is generated from a surgical video stream. In some embodiments, the surgical video stream is an arthroscopic surgery video stream. In some embodiments, the surgical video stream is monocular. In some embodiments, the surgical video stream is stereoscopic.
[0032] Another aspect of the present invention provides a method of training an algorithm for guiding an arthroscopic procedure. In some embodiments, the method comprises: receiving a set of image features based on one or more images relating to the arthroscopic procedure; receiving a training dataset; recognizing one or more of the image features in images of the training dataset; and building an image recognition algorithm based at least partially on the recognition of the one or more image features and the received training dataset. In some embodiments, the training dataset comprises one or more labeled images relating to the arthroscopic procedure. In some embodiments, the training dataset comprises one or more labeled images relating to the arthroscopic procedure one or more image features relate to visual properties of one or more of an anatomical structure, a surgical tool, a surgical tool element, an operational procedure or action, or a pathology. In some embodiments, the image recognition algorithm is configured to identify and label the one or more image features in an unlabeled image relating to the arthroscopic procedure.
[0033] In some embodiments, the labeled one or more image features are displayed in real time or concurrent to the arthroscopic procedure. In some embodiments, the image recognition algorithm comprises a hierarchical arrangement of processing modules. In some embodiments, the processing modules comprise a plurality of individual image processing modules. In some embodiments, the plurality of individual image processing modules comprises a first module for identifying the arthroscopic procedure in place, a second module for recognizing and labeling one or more surgical tools and surgical tool elements, a third module for recognizing and labeling one or more anatomical structures, or a combination thereof.
[0034] In some embodiments, the processing modules comprise a plurality of Artificial Intelligence (AI) modules. In some embodiments, the processing modules comprise at least a machine learning
algorithm, a deep learning algorithm, or a combination of both. In some embodiments, the machine learning algorithm comprises an artificial neural network.
[0035] In some embodiments, the processing modules comprise at least one dataset. In some embodiments, the processing modules comprise at least one training dataset.
[0036] In some embodiments, the processing modules comprise an upstream module and a downstream module, where the downstream module being more specialized than the upstream module. In some embodiments, the upstream module is configured to identify one or more of an anatomy being operated upon by an operational procedure or an action being performed during the arthroscopic procedure. In some embodiments, the downstream module is configured to one or more of recognize an anatomical feature of the identified anatomy or recognize a treatment tool feature associated with the operational procedure or the action being performed. In some embodiments, the processing modules comprise a plurality of upstream modules or a plurality of downstream modules. In some embodiments, at least one of the modules of the plurality of upstream modules is configured to select an individual downstream module from the processing modules for use.
[0037] In some embodiments, identifying one or more image features in the at least one image further comprises selecting one or more processing modules from a plurality of processing modules, where the selection is at least partially based on the region and/or the portal of the arthroscopic surgery. In some embodiments, the operational procedure or the action is identified partially based on the identifying the surgical tool. In some embodiments, the unlabeled image is captured by an interventional imaging device. In some embodiments, the interventional imaging device is an endoscope. In some embodiments, the unlabeled image is generated from a surgical video stream.
[0038] In some embodiments, the surgical video stream is an arthroscopic surgery video stream where the endoscopic surgery is an arthroscopic surgery. In some embodiments, the surgical video stream is monocular. In some embodiments, the surgical video stream is stereoscopic.
[0039] In some embodiments, the image recognition algorithm is configured to identify one or more of a region of operation or an approach portal of entry of the arthroscopic procedure. In some embodiments, the region of operation is a shoulder. In some embodiments, the region of operation is a knee. In some embodiments, at least one module from the processing modules is selected based at least on the identifying the one or more of the region of operation or the approach portal of entry.
[0040] In some embodiments, the image recognition algorithm stores the labeled image in a memory device. In some embodiments, the image recognition algorithm discards the labeled image to minimize memory usage.
[0041] In some embodiments, the labeled image comprises: a pixel-level masked labeling, a bounding box labeling, a frame-level labeling, or a temporal labeling. In some embodiments, the pixel- level masked labeling is used to display the labeled anatomical structure or the surgical tool. In some embodiments, the bounding box labeling is used to display the labeled pathology, the surgical tool, or a foreign object. In some embodiments, the frame-level labeling is used to display a labeled anatomical
region. In some embodiments, the temporal labeling is used to display the labeling the operational procedure or action.
[0042] In some embodiments, the training dataset is configured for a shoulder surgery. In some embodiments, the image recognition algorithm is trained for a shoulder surgery using the training dataset configured for a shoulder surgery. In some embodiments, the training dataset is configured for a knee surgery. In some embodiments, the image recognition algorithm is trained for a knee surgery using the training dataset configured for a knee surgery. In some embodiments, the training dataset comprises a plurality of training images comprising one or more surgical procedures, surgical tools, surgical tool elements, anatomical structures, or pathologies.
[0043] In some embodiments, a plurality of augmentation techniques is used to improve the training dataset to improve a robustness of the image recognition algorithm. In some embodiments, the augmentation techniques comprise rotating the training images to improve the robustness against a position or an orientation of a patient during the arthroscopic procedure. In some embodiments, the augmentation techniques comprise flipping the training images along a vertical axis to improve the robustness against procedures performed on a right or a left side of a patient. In some embodiments, the augmentation techniques comprise enlarging or cropping the training images to improve the robustness against changes in a depth of view.
[0044] Another aspect of the present invention provides a method for implementing a hierarchical pipeline for guiding an arthroscopic surgery. In some embodiments, the system comprises one or more computer processors and one or more non-transitory computer-readable storage media storing instructions that are operable, when executed by the one or more computer processors, to cause the one or more computer processors to perform operations comprising: (a) receiving at least one image captured by an interventional imaging device; (b) identify one or more image features of a region of treatment or a portal of entry in the region based on at least one upstream module; (c) activating a first downstream module to identify one or more image features of an anatomical structure, or a pathology based at least partially on the identified one or more image features in step (b); (d) activating a second downstream module to identify one or more image features of a surgical tool, a surgical tool element, an operational procedure or action relating to the arthroscopic surgery based at least partially on the identified one or more image features in step (b); (e) labeling the identified one or more image features; and (f) displaying the labeled one or more image features in the at least one image continuously to an operator in the course of the arthroscopic surgery.
[0045] In some embodiments, the at least one upstream module comprises a first trained image processing algorithm. In some embodiments, the at least one upstream module comprises a first trained image processing algorithm first downstream module comprises a second trained image processing algorithm. In some embodiments, the at least one upstream module comprises a first trained image processing algorithm first downstream module comprises a second trained image processing algorithm second downstream module comprises a third trained image processing algorithm. In some embodiments,
the steps (c) and (d) are independent from one another. In some embodiments, the first, second, or third trained image processing algorithms comprise at least a machine learning algorithm, a deep learning algorithm, or a combination of both. In one or more embodiments, the machine learning algorithm includes an artificial neural network.
[0046] In some embodiments, the machine learning or the deep learning algorithms are trained using at least one training dataset. In some embodiments, the training dataset is configured for a shoulder surgery. In some embodiments, the training dataset is configured for a knee surgery. In some embodiments, the training dataset comprises a plurality of training images comprising one or more surgical procedures, surgical tools, surgical tool elements, anatomical structures, or pathologies.
[0047] In some embodiments, a plurality of augmentation techniques is used to improve the training dataset to improve a robustness of the image recognition algorithm. In some embodiments, the augmentation techniques comprise rotating the training images to improve the robustness against a position or an orientation of a patient during the arthroscopic procedure. In some embodiments, the augmentation techniques comprise flipping the training images along a vertical axis to improve the robustness against procedures performed on a right or a left side of a patient. In some embodiments, the augmentation techniques comprise enlarging or cropping the training images to improve the robustness against changes in a depth of view.
[0048] In some embodiments, the first, second, or third trained image processing algorithms store the displayed images with labeled features in a memory device. In some embodiments, the first, second, or third trained image processing algorithms discard the displayed images with labeled features to minimize memory usage.
[0049] In some embodiments, the arthroscopic procedure is an arthroscopic surgery. In some embodiments, the at least one image is generated from a surgical video stream. In some embodiments, the surgical video stream is an endoscopic surgery video stream. In some embodiments, the surgical video stream is monocular. In some embodiments, the surgical video stream is stereoscopic.
[0050] Another aspect of the present invention provides a non-transitory computer readable medium comprising machine executable code that, upon execution by one or more computer processors, implements any of the methods above or elsewhere herein.
[0051] Another aspect of the present invention provides a system comprising one or more computer processors and computer memory coupled thereto. The computer memory comprises machine executable code that, upon execution by the one or more computer processors, implements any of the methods above or elsewhere herein.
[0052] Embodiments of the systems and methods of the invention are particularly useful in aiding the surgeon during surgery such as arthroscopic surgier by providing real time display of anatomical structures, pathologies, pathology repair and selected measurements within the surgical field. Such display reduces errors due to misidentified tissue structures and measurements in turn leading to improved surgical and patient outcomes. It also does so by reducing the cognitive load on the surgeon allowing
them to focus on the procedure and at the time alerting them to potentially adverse actions, conditions or results during the course of the surgery that they may otherwise miss due to focus or hyper focus on critical tasks. Such alerts prevent or reduce potential adverse events during surgery by alerting the surgeon before they happen and/or contemporaneously to their occurrence to allow the surgeon to take appropriate action for their prevention or mitigation. In use such alerts lead to both reduced morbidity and mortality as well as improved acute surgical outcomes and long-term outcomes for the patient as the surgery is performed with no or fewer errors and with more precision and accuracy in the surgical actions performed (e.g., accuracy of implant placement or removal of damaged or diseased, tissue while preserving healthy tissue).
[0053] Additional aspects and advantages of the present invention will become readily apparent to those skilled in this art from the following detailed description, wherein only illustrative embodiments of the present invention are shown and described. As will be realized, the present invention is capable of other and different embodiments, and its several details are capable of modifications in various obvious respects, all without departing from the disclosure. Accordingly, the drawings and description are to be regarded as illustrative in nature, and not as restrictive.
INCORPORATION BY REFERENCE
[0054] All publications, patents, and patent applications mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. To the extent publications and patents or patent applications incorporated by reference contradict the disclosure contained in the specification, the specification is intended to supersede and/or take precedence over any such contradictory material.
BRIEF DESCRIPTION OF THE DRAWINGS
[0055] The novel features of the present invention are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present invention will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the invention are utilized, and the accompanying drawings (also “Figure” and “FIG.” herein), of which:
[0056] FIG. 1 shows a flow chart of an example of hierarchical arrangement of modules in a system for AI-assisted surgery, according to some embodiments.
[0057] FIG. 2A shows an image of an arthroscopic surgical procedure, according to some embodiments. [0058] FIG. 2B shows an example of labeling features in an arthroscopic surgical procedure, according to some embodiments.
[0059] FIG. 3A shows an example of labeling a tool, an action, or a procedure in an image captured by an arthroscope, according to some embodiments.
[0060] FIG. 3B shows another example of labeling a tool, an action, or a procedure in an image captured by an arthroscope, according to some embodiments.
[0061] FIG. 3C shows an example of recognizing an arbitrary tool in a field of surgery, according to some embodiments.
[0062] FIG. 4 shows a flowchart of an example of training an image recognition algorithm, according to some embodiments.
[0063] FIG. 5 shows a computer system that is programmed or otherwise configured to implement methods provided herein, according to some embodiments.
[0064] FIG. 6 shows an example of an AI pipeline used in a simulated knee surgery, according to some embodiments.
[0065] FIG. 7A shows an example of recognizing and labeling anatomical structures, according to some embodiments.
[0066] FIG. 7B shows an example of recognizing and labeling a pathology, according to some embodiments.
[0067] FIGs. 8A shows an example of an image of an arthroscopic surgical procedure, according to some embodiments.
[0068] FIG. 8B shows an example of an image of an arthroscopic surgical procedure labeled using image recognition algorithms, according to some embodiments.
[0069] FIGs. 8C-8L show examples of anatomical structures in an image labeled by subject matter experts, according to some embodiments.
[0070] FIGs. 8M-8V show examples of anatomical structures in an image labeled by AI algorithms, according to some embodiments.
[0071] FIG. 9 shows an example of an action recognized by the system for AI-assisted surgery, according to some embodiments.
[0072] FIG. 10 shows an example flowchart of the process of identifying a surgical procedure, according to some embodiments.
[0073] FIG. 11A shows another example of the process of identifying a surgical procedure, according to some embodiments.
[0074] FIG. 11B shows an example of an output from tool detection, anatomy detection, and activity detection models across time, according to some embodiments.
[0075] FIG. 12 shows a schematic diagram of elements in an AI assisted surgery system, according to some embodiments.
DETAILED DESCRIPTION
[0076] While various embodiments of the present invention have been shown and described herein, it will be obvious to those skilled in the art that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions may occur to those skilled in the art without departing
from the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed as is explained in more detail below.
[0077] Various embodiments of the invention provide icomputer-implemented medical systems, devices, and methods for assisting surgeons in an intraoperative setting using AI. The systems, devices, and methods disclosed herein may improve upon existing methods of surgical assistance by providing an improved classification (e.g., real-time) of various elements involved in a surgical operation (e.g., surgical tools, anatomical features, anatomical pathology/injury features, operation procedures). One or more embodiments of the systems, devices, and methods provided herein may achieve this objective through the use AI methods (e.g., machine learning, deep learning) to build a classifier which improves a real-time classification of elements involved in a surgical operation. Various embodiment of AI approaches and implementation described herein may leverage large datasets in order gain new insights from the datasets. The classifier model may improve real-time characterization of various elements involved in an operation which may lead to higher operation success rate for example due to fewer errors due to misidentified anatomical structures. The classifier model may provide an operator (e.g., surgeon, operating room nurse, surgical technician) with information for making more accurate and timely decisions (e.g., labeling critical anatomical features in real-time). This leads to fewer errors and more the performance of more accurate and precise surgical actions such as the placement of an implant (e.g., an anchor) at desired location or more complete removal of damaged or diseased tissue while preserving healthy tissue (e.g., such as a frayed tendon or cartilage or tumorous tissue) from a particular anatomical structure(s) at the surgical site. [0078] The computer-implemented medical systems, devices, and methods as disclosed herein may improve upon existing methods of clinical decision support systems by leveraging parameters related to various elements in the context of a surgical operation to generate accurate real-time decisions continuously. A surgical operation involves various elements such as the patient’s body with various parts and anatomical complexity, plurality of tools and devices, actions based on the surgical procedure as well as actions based on a non-routine event that may occur in the surgery. The systems, devices, and methods disclosed herein can operate continuously with the changing environment of the surgical operation to make classifications and suggestions based on a plurality of AI modules organized to make decisions in a hierarchical manner. For example, the classifier described herein can classify anatomical features (e.g., shoulder, knee, organ, tissue, or pathology) while a view of the endoscopic camera may change during the operation. Similarly, the systems, devices, and method described here can recognize surgical tools as they appear in the field of view.
[0079] The systems, devices, and methods as disclosed herein may be used to classify various elements involved in an operation. For example, the classifier disclosed herein can identify and label anatomical structures (e.g., anatomical parts, organs, tissues), surgical tools, or a procedure being performed in an operation.
[0080] The invention may help with recognizing critical structures (e.g., nerves, arteries, veins, bone, cartilage, ligaments) or pathologies (e.g., a tissue that needs to be removed). The critical structures may be
visible or hidden in the field of view. The systems, devices, and methods described herein can identify and mark the critical structures (e.g., color markings on the video stream). Systems according to many embodiments may comprise a plurality of surgical AI modules organized to make decisions in a hierarchical manner. The surgical AI assist modules disclosed herein may analyze video feeds of the surgical field of view and render decisions in a substantially continuous manner. These modules can be connected by a messaging framework which can process the flow of video data, and a plurality of outputs (e.g., decisions). The AI systems may react to changes in the surgical field of view. The methods and systems of the disclosed invention may be applied to various surgical operations including different anatomical regions and organs. The surgical AI assist can work when surgeons change views, i.e., access the surgical field from different approach portals.
[0081] Reference will now be made in detail to various embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present invention and the described embodiments. However, the embodiments of the present invention are optionally practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments. In the drawings, like reference numbers designate like or similar steps or components.
[0082] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the claims. As used in the description of the embodiments and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
[0083] As used herein, the term "if1 is optionally construed to mean "when" or "upon" or "in response to determining" or "in accordance with a determination" or "in response to detecting," that a stated condition precedent is true, depending on the context. Similarly, the phrase "if it is determined [that a stated condition precedent is true]" or "if [a stated condition precedent is true]" or "when [a stated condition precedent is true]" is optionally construed to mean "upon determining" or "in response to determining" or "in accordance with a determination" or "upon detecting" or "in response to detecting" that the stated condition precedent is true, depending on the context.
[0084] As used herein, and unless otherwise specified, the term "about" or "approximately" means an acceptable error for a particular value as determined by one of ordinary skill in the art, which depends in part on how the value is measured or determined. In certain embodiments, the term "about" or "approximately" means within 1, 2, 3, or 4 standard deviations. In certain embodiments, the term "about" or "approximately" means within 30%, 25%, 20%, 15%, 10%, 9%, 8%, 7%, 6%, 5%, 4%, 3%, 2%, 1%, 0.5%, 0.1%, or 0.05% of a given value or range.
[0085] As used herein, the terms “comprises,” “comprising,” or any other variation thereof, are intended to cover a nonexclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
[0086] As used herein, the terms “subject” and “patient” are used interchangeably. As used herein, the terms “subject” and “subjects” refers to a human being. In certain embodiments, the subject is going through a surgical operation. In certain embodiments, the subject is 0 to 6 months old, 6 to 12 months old, 1 to 5 years old, 5 to 10 years old, 10 to 15 years old, 15 to 20 years old, 20 to 25 years old, 25 to 30 years old, 30 to 35 years old, 35 to 40 years old, 40 to 45 years old, 45 to 50 years old, 50 to 55 years old, 55 to 60 years old, 60 to 65 years old, 65 to 70 years old, 70 to 75 years old, 75 to 80 years old, 80 to 85 years old, 85 to 90 years old, 90 to 95 years old or 95 to 100.
[0087] Whenever the term “at least,” “greater than,” or “greater than or equal to” precedes the first numerical value in a series of two or more numerical values, the term “at least,” “greater than” or “greater than or equal to” applies to each of the numerical values in that series of numerical values. For example, greater than or equal to 1, 2, or 3 is equivalent to greater than or equal to 1, greater than or equal to 2, or greater than or equal to 3.
[0088] Whenever the term “no more than,” “less than,” or “less than or equal to” precedes the first numerical value in a series of two or more numerical values, the term “no more than,” “less than,” or “less than or equal to” applies to each of the numerical values in that series of numerical values. For example, less than or equal to 3, 2, or 1 is equivalent to less than or equal to 3, less than or equal to 2, or less than or equal to 1.
[0089] The term “surgical AI” or “surgical AI module”, as used herein, generally refer to a system, device, or method that uses Artificial Intelligence algorithms to assist before, during, and/or after a surgical operation. A surgical AI module can be defined as a combination of input data, machine learning or deep learning algorithms, training datasets, or other datasets.
[0090] The term “machine learning”, as used herein, may generally refer to computer algorithms that can improve automatically overtime. Any description herein of machine learning can be applied to Artificial Intelligence, and vice versa, or any combination thereof.
[0091] As used herein, the terms “continuous,” “continuously” or any other variation thereof, generally refer to a substantially uninterrupted process or a process with time delay that is acceptable in the context of the process.
[0092] The terms “video stream” or “video feed”, as used herein, refer to data generated by a digital camera. Video feed may be a sequence of static or moving pictures.
[0093] The terms “region,” “organ,” “tissue,” “structure”, as used herein, may generally refer to anatomical features of the human body. A region may be larger than an organ and may comprise an organ. An organ may comprise one or more tissue types and structures. A Tissue may refer to a group of cells structurally joined to complete a common function. A structure can refer to a part of a tissue. In some
embodiments, a structure may refer to one or more parts of one or more tissues joined together to create an anatomical feature.
[0094] The terms “surgical field of view,” or “field of view,” as used herein, refer to the extent of visibility captured by an interventional imaging device. Field of view may refer to the extent of visual data captured by a digital camera that is observable by human eye.
[0095] The term “decision,” as described herein, may refer to outputs from a machine learning or AI algorithm. A decision may comprise labeling, classification, prediction, etc.
[0096] The term “interventional imaging device,” as used herein, generally refers to an imaging device used for medical purposes. The interventional imaging device may refer to an imaging device that is used in a surgical operation. The surgical operation, in some embodiments, may be a simulation of an operation.
[0097] The term “operator,” used herein, refers to a medical professional involved in a surgical operation. An operator can be a surgeon, an operating room nurse, a surgical technician.
[0098] One aspect of the invention provides a system for guiding an arthroscopic procedure. The system may comprise one or more computer processors and one or more non-transitory computer- readable storage media storing instructions that are operable, when executed by the one or more computer processors, to cause the one or more computer processors to perform operations comprising: receiving at least one image captured by an interventional imaging device; identifying one or more image features in the received at least one image using an image recognition algorithm; labeling the identified one or more image features, wherein the identified one or more image features comprising one or more of an anatomical structure, a surgical tool, a surgical tool element, an operational procedure or an action, or a pathology; and displaying the labeled one or more image features in the at least one image to an operator continuously in the course of the arthroscopic procedure. For example, in some embodiments, the operational procedure or the action may be identified partially based on identifying the surgical tool. In some embodiments, an interventional imaging device is an endoscope. In some embodiments, an interventional imaging device is an arthroscope. The operations may further comprise storing the image in a memory device. In some embodiments, the operations further comprise discarding the image after displaying the labeled image features, to optimize a memory usage.
[0099] In some embodiments, the arthroscopic procedure may be an arthroscopic surgery (or arthroscopy). An arthroscopic surgery, also known as arthroscopy or keyhole surgery can be a minimally invasive surgical procedure (e.g., it requires only a small incision) on a joint. The arthroscopy may comprise either a diagnostic or therapeutic procedure. Therapeutic aspects or types of arthroscopy may also comprise surgical repairs, such as, for example, debridement, or cleaning, of a joint to remove bits of tom cartilage, ligament reconstruction, or synovectomy (removal of the joint lining). The arthroscopy can be performed using an arthroscope. The arthroscope may be inserted into a subject’s body to perform the arthroscopic procedure in or around the joint through a small incision. The arthroscope may be an endoscope. The arthroscope may comprise a fiberoptic scope. The arthroscope may be flexible or rigid.
The arthroscope may comprise a camera (e.g., a digital camera), a light source, a lens to create the image of the field of view, or a mechanism to carry the image to a sensor. The sensor may comprise a global shutter (e.g., a CCD sensor) or a rolling shutter (e.g., a CMOS sensor). The images captured by the arthroscope may be displayed on a display (e.g., a monitor). An operator (e.g., a surgeon) may use the displayed images (e.g., video feed from the arthroscope) to perform the surgery.
[00100] Arthroscopic surgeries may use small incisions through which tools and endoscopes may access a joint to diagnose or repair a pathology. Due to the minimally invasive nature of the procedure, patients may experience less pain, heal faster than conventional surgeries or experience less bleeding. However, arthroscopic surgeries can be more technically demanding than conventional surgeries (e.g., an open surgery). In arthroscopy, an operator (e.g., a surgeon) may operate with limited field of view, using small tools and with a restricted range of movement. Since the field of view may restricted, the surgeon may, for example, have to pan the scope to cover a target anatomy. Nonlimiting example of challenges associated with an arthroscopy may comprise tracking a target location, such as, for example, an intended repair location, a recognition of a pathology (e.g., cancerous tissue) , critical structures (e.g., nerves, arteries, veins, bone, cartilage, ligaments), etc.; visual orientation with respect to an anatomical structure; intraoperatively measuring of a dimension (e.g., sizing a rotator cuff tear); keeping track of bones or tissue boundaries for placing an implant when using a graft; a field of view that may not be sufficient for an operator to place an anchor in a predefined location, where the procedure may require a larger field of view; or correlating preoperative diagnostic imaging with intraoperative field of view to, for example recognize a critical sight. The critical sight may comprise a pathology (e.g., a tumor or a cyst), or a predefined implant or repair site. In some embodiments, the systems and methods provided herein may be configured for a shoulder surgery. In some embodiments, the systems and methods provided herein may be configured for a knee surgery.
[00101] In some embodiments, an image captured by imaging device (e.g., an arthroscope) may be received by a computer system. The computer system may comprise an image recognition or algorithm or another related algorithm. The image recognition algorithm may identify one or more features in the image received from the arthroscope. The image recognition algorithm may be configured to identify one or more of a region of operation or an approach portal of entry (or a portal) of the arthroscopic procedure (e.g., an arthroscopic surgery). In some embodiments, the region of operation is a shoulder. In some embodiments, the region of operation is a knee.
[00102] The image recognition algorithm may comprise a processing module. The image recognition algorithm may comprise an arrangement of processing modules. The arrangement of processing modules may be hierarchical. For example, a hierarchical arrangement of processing modules may comprise a first processing module that may be upstream of a second processing module and/or downstream of a third processing module. The image recognition algorithm may comprise at least two processing modules an upstream module and a downstream module. In some embodiments, the downstream module may be more specialized (e.g., configured to identify features associated with a particular anatomical structure,
pathology, tissue type, procedure, etc.) than the upstream module. In some embodiments, a module may be a generic module. A specialized module may comprise a portal recognition module, an anatomy recognition module, modules associated with specific anatomical structures (e.g., a shoulder module, a knee module), a pathology recognition module, modules associated with a specific pathology (e.g., cancer, defects in the cartilages, rotator cuff tears, dislocated / tom labrum, tom ACL, anterior cruciate ligament, tom meniscus, tom bicep tendon, inflamed synovial tissue, or femoral acetabular impingement (FAI)) or a pathology measurement module. A generic module may comprise a tool recognition module, an action recognition module (e.g., drilling, grinding, cleaning, etc.). In some embodiments, the processing module may comprise a plurality of upstream modules, a plurality of downstream modules, or a combination of both. In some embodiments, at least one of the plurality of upstream modules may be configured to select at least one downstream module from the plurality of downstream modules of the processing modules to further process an image. In some embodiments, identifying one or more features identified in the image may further comprise selecting one or more processing modules from the plurality of AI modules, wherein the selection is at least partially based on an output from at least one upstream module. In some embodiments, a processing module is selected based at least on the identifying the one or more of the region of operation or the approach portal of entry (or portal).
[00103] In some embodiments, an upstream module may be configured to identify one or more of an anatomy being operated upon by an operational procedure or an action being performed during the arthroscopic procedure. In some embodiments, a downstream module may be configured to recognize one or more of an anatomical feature of the identified anatomy associated with the operational procedure or the action being performed. In some embodiments, a downstream module may be configured to recognize a treatment tool feature associated with the operational procedure or the action being performed.
[00104] The processing modules may comprise an Artificial Intelligence (AI) module. In some embodiments, the processing modules may comprise a plurality of AI modules. In some embodiments, the processing modules or the plurality of AI modules may comprise at least a machine learning algorithm, a deep learning algorithm, or a combination of both. The machine learning algorithm may comprise a trained machine learning algorithm. The machine learning algorithm may comprise an artificial neural network. In some embodiments, the processing modules or an AI in the processing modules may comprise at least one dataset. In some embodiments, the processing modules or an AI in the processing modules may comprise at least a training dataset.
[00105] FIG. 1 is a flow chart that shows an example of a hierarchical arrangement 100 of modules. In some embodiments, a first module may comprise a region recognition module 110 to determine a field of surgery. A region recognition module may receive an image 101 (e.g., a frame from a video feed) from an imaging device (e.g., an arthroscope). In some embodiments, the region recognition module 110 may not recognize a field of surgery in the image. The region recognition module 110 may then stop transfer of the image to other modules and/or discard the frame. In some embodiments, the region recognition module 110 may recognize a field of surgery in the image. The region recognition module 110 may then send the
image to other modules downstream of the region recognition module 110 comprising a generic module, a specialized module, or both. For example, modules downstream of region recognition module 110 may comprise a portal recognition module 111, a tool recognition module 120, or both. The image may be sent to two or more modules substantially simultaneously (e.g., in parallel). The image may be sent to two or more modules consecutively (e.g., in series). The image may be sent to a tool recognition module 120, an action recognition module 121, or a procedure recognition module 122 in order mentioned herein. In some embodiments, the tool recognition module 120 may recognize a tool in the image. Subsequently, the module 121 may determine an action based at least in part on the recognized tool. The module 121 may then send the image to a procedure recognition module, which then determines a surgical procedure being performed based at least in part on the determined action and/or the recognized tool. The region recognition module 110 may also send the image to the portal recognition module 111 in parallel to the tool recognition module 120. The portal recognition module 111 may then determine a portal of entry associated with the region recognized by region recognition module 110. Based at least in part on the portal of entry recognized by portal recognition module 111, an image may be sent to one or more specialized modules which form a specialized or customized pipeline. Such a specialized pipeline may be specialized to a particular anatomical region or site, (e.g., the shoulder, knee or hip) and may comprise specialized modules (e.g., AI trained for a specific anatomical structure or pathology). One or more modules in the specialized pipeline may be activated upon recognition of region of entry (e.g., a bursal region) by the portal recognition module 111.
[00106] In some embodiments, labeling an image feature further comprises: a pixel-level masked labeling, a bounding box labeling, a frame-level labeling, or a temporal labeling. In some embodiments, the pixel-level masked labeling is used to display the labeled anatomical structure or the surgical tool. In some embodiments, the bounding box labeling is used to display the labeled pathology, the surgical tool, or a foreign object. In some embodiments, the frame-level labeling is used to display a labeled anatomical region. In some embodiments, temporal labeling is used to display the labeling of the operational procedure or the recognized action.
[00107] Downstream of region recognition module 110 and portal recognition module 111, one or more images may be sent to one or more modules to recognize and/or label one or more features on the image(s). Examples of labeled images are shown in FIGs.7A-7B. An anatomy recognition module 130 shown in FIG. 1 may receive one or more images from the region recognition module 110 and portal recognition module 111 and may recognize and label anatomical structures (as shown in Fig. 7A). Different color masks 701a, 701b, 701c, 70 Id, along with descriptive labels 702, 703 may be used to differentiate the recognized anatomical structures. Downstream of the anatomy recognition module 130, a pathology recognition module 131 shown in FIG. 1 may receive the image from the anatomy recognition module 131 and may recognize and label one or more pathologies on the recognized anatomical structure (as shown in Fig. 7A). As described above in various embodiment, region recognition module 130, anatomy recognition module and any subsequent recognition or measurement module are specialized to a
particular anatomical site (e.g., the shoulder or knee) be virtue of information identified or otherwise obtained by portal recognition module 111 or a similar module. Because of this specialization, less memory, programming and processing capability/r are required of a computer system performing analysis and processing of images received from an imaging device used during the surgery. These reduced performance requirements (e.g., reduced memory, processing power) allows for various image processing functions such as one or more of anatomical, pathology recognition and associated be done by device in or near operating room (aka an edge device), rather than having to be done remotely e.g. via the Cloud which further speeds up the process and makes it more reliable since no data need be sent up and back over the Internet to the Cloud or an external computer device. Moreover, in various embodiments no Internet or other network connection is required at all. In this way, one or more of the reliability, speed and cybersecurity of embodiments of the invention are substantially improved.
[00108] FIG. 7B shows an example of a pathology which has been identified and labeled using a colored mask 70 Id as well as a descriptive label 705. For example, a tear has been detected and labeled in the labrum (FIG. 7B). Downstream of the pathology recognition module 131, a pathology measurement module 132 shown in FIG. 1 may receive the image from the pathology recognition module 131 and may provide one or more measurements for the pathology identified and labeled.
[00109] FIG. 2A shows an image of an arthroscopic surgical procedure. The image was then provided as an input to the system described herein. The image was processed using an image recognition model configured to analyze shoulder surgery images. The image recognition model may comprise specialized modules. For example, the specialized module may be used to analyze intraarticular region (FIG. 2A-2B). FIG. 2B shows an example of labeling features in the image provided in FIG. 2A using the methods and systems described herein. Features 210 (bicep tendon (Bicep_Tndn)), 220 (Labrum), 230 (Glenoid), 240 (coracoid process(Cora-pro)), 250 (middle glenohumeral ligament (MGH-Liga)), and 260 (subscapularis (Subscap)) may be provided labels, such as with color masks and a legend LG (which also can provide color or pattern guide for other anatomical features that may be labeled and/or masked, such as the humeral head (Humrl Hd), supra articular process (Supra artclr), descending portion of coracoid (Cora- dee), inner glenohumeral ligament (IGH-Liga), to name a few) as shown in FIG. 2B.
[00110] FIG. 8A-8V show examples of identifying various features in a surgical procedure using the systems and methods described herein. An image from the surgical field of view (e.g., a frame of a video stream) may be provided (FIG. 8A). The image recognition model may produce predictions for different classes of anatomical structures (e.g., humeral head, glenoid, subscapularis, bicep tendon, ligaments, tendons, etc.) that can be recognized in the input image (FIGs. 8M (humeral head), 8N (bicep tendon), 80 (supra articular surface), 8P (subscapularis), 8Q (labrum), 8R (middle glenohumeral ligament), 8S (glenoid), 8T (descending portion of coracoid), 8U (coracoid process), 8V (inner glenohumeral ligament)). The predictions made by the recognition model were compared to a series of labels generated for the same classes of anatomical structures made by subject matter experts (FIGs. 8C (humeral head), 8D (bicep tendon), 8E (supra articular surface), 8F (subscapularis), 8G (labrum), 8H (middle
glenohumeral ligament), 81 (glenoid), 8J (descending portion of coracoid), 8K (coracoid process), 8L (inner glenohumeral ligament)). After predicting distinct classes of anatomical structures, the predictions are combined into an output image provided with a prediction mask (FIG. 8B). The output image may contain different color masks to differentiate different classes of anatomical structures identified. The classification process described herein may be applied to consecutive frames received from a video camera. The labeled output image may then be overlaid on the video stream. The overlay may be performed in real-time or in substantially close to real-time.
[00111] FIG. 3A shows an example of labeling a tool, an action, or a procedure in an image captured by an arthroscope. The image recognition algorithm recognized and labeled a tool 310 (e.g., a radiofrequency ablation tool). Based on the recognized tool 310, a tool identify label 320 may be provided for the type of tool identified (e.g., a radiofrequency ablation tool), an action recognition module may identify the action being performed was and provide an activity label 330 (e.g., showing a cleaning being performed). A procedure recognition module may determine the procedure performed and provide a procedural label 340, for example showing a coronoid decompression procedure being performed.
[00112] FIG. 3B shows another example of labeling a tool, an action, or a procedure in an image captured by an arthroscope. The image recognition algorithm recognized and labeled an arthroscopic burrs tool at label 350. A tool may comprise a surgical probe, a shaver, a burr grinder, a drill tool, an implant, a drill guides, a radio frequency or other ablation tool, an anchor, a graspers, suture passers, or a scissors. Based on the recognized tool, grinding was determined to be the action being performed at a label 360 using an action recognition module. A procedure recognition module determined the procedure to be a coronoid decompression procedure at a label 370. An arthroscopic surgery may comprise actions being performed on areas comprising a shoulder, a knee or a hip. A shoulder surgery may be performed in an intraarticular region and/or the bursal region. Anatomical structures that may be recognized in the intraarticular region may comprise humeral head, labrum, glenoid, supraspinatus, bicep tendon, glenoid humeral ligaments, subscapularis. Anatomical structures that may be recognized in the bursal region may comprise acromian, clavicle, humeral head, bursal, or rotator cuff. A surgical procedure may be initiated from one or more portals of entry into the region of operation (e.g., intraarticular region, bursal region, a portal of entry adjacent to a knee). A view of the surgical field of view may therefore vary based at least on the portal or an angle of entry used in an operation. FIG. 3C shows an example of recognizing an arbitrary tool 380 in a field of surgery.
[00113] In some embodiments, the operations further comprise providing a suggested action to an operator (e.g., a surgeon). The operator may be performing the procedure (e.g., arthroscopic surgery). The operator may be a person other than the surgeon. The operator may operate the imaging device (e.g., an arthroscope). In some embodiments, the suggested action is to assist an operator in the course of the surgery (e.g., an arthroscope). For example, a surgeon may require a measurement of a tissue or pathology, and the methods and systems described herein may provide the measurement to the surgeon intraoperatively. In some embodiments, the suggested action may comprise a safety warning to allow the
operator to know of a potential safety issue. For example, the system may recognize a distance of one or more implants being implanted in a patient to be different from a predefined safe distance. The distance may be a distance between two or more implants from one another. In some embodiments, the distance may be a distance of one or more implants from an anatomical structure, an anatomical feature, or a pathology. A suggested action may comprise avoiding a critical anatomical feature such as, for example, a vein, an artery, nerves, bone, cartilage, or ligaments. In some embodiments, the suggested action comprising a safety warning is based at least on an identified anatomical feature, a recognized tool, an identified action or a combination thereof. For example, a tool may be recognized (e.g., a burr grinder) that may potentially damage a tissue (e.g., a cartilage). When the system recognizes a tool, for example, a burr grinder in close proximity of a tissue recognized, for example, as cartilage, the system may generate a safety warning. A suggested action may comprise suggesting an approach angle for a device. The approach angle may be an approach angle of a drilling tool. In some embodiments, the suggested action is provided for educational purposes. For example, a video stream of a medical procedure (e.g., arthroscopy) can be used for educational purposes. The methods and systems described herein may be used to label features overlaid on the video stream (e.g., images or frames of the video stream). The suggested actions may also be overlaid on the video stream for educational purposes.
[00114] In some embodiments, the suggested action is at least partially based on the labeling (e.g., labeling of the features) of the image. The suggested action may be at least in part based on an output from one of the plurality of modules (e.g., upstream modules, or downstream modules). In some embodiments, the suggested action is at least partially based on the output from at least one of the modules of the plurality of upstream modules. In some embodiments, the suggested action is at least partially based on the output from at least one of the modules of the plurality of downstream modules. [00115] In some embodiments, the image may be generated from a surgical video stream. In some embodiments, the surgical video stream is an arthroscopic surgery video stream. In some embodiments, the surgical video stream is monocular or stereoscopic. In some embodiments, the labeling of features on an image may be performed at a speed similar to a rate of acquiring images from an imaging device. The arthroscope may generate consecutive images (e.g., a video feed) at a rate of at least about 10 frames per second (fps).
[00116] In some embodiments, the image recognition algorithm is trained using a database (e.g., a training dataset). In some embodiments, the database may comprise a plurality of training images. The plurality of training images may comprise one or more surgical procedures, surgical tools, surgical tool elements, anatomical structures, or pathologies. In some embodiments, a training dataset may be generated using an image editing technique. An image editing technique may comprise augmenting an image of a region, a portal, an anatomical structure or an anatomical feature with an image of a surgical tool. The augmented image may then be used to train the image recognition algorithm to recognize the tool within a context of the region, the portal, the anatomical structure or the anatomical feature. In some embodiments, the image editing, or augmentation techniques may comprise rotating a training image to
improve a robustness of the image recognition algorithm against a position or an orientation of a patient, for example, during the arthroscopic surgery. In some embodiments, the image editing, or augmentation techniques may comprise enlarging or cropping the training images to improve the robustness against changes in a depth of view. In some embodiments, the image editing, or augmentation techniques may comprise flipping a training image along a vertical axis to improve the robustness against procedures performed on a right or a left side of a patient. The surgical images used for such training including image editing and augmented training may be selected from a variety of procedures including one or more minimally invasive procedures such as arthroscopic, endoscopic, laparoscopic and cardioscopic procedures which may correspond to one or more of arthroscopic, bariatric, cardiovascular, intestinal, gynecological, urological surgeries or related procedures.
[00117] FIG. 4 is a flowchart of an example of training an image recognition algorithm. An AI training method 400 may comprise a dataset 410. The dataset 410 may comprise images of a surgical tool, an anatomical structure, an anatomical feature, a surgical tool element, an image acquired from a video feed of an arthroscope, a portal of a surgery, a region of a surgery, etc. The dataset may further comprise an imaged that has been edited or augmented using the methods described hereinbefore. The images in the dataset 410 may be separated into at least a test dataset 420 and a training dataset 430. The dataset 410 may be divided into a plurality of test datasets and/or a plurality of training datasets. At a model training step 440, a training dataset may be used to train an image recognition algorithm. For example, a plurality of labeled images may be provided to the image recognition algorithm to train an image recognition algorithm comprising a supervised learning algorithm (e.g., a supervised machine learning algorithm, or a supervised deep learning algorithm). Unlabeled images may be used to build and train an image recognition algorithm comprising an unsupervised learning algorithm (e.g., an unsupervised machine learning algorithm, or an unsupervised deep learning algorithm). A trained model may be tested using a test dataset (or a validation dataset). A test dataset may comprise unlabeled images (e.g., labeled images where a label is removed for testing a trained model). The trained image recognition algorithm may be applied to the test dataset and the predictions may be compared with actual labels associated with the data (e.g., images) that were removed to generate the test dataset in a testing model predictions step 460. A model training step 440 and a testing model predictions step 460 may be repeated with different training datasets and/or test datasets until a predefined outcome is met. The predefined outcome may be an error rate. The error rate may be defined as one more of an accuracy, a specificity, or a sensitivity or a combination thereof. The tested model 450 may then be used to make a prediction 470 for labeling features in an image from an imaging device (e.g., an arthroscope) being used in the course of a medical procedure (e.g., arthroscopy). The prediction may comprise a plurality of predictions 480 comprising a region of a surgery, a portal of the surgery, an anatomy, a pathology, a tool, an action being performed, a procedure being performed, etc.
[00118] An example of an AI pipeline, as described herein, used in a simulated knee surgery as described herein is provided in FIG. 6. In this ad related examples, a video feed of the surgery may be
provided on a first screen 601. A second screen may provide an AI inference of the surgical field 602 comprising anatomical masks as described hereinbefore. The AI feed may be overlaid on the video feed of the surgery in real-time. The AI feed and the video feed of the surgery may be provided on one screen simultaneously.
[00119] Another aspect of the invention provides a computer-implemented method for guiding an arthroscopic procedure. In some embodiments, the computer-implemented method may comprise receiving at least one image captured by an interventional imaging device; identifying one or more image features in the received at least one image using an image recognition algorithm; labeling the identified one or more image features, wherein the identified one or more image features; and displaying the labeled one or more image features in the at least one image to an operator continuously in the course of the arthroscopic procedure. The labeled one or more image features may be displayed in real time or concurrent to the arthroscopic procedure. The identified one or more image features may comprise one or more of an anatomical structure, a surgical tool, a surgical tool element, an operational procedure or action, or a pathology.
[00120] Another aspect of the invention provides a method of training an algorithm for guiding an arthroscopic procedure. The method of training an algorithm may comprise receiving a set of image features based on one or more images relating to the arthroscopic procedure; receiving a training dataset, wherein the training dataset comprises one or more labeled images relating to the arthroscopic procedure; recognizing one or more of the image features in images of the training dataset; building an image recognition algorithm based at least partially on the recognition of the one or more image features and the received training dataset. The one or more image features may relate to relate to visual properties of one or more of an anatomical structure, a surgical tool, a surgical tool element, an operational procedure or action, or a pathology. The image recognition algorithm may be configured to identify and label the one or more image features in an unlabeled image relating to the arthroscopic procedure.
[00121] The image recognition algorithm may be trained using training data. Training data may comprise images, where subject matter experts trace outlines of various anatomical structures, pathologies, tools, etc. The process of training is similar to the example shown in FIG.8A-8V, as described elsewhere herein. These subject matter experts labeled images may be also used to train the algorithm to, for example, recognize a given tool performing a given action. A combination of tool recognition, anatomical recognition, and an action recognition may be used to predict a surgical procedure being performed. For example, first an anatomical feature or region (e.g., bursa) may be recognized, which may then trigger other recognition modules to recognize a tool or an action being performed. FIG. 3B shows an example, where a region of operation is recognized as being bursa, then a burr tool identified with label 350 and a grinding action identified with label 360 may be recognized. The combination of these predictions may lead to recognizing the procedure as the coracoid decompression identified with label 370.
[00122] In some embodiments, a tool may not be present in the image. The system may still recognize an activity at least partially based on anatomical features being recognized, a previous image analyzed, or a combination thereof. The previous image processed by the system may have comprised recognizing a tool, an action, a surgical procedure, or combinations thereof. FIG. 9 shows an example of an action (e.g., clearing) identified with a label 920 being recognized by the system, while a tool is not present in the image as shown by a label 910 (showing NA or non-applicability). A surgical procedure identified with a label 930 may not be assigned to an image (FIG. 9). For example, the operator may swap tools during the course of a surgical procedure. In some embodiments, the swapping of the tools may take several seconds or minutes. The system and method described herein, can retain a memory of the recognized tool. In some other cases, the tool may be obscured or not recognizable (e.g., by movement of camera, orientation of the tool may render the tool unrecognizable). To compensate for the short term loss of tool recognition, an AI architecture may be used. The AI architecture may comprise a neural network architecture. The neural network architecture may comprise a long short term memory (LSTM).
[00123] FIG. 10 shows an example flowchart of the process of identifying a surgical procedure, as described herein. Image frames with annotations 1001 may be received and segmented into one or more segments using one or more classifier models. The classifier models may comprise a tool recognition model 1002, an anatomy detection model 1003, an activity detection model 1004, or a feature learning model 1005. The outputs from the one or more classifiers may be combined using a long short term memory (LSTM) 1006. LSTM is an artificial recurrent neural network (RNN) classifier that may be used to predict based on image recognition at one moment compared to what has been recognized prior. In other words, LSTM may be used to generate a memory of a context of the images being processed, as described herein. The context of the images is then used to predict a stage of the surgery comprising a surgical procedure. A rule-based decision to combine the classified segments into one image may then be processed to identify/predict a surgical procedure 1007 being performed.
[00124] Another aspect of the invention provides a system for implementing a hierarchical pipeline for guiding an arthroscopic surgery. The system may comprise one or more computer processors and one or more non-transitory computer-readable storage media storing instructions that are operable, when executed by the one or more computer processors, to cause the one or more computer processors to perform operations. The operations may comprise (a) receiving at least one image captured by an interventional imaging device; (b) identify one or more image features of a region of treatment or a portal of entry in the region based on at least one upstream module; (c) ) activating a first downstream module to identify one or more image features of an anatomical structure, or a pathology based at least partially on the identified one or more image features in step (b); (d) activating a second downstream module to identify one or more image features of a surgical tool, a surgical tool element, an operational procedure or action relating to the arthroscopic surgery based at least partially on the identified one or more image features in step (b); (e) labeling the identified one or more image features; and displaying the labeled one or more image features in the at least one image continuously to an operator in the course of the
arthroscopic surgery. The at least one upstream module may comprise a first trained image processing algorithm. The downstream module may comprise a second trained image processing algorithm. The second downstream module may comprise a third trained image processing algorithm.
[00125] FIG. 11A shows another example of the process of identifying a surgical procedure, as described herein. Image frames with annotations 1101 may be received and segmented into one or more segments using one or more classifier models. The classifier models may comprise a tool recognition model 1102, an anatomy detection model 1103, and/or an activity detection model 1104. The outputs from the one or more classifiers may be processed and combined using a rule-based decision algorithm 1106. A predicted surgical procedure 1107 may be provided as a label overlaid on the surgical video stream. [00126] FIG. 1 IB shows an example of an output from each of the components across time (e.g., duration of a procedure), described herein. The output shown in FIG. 1 IB may comprise a movement of the recognized tool, anatomy, and/or a change in the activity. The movement of a tool may be due to the movement of the camera or a change on the procedure, where a tool is swapped. The movement in the anatomy may be due the structural changes in the anatomy due, for example, to the procedure or due to the movement of the camera. In some embodiments, the operator may pause or change an activity during the procedure and resume the action after the pause. An example of an output from the tool recognition model 1102 (e.g., a movement of a recognized tool) is shown in graph 1112; an example of an output from the anatomy detection model 1103 (e.g., a movement of a recognized anatomy) is shown in graph 1113; and an example of an output from the activity detection model 1104 (e.g., a pause or change in the activity or the surgical procedure) is shown in graph 1114. To generate a seamless labeling of the predicted procedure, the outputs from the tool recognition model 1102, the anatomy detection model 1103, and/or the activity detection model 1104 are combined (e.g., summed) as shown in graphl 115. In some embodiments, the sum of the outputs is averaged over time to generate an averaged sum as shown in graph 1116. In some embodiments, the average sum is processed using a smoothing algorithm to generate a smooth average sum. The smoothing algorithm may comprise a gaussian smoothing function. The smoothing algorithm may comprise a convolutional neural network (e.g., convolutional neural network) with its activity shown in graph 1117.
[00127] FIG. 12 shows a schematic diagram of exemplary elements in AI assisted surgery methods and systems, described herein. The systems and methods may comprise elements comprising model development 1201, model repository 1202, and providing (e.g., visualizing) a label or mask 1203. Developing AI/machine learning models or algorithms used to process a surgical image may comprise training and/or validation steps 1210. Test data 1211 may be used to train and validate AI/machine learning models. A feature identified/recognized in an image obtained from a surgery (e.g., a frame of a surgical video) may be visualized (e.g., a mask, or a label) overlaid 1203 on a video or image of the surgical field of view.
Computer systems
[00128] Various embodiments of the invention also provide computer systems that are programmed to implement methods of the invention. Accordingly, a description of one or more embodiments of such computer systems will now be described. FIG. 5 shows a computer system 501 that is programmed or otherwise configured to perform one or more functions or operations of methods of the present invention. The computer system 501 can regulate various aspects of the present invention, such as, for example, of receiving an image from an interventional imaging device, identifying features in the image using an image recognition algorithm, overlaying the features on a video feed on a display device, make recommendations or suggestion to an operator based on the identified features in the image. The computer system 501 can be an electronic device of a user or a computer system that is remotely located with respect to the electronic device. The electronic device can be a mobile electronic device.
[00129] The computer system 501 includes a central processing unit (CPU, also “processor” and “computer processor” herein) 505, which can be a single core or multi core processor, or a plurality of processors for parallel processing. The computer system 501 also includes memory or memory location 510 (e.g., random-access memory, read-only memory, flash memory), electronic storage unit 515 (e.g., hard disk), communication interface 520 (e.g., network adapter) for communicating with one or more other systems, and peripheral devices 525, such as cache, other memory, data storage and/or electronic display adapters. The memory 510, storage unit 515, interface 520 and peripheral devices 525 are in communication with the CPU 505 through a communication bus (solid lines), such as a motherboard. The storage unit 515 can be a data storage unit (or data repository) for storing data. The computer system 501 can be operatively coupled to a computer network (“network”) 530 with the aid of the communication interface 520. The network 530 can be the Internet, an internet and/or extranet, or an intranet and/or extranet that is in communication with the Internet. The network 530 in some embodiments is a telecommunication and/or data network. The network 530 can include one or more computer servers, which can enable distributed computing, such as cloud computing. The network 530, in some embodiments with the aid of the computer system 501, can implement a peer-to-peer network, which may enable devices coupled to the computer system 501 to behave as a client or a server.
[00130] The CPU 505 can execute a sequence of machine -readable instructions, which can be embodied in a program or software. The instructions may be stored in a memory location, such as the memory 510. The instructions can be directed to the CPU 505, which can subsequently program or otherwise configure the CPU 505 to implement methods of the present invention. Examples of operations performed by the CPU 505 can include fetch, decode, execute, and writeback.
[00131] The CPU 505 can be part of a circuit, such as an integrated circuit. One or more other components of the system 501 can be included in the circuit. In some embodiments, the circuit is an application specific integrated circuit (ASIC).
[00132] The storage unit 515 can store files, such as drivers, libraries and saved programs. The storage unit 515 can store user data, e.g., user preferences and user programs. The computer system 501
in some embodiments can include one or more additional data storage units that are external to the computer system 501, such as located on a remote server that is in communication with the computer system 501 through an intranet or the Internet.
[00133] The computer system 501 can communicate with one or more remote computer systems through the network 530. For instance, the computer system 501 can communicate with a remote computer system of a user (e.g., a portable computer, a tablet, a smart display device, a smart tv, etc.). Examples of remote computer systems include personal computers (e.g., portable PC), slate or tablet PC’s (e.g., Apple® iPad, Samsung® Galaxy Tab), telephones, Smart phones (e.g., Apple® iPhone, Android- enabled device, Blackberry®), or personal digital assistants. The user can access the computer system 501 via the network 530.
[00134] Methods as described herein can be implemented by way of machine (e.g., computer processor) executable code stored on an electronic storage location of the computer system 501, such as, for example, on the memory 510 or electronic storage unit 515. In one or more embodiments, the machine executable or machine-readable code can be provided in the form of software. During use, the code can be executed by the processor 505. In some embodiments, the code can be retrieved from the storage unit 515 and stored on the memory 510 for ready access by the processor 505. In some situations, the electronic storage unit 515 can be precluded, and machine-executable instructions are stored on memory 510.
[00135] The code can be pre-compiled and configured for use with a machine having a processer adapted to execute the code or can be compiled during runtime. The code can be supplied in a programming language that can be selected to enable the code to execute in a pre-compiled or as- compiled fashion.
[00136] Aspects of the systems and methods provided herein, such as the computer system 501, can be embodied in programming. Various aspects of the technology may be thought of as “products” or “articles of manufacture” typically in the form of machine (or processor) executable code and/or associated data that is carried on or embodied in a type of machine readable medium. Machine- executable code can be stored on an electronic storage unit, such as memory (e.g., read-only memory, random-access memory, flash memory) or a hard disk. “Storage” type media can include any or all of the tangible memory of the computers, processors or the like, or associated modules thereof, such as various semiconductor memories, tape drives, disk drives and the like, which may provide non-transitory storage at any time for the software programming. All or portions of the software may at times be communicated through the Internet or various other telecommunication networks. Such communications, for example, may enable loading of the software from one computer or processor into another, for example, from a management server or host computer into the computer platform of an application server. Thus, another type of media that may bear the software elements includes optical, electrical and electromagnetic waves, such as used across physical interfaces between local devices, through wired and optical landline networks and over various air-links. The physical elements that carry such waves, such as wired or
wireless links, optical links or the like, also may be considered as media bearing the software. As used herein, unless restricted to non-transitory, tangible “storage” media, terms such as computer or machine “readable medium” refer to any medium that participates in providing instructions to a processor for execution.
[00137] Hence, a machine readable medium, such as computer-executable code, may take many forms, including but not limited to, a tangible storage medium, a carrier wave medium or physical transmission medium. Non-volatile storage media include, for example, optical or magnetic disks, such as any of the storage devices in any computer(s) or the like, such as may be used to implement the databases, etc. shown in the drawings. Volatile storage media include dynamic memory, such as main memory of such a computer platform. Tangible transmission media include coaxial cables; copper wire and fiber optics, including the wires that comprise a bus within a computer system. Carrier-wave transmission media may take the form of electric or electromagnetic signals, or acoustic or light waves such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer-readable media therefore include for example: a floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD or DVD-ROM, any other optical medium, punch cards paper tape, any other physical storage medium with patterns of holes, a RAM, a ROM, a PROM and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave transporting data or instructions, cables or links transporting such a carrier wave, or any other medium from which a computer may read programming code and/or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution. [00138] [00136] The computer system 501 can include or be in communication with an electronic display 535 that comprises a user interface (UI) 540 for providing, for example, an overlay of the identified features on a video feed from an arthroscope or to provide a recommendation to an operator in the course of a surgery. Examples of UFs include, without limitation, a graphical user interface (GUI) and web-based user interface.
[00139] Methods and systems of the present invention can be implemented by way of one or more algorithms. An algorithm can be implemented by way of software upon execution by the central processing unit 505. The algorithm can, for example, receiving an image from an interventional imaging device, identifying a feature in the image using an image recognition algorithm, overlaying the features on a video feed on a display device, make recommendations or suggestion to an operator based on the identified feature in the image.
[00140] While preferred embodiments of the present invention have been shown and described herein, it will be obvious to those skilled in the art that such embodiments are provided by way of example only.
It is not intended that the invention be limited by the specific examples provided within the specification. While the invention has been described with reference to the aforementioned specification, the descriptions and illustrations of the embodiments herein are not meant to be construed in a limiting sense. Numerous variations, changes, and substitutions will now occur to those skilled in the art without
departing from the invention. Furthermore, it shall be understood that all aspects of the invention are not limited to the specific depictions, configurations or relative proportions set forth herein which depend upon a variety of conditions and variables. It should be further understood that various alternatives to the embodiments of the invention described herein may be employed in practicing the invention.
Accordingly, it should be understood that the invention covers various alternatives, modifications, variations or equivalents to the embodiments of the invention described herein.
[00141] Also, elements, characteristics, or acts from one embodiment can be readily recombined or substituted with one or more elements, characteristics or acts from other embodiments to form numerous additional embodiments within the scope of the invention. Moreover, elements that are shown or described as being combined with other elements, can, in various embodiments, exist as standalone elements. Further, embodiments of the invention specifically contemplate the exclusion of an element, act, or characteristic, etc. when that element, act or characteristic is positively recited. Hence, the scope of the present invention is not limited to the specifics of the described embodiments but is instead limited solely by the appended claims.
Claims
1. A system for guiding an arthroscopic procedure, the system comprising one or more computer processors and one or more non-transitory computer-readable storage media storing instructions that are operable, when executed by said one or more computer processors, to cause said one or more computer processors to perform operations comprising: receiving at least one image captured by an interventional imaging device; identifying one or more image features in said received at least one image using an image recognition algorithm; labeling said identified one or more image features, wherein said identified one or more image features comprising one or more of an anatomical structure, a surgical tool, a surgical tool element, an operational procedure or an action, or a pathology; and displaying said labeled one or more image features in said at least one image to an operator continuously in the course of said arthroscopic procedure.
2. The system of claim 1, wherein said labeled one or more image features are displayed in real time or concurrent to said arthroscopic procedure.
3. The system of claim 1, wherein said arthroscopic procedure is an arthroscopic surgery.
4. The system of claim 1, wherein said image recognition algorithm comprises a hierarchical arrangement of processing modules.
5. The system of claim 4, wherein the processing modules comprise a plurality of Artificial Intelligence (AI) modules.
6. The system of claim 4, wherein said processing modules comprise at least a machine learning algorithm, a deep learning algorithm, or a combination of both.
7. The system of claim 6, wherein said machine learning algorithm comprises an artificial neural network.
8. The system of claim 4, wherein said processing modules comprise at least one dataset.
9. The system of claim 4, wherein said processing modules comprise at least one training dataset.
10. The system of claim 4, wherein said processing modules comprise an upstream module and a downstream module, wherein the downstream module being more specialized than the upstream module.
11. The system of claim 10, wherein the upstream module is configured to identify one or more of an anatomy being operated upon by an operational procedure or an action being performed during said arthroscopic procedure.
12. The system of claim 10, wherein the downstream module is configured to one or more of recognize an anatomical feature of the identified anatomy or recognize a treatment tool feature associated with said operational procedure or said action being performed.
13. The system of claim 10, wherein the processing modules comprise a plurality of upstream modules or a plurality of downstream modules.
14. The system of claim 13, wherein at least one of the plurality of upstream modules is configured to select an individual downstream module from the processing modules for use.
15. The system of claim 1, wherein said operational procedure or said action is identified partially based on said identifying said surgical tool.
16. The system of claim 1, wherein said interventional imaging device is an arthroscope.
17. The system of claim 1, wherein said interventional imaging device is an endoscope.
18. The system of claim 1, wherein said image recognition algorithm is configured to identify one or more of a region of operation or an approach portal of entry of said arthroscopic procedure.
19. The system of claim 18, wherein said region of operation is a shoulder.
20. The system of claim 18, wherein said region of operation is a knee.
21. The system of claim 4, wherein at least one module from said processing modules is selected based at least on said identifying said one or more of said region of operation or said approach portal of entry.
22. The system of claim 1, wherein said operations performed further comprise storing said at least one image in a memory device.
23. The system of claim 1, wherein said operations further comprise discarding said at least one image after said displaying said label elements to optimize memory usage.
24. The system of claim 1, wherein said labeling one or more image features further comprises: a pixel-level masked labeling, a bounding box labeling, a frame-level labeling, or a temporal labeling.
25. The system of claim 24, wherein said pixel-level masked labeling is used to display said labeled anatomical structure or said surgical tool.
26. The system of claim 24, wherein said bounding box labeling is used to display said labeled pathology, said surgical tool, or a foreign object.
27. The system of claim 24, wherein said frame-level labeling is used to display a labeled anatomical region.
28. The system of claim 24, wherein said temporal labeling is used to display said labeling said operational procedure or said action.
29. The system of claim 1, wherein said operations further comprise providing a suggested action.
30. The system of claim 29, wherein said suggested action is at least partially based on said labeling of said at least one image.
31. The system of claim 29, wherein said suggested action is at least partially based on said upstream module.
32. The system of claim 29, wherein said suggested action is at least partially based on said downstream module.
33. The system of claim 29, wherein said suggested action is to assist said operator in the course of said surgery.
34. The system of claim 29, wherein said suggested action is provided for educational purposes.
35. The system of claim 29, wherein said suggested action comprises providing a safety warning based at least on identification of a critical anatomical structure, said surgical tool, said action, or a distance of two or more implants.
36. The system of claim 35, wherein the distance of said two or more implants creates a health risk.
37. The system of claim 35, wherein the distance of said two or more implants is different from a predefined distance of said two or more implants.
38. The system of claim 29, wherein said suggested action comprises providing a suggested approach angle.
39. The system of claim 38, wherein said suggested approach angle may comprise a drilling angle.
40. The system of claim 1, wherein said system is configured for a shoulder surgery.
41. The system of claim 1, wherein said system is configured for a knee surgery.
42. The system of claim 1, wherein said image recognition algorithm is trained using a database.
43. The system of claim 42, wherein said database comprises a plurality of training images comprising one or more surgical procedures, surgical tools, surgical tool elements, anatomical structures, or pathologies.
44. The system of claim 42, wherein a plurality of augmentation techniques is used to improve said training dataset to improve a robustness of said image recognition algorithm.
45. The system of claim 44, wherein said augmentation techniques comprise rotating said training images to improve said robustness against a position or an orientation of a patient during said arthroscopic procedure.
46. The system of claim 44, wherein said augmentation techniques comprise flipping said training images along a vertical axis to improve said robustness against procedures performed on a right or a left side of a patient.
47. The system of claim 44, wherein said augmentation techniques comprise enlarging or cropping said training images to improve said robustness against changes in a depth of view.
48. The system of claim 1, wherein said at least one image is generated from a surgical video stream.
49. The system of claim 48, wherein said surgical video stream is an arthroscopic surgery video stream.
50. The system of claim 48, wherein said surgical video stream is monocular.
51. The system of claim 48, wherein said surgical video stream is stereoscopic.
52. A computer-implemented method for guiding an arthroscopic procedure, the method comprising: receiving at least one image captured by an interventional imaging device; identifying one or more image features in said received at least one image using an image recognition algorithm; labeling said identified one or more image features, wherein said identified one or more image features comprising one or more of an anatomical structure, a surgical tool, a surgical tool element, an operational procedure or action, or a pathology; and displaying said labeled one or more image features in said at least one image to an operator continuously in the course of said arthroscopic procedure, wherein said labeled one or more image features are displayed in real time or concurrent to said arthroscopic procedure.
53. The method of claim 52, wherein said arthroscopic procedure is an arthroscopic surgery.
54. The method of claim 52, wherein said image recognition algorithm comprises a hierarchical arrangement of processing modules.
55. The method of claim 54, wherein the processing modules comprise a plurality of Artificial Intelligence (AI) modules.
56. The method of claim 54, wherein said processing modules comprise at least a machine learning algorithm, a deep learning algorithm, or a combination of both.
57. The method of claim 56, wherein said machine learning algorithm comprises an artificial neural network.
58. The method of claim 54, wherein said processing modules comprise at least one dataset.
59. The method of claim 54, wherein said processing modules comprise at least one training dataset.
60. The method of claim 54, wherein said processing modules comprise an upstream module and a downstream module, wherein the downstream module being more specialized than the upstream module.
61. The method of claim 60, wherein the upstream module is configured to identify one or more of an anatomy being operated upon by an operational procedure or an action being performed during said arthroscopic procedure.
62. The method of claim 60, wherein the downstream module is configured to one or more of recognize an anatomical feature of the identified anatomy or recognize a treatment tool feature associated with said operational procedure or said action being performed.
63. The method of claim 60, wherein the processing modules comprise a plurality of upstream modules or a plurality of downstream modules.
64. The method of claim 63, wherein at least one of the plurality of upstream modules is configured to select an individual downstream module from the processing modules for use.
65. The method of claim 52, wherein said operational procedure or said action is identified partially based on said identifying said surgical tool.
66. The method of claim 52, wherein said interventional imaging device is an arthroscopel
67. The method of claim 52, wherein said interventional imaging device is an endoscope.
68. The method of claim 52, wherein said image recognition algorithm is configured to identify one or more of a region of operation or an approach portal of entry of said arthroscopic procedure.
69. The method of claim 68, wherein said region of operation is a shoulder.
70. The method of claim 68, wherein said region of operation is a knee.
71. The method of claim 54, wherein at least one module from said processing modules is selected based at least on said identifying said one or more of said region of operation or said approach portal of entry.
72. The method of claim 52, wherein said operations performed further comprise storing said at least one image in a memory device.
73. The method of claim 52, wherein said operations further comprise discarding said at least one image after said displaying said label elements to optimize memory usage.
74. The method of claim 52, wherein said labeling one or more image features further comprises: a pixel-level masked labeling, a bounding box labeling, a frame-level labeling, or a temporal labeling.
75. The method of claim 74, wherein said pixel-level masked labeling is used to display said labeled anatomical structure or said surgical tool.
76. The method of claim 74, wherein said bounding box labeling is used to display said labeled pathology, said surgical tool, or a foreign object.
77. The method of claim 74, wherein said frame-level labeling is used to display a labeled anatomical region.
78. The method of claim 74, wherein said temporal labeling is used to display said labeling said operational procedure or said action.
79. The method of claim 52, wherein said operations further comprise providing a suggested action.
80. The method of claim 79, wherein said suggested action is at least partially based on said labeling of said at least one image.
81. The method of claim 79, wherein said suggested action is at least partially based on said upstream module.
82. The method of claim 79, wherein said suggested action is at least partially based on said downstream module.
83. The method of claim 79, wherein said suggested action is to assist said operator in the course of said surgery.
84. The method of claim 79, wherein said suggested action is provided for educational purposes.
85. The method of claim 79, wherein said suggested action comprises providing a safety warning based at least on identification of a critical anatomical structure, or a distance of two or more implants.
86. The method of claim 85, wherein the distance of said two or more implants creates a health risk.
87. The method of claim 85, wherein the distance of said two or more implants is different from a predefined distance of said two or more implants.
88. The method of claim 79, wherein said suggested action comprises providing a suggested approach angle.
89. The method of claim 88, wherein said suggested approach angle may comprise a drilling angle.
90. The method of claim 52, wherein said method is configured for a shoulder surgery.
91. The method of claim 52, wherein said method is configured for a knee surgery.
92. The method of claim 52, wherein said image recognition algorithm is trained using a database.
93. The method of claim 92, wherein said database comprises a plurality of training images comprising one or more surgical procedures, surgical tools, surgical tool elements, anatomical structures, or pathologies.
94. The method of claim 92, wherein a plurality of augmentation techniques is used to improve said training dataset to improve a robustness of said image recognition algorithm.
95. The method of claim 94, wherein said augmentation techniques comprise rotating said training images to improve said robustness against a position or an orientation of a patient during said arthroscopic procedure.
96. The method of claim 94, wherein said augmentation techniques comprise flipping said training images along a vertical axis to improve said robustness against procedures performed on a right or a left side of a patient.
97. The method of claim 94, wherein said augmentation techniques comprise enlarging or cropping said training images to improve said robustness against changes in a depth of view.
98. The method of claim 52, wherein said at least one image is generated from a surgical video stream.
99. The method of claim 98, wherein said surgical video stream is an arthroscopic surgery video stream.
100. The method of claim 98, wherein said surgical video stream is monocular.
101. The method of claim 98, wherein said surgical video stream is stereoscopic.
102. A method of training an algorithm for guiding an arthroscopic procedure, the method comprising: receiving a set of image features based on one or more images relating to said arthroscopic procedure; receiving a training dataset, wherein said training dataset comprises one or more labeled images relating to said arthroscopic procedure; recognizing one or more of the image features in images of said training dataset, wherein said one or more image features relate to visual properties of one or more of an anatomical structure, a surgical tool, a surgical tool element, an operational procedure or action, or a pathology; and building an image recognition algorithm based at least partially on said recognition of the one or more image features and said received training dataset, wherein said image recognition algorithm is configured to identify and label said one or more image features in an unlabeled image relating to said arthroscopic procedure.
103. The method of claim 102, wherein said labeled one or more image features are displayed in real time or concurrent to said arthroscopic procedure.
104. The method of claim 102, wherein said image recognition algorithm comprises a hierarchical arrangement of processing modules.
105. The method of claim 104, where the processing modules comprise a plurality of individual image processing modules.
106. The method of claim 105, wherein said plurality of individual image processing modules comprise a first module for identifying said arthroscopic procedure in place, a second module for recognizing and labeling one or more surgical tools and surgical tool elements, a third module for recognizing and labeling one or more anatomical structures, or a combination thereof.
107. The method of claim 104, wherein the processing modules comprise a plurality of Artificial Intelligence (AI) modules.
108. The method of claim 104, wherein said processing modules comprise at least a machine learning algorithm, a deep learning algorithm, or a combination of both.
109. The method of claim 108, wherein said machine learning algorithm comprises an artificial neural network.
110. The method of claim 104, wherein said processing modules comprise at least one dataset.
111. The method of claim 104, wherein said processing modules comprise at least one training dataset.
112. The method of claim 104, wherein said processing modules comprise an upstream module and a downstream module, wherein the downstream module being more specialized than the upstream module.
113. The method of claim 112, wherein the upstream module is configured to identify one or more of an anatomy being operated upon by an operational procedure or an action being performed during said arthroscopic procedure.
114. The method of claim 112, wherein the downstream module is configured to one or more of recognize an anatomical feature of the identified anatomy or recognize a treatment tool feature associated with said operational procedure or said action being performed.
115. The method of claim 112, wherein the processing modules comprise a plurality of upstream modules or a plurality of downstream modules.
116. The method of claim 115, wherein at least one of the plurality of upstream modules is configured to select an individual downstream module from the processing modules for use.
117. The method of claim 102, wherein said identifying one or more image features in said at least one image further comprises selecting one or more processing modules from a plurality of processing modules, wherein said selection is at least partially based on said region and/or said portal of said arthroscopic surgery.
118. The method of claim 102, wherein said operational procedure or said action is identified partially based on said identifying said surgical tool.
119. The method of claim 102, wherein said unlabeled image is captured by an interventional imaging device.
120. The method of claim 119, wherein said interventional imaging device is an endoscope.
121. The method of claim 102, wherein said unlabeled image is generated from a surgical video stream.
122. The method of claim 121, wherein said surgical video stream is an arthroscopic surgery video stream wherein said endoscopic surgery is an arthroscopic surgery.
123. The method of claim 121, wherein said surgical video stream is monocular.
124. The method of claim 121, wherein said surgical video stream is stereoscopic.
125. The method of claim 102, wherein said image recognition algorithm is configured to identify one or more of a region of operation or an approach portal of entry of said arthroscopic procedure.
126. The method of claim 125, wherein said region of operation is a shoulder.
127. The method of claim 125, wherein said region of operation is a knee.
128. The method of claim 125, wherein at least one module from said processing modules is selected based at least on said identifying said one or more of said region of operation or said approach portal of entry.
129. The method of claim 102, wherein said image recognition algorithm stores said labeled image in a memory device.
130. The method of claim 102, wherein said image recognition algorithm discards said labeled image to minimize memory usage.
131. The method of claim 102, wherein said labeled image comprises: a pixel-level masked labeling, a bounding box labeling, a frame-level labeling, or a temporal labeling.
132. The method of claim 131, wherein said pixel-level masked labeling is used to display said labeled anatomical structure or said surgical tool.
133. The method of claim 131, wherein said bounding box labeling is used to display said labeled pathology, said surgical tool, or a foreign object.
134. The method of claim 131, wherein said frame-level labeling is used to display a labeled anatomical region.
135. The method of claim 131, wherein said temporal labeling is used to display said labeling said operational procedure or action.
136. The method of claim 102, wherein said training dataset is configured for a shoulder surgery.
137. The method of claim 136, wherein said image recognition algorithm is trained for a shoulder surgery using said training dataset configured for a shoulder surgery.
138. The method of claim 102, wherein said training dataset is configured for a knee surgery.
139. The method of claim 138, wherein said image recognition algorithm is trained for a knee surgery using said training dataset configured for a knee surgery.
140. The method of claim 102, wherein said training dataset comprises a plurality of training images comprising one or more surgical procedures, surgical tools, surgical tool elements, anatomical structures, or pathologies.
141. The method of claim 102, wherein a plurality of augmentation techniques is used to improve said training dataset to improve a robustness of said image recognition algorithm.
142. The method of claim 141, wherein said augmentation techniques comprise rotating said training images to improve said robustness against a position or an orientation of a patient during said arthroscopic procedure.
143. The method of claim 141, wherein said augmentation techniques comprise flipping said training images along a vertical axis to improve said robustness against procedures performed on a right or a left side of a patient.
144. The method of claim 141, wherein said augmentation techniques comprise enlarging or cropping said training images to improve said robustness against changes in a depth of view.
145. A method for implementing a hierarchical pipeline for guiding a minimally invasive procedure, the system comprising one or more computer processors and one or more non-transitory computer-readable storage media storing instructions that are operable, when executed by said one or more computer processors, to cause said one or more computer processors to perform operations comprising:
(a) receiving at least one image captured by an interventional imaging device;
(b) identify one or more image features of a region of treatment or a portal of entry in said region based on at least one upstream module, wherein said at least one upstream module comprises a first trained image processing algorithm;
(c) activating a first downstream module to identify one or more image features of an anatomical structure, or a pathology based at least partially on said identified one or more image features in step (b), wherein said first downstream module comprises a second trained image processing algorithm;
(d) activating a second downstream module to identify one or more image features of a surgical tool, a surgical tool element, an operational procedure or action relating to said minimally invasive procedure based at least partially on said identified one or more image features in step (b), wherein said second downstream module comprises a third trained image processing algorithm;
(e) labeling said identified one or more image features; and
(f) displaying said labeled one or more image features in said at least one image continuously to an operator in the course of said minimally invasive procedure.
146. The system of claim 145, wherein the minimally invasive procedure is an arthroscopic surgical procedure.
147. The system of claim 145, wherein said steps (c) and (d) are independent from one another.
148. The system of claim 145, wherein said first, second, or third trained image processing algorithms comprise at least a machine learning algorithm, a deep learning algorithm, or a combination of both.
149. The system of claim 148, wherein said machine learning algorithm includes an artificial neural network.
150. The system of claim 148, wherein said machine learning or said deep learning algorithms are trained using at least one training dataset.
151. The system of claim 150, wherein said training dataset is configured for a shoulder surgery.
152. The system of claim 150, wherein said training dataset is configured for a knee surgery.
153. The system of claim 150, wherein said training dataset comprises a plurality of training images comprising one or more surgical procedures, surgical tools, surgical tool elements, anatomical structures, or pathologies.
154. The system of claim 150, wherein a plurality of augmentation techniques is used to improve said training dataset to improve a robustness of said image recognition algorithm.
155. The system of claim 154, wherein said augmentation techniques comprise rotating said training images to improve said robustness against a position or an orientation of a patient during said minimally invasive procedure.
156. The system of claim 154, wherein said augmentation techniques comprise flipping said training images along a vertical axis to improve said robustness against procedures performed on a right or a left side of a patient.
157. The system of claim 154, wherein said augmentation techniques comprise enlarging or cropping said training images to improve said robustness against changes in a depth of view.
158. The system of claim 145, wherein said first, second, or third trained image processing algorithms store said displayed images with labeled features in a memory device.
159. The system of claim 145, wherein said first, second, or third trained image processing algorithms discard said displayed images with labeled features to minimize memory usage.
160. The system of claim 145, wherein said at least one image is generated from a surgical video stream.
161. The system of claim 160, wherein said surgical video stream is an endoscopic surgery video stream.
162. The system of claim 160, wherein said surgical video stream is monocular or stereoscopic.
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2022562500A JP2023523560A (en) | 2020-04-13 | 2021-04-13 | Systems and methods for AI-assisted surgery |
US17/918,873 US20230245753A1 (en) | 2020-04-13 | 2021-04-13 | Systems and methods for ai-assisted surgery |
EP21788903.9A EP4135566A4 (en) | 2020-04-13 | 2021-04-13 | Systems and methods for ai-assisted surgery |
Applications Claiming Priority (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
IN202041015990 | 2020-04-13 | ||
IN202041015990 | 2020-04-13 | ||
US202063030695P | 2020-05-27 | 2020-05-27 | |
US63/030,695 | 2020-05-27 | ||
US202163143367P | 2021-01-29 | 2021-01-29 | |
US63/143,367 | 2021-01-29 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2021211603A1 true WO2021211603A1 (en) | 2021-10-21 |
Family
ID=78084646
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2021/027109 WO2021211603A1 (en) | 2020-04-13 | 2021-04-13 | Systems and methods for ai-assisted surgery |
Country Status (4)
Country | Link |
---|---|
US (1) | US20230245753A1 (en) |
EP (1) | EP4135566A4 (en) |
JP (1) | JP2023523560A (en) |
WO (1) | WO2021211603A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11928834B2 (en) | 2021-05-24 | 2024-03-12 | Stryker Corporation | Systems and methods for generating three-dimensional measurements using endoscopic video data |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US12004966B2 (en) | 2021-04-12 | 2024-06-11 | Kaliber Labs Inc. | Systems and methods for using image analysis in superior capsule reconstruction |
US20230177082A1 (en) * | 2021-12-06 | 2023-06-08 | Genesis Medtech (USA) Inc. | Intelligent surgery video management and retrieval system |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140058406A1 (en) * | 2012-08-24 | 2014-02-27 | Nikolaos V. Tsekos | Robotic Device and Systems for Image-Guided and Robot-Assisted Surgery |
US20160378861A1 (en) * | 2012-09-28 | 2016-12-29 | Sri International | Real-time human-machine collaboration using big data driven augmented reality technologies |
US20180168740A1 (en) * | 2016-08-16 | 2018-06-21 | Insight Medical Systems, Inc. | Systems and methods for sensory augmentation in medical procedures |
-
2021
- 2021-04-13 US US17/918,873 patent/US20230245753A1/en active Pending
- 2021-04-13 WO PCT/US2021/027109 patent/WO2021211603A1/en unknown
- 2021-04-13 JP JP2022562500A patent/JP2023523560A/en active Pending
- 2021-04-13 EP EP21788903.9A patent/EP4135566A4/en active Pending
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140058406A1 (en) * | 2012-08-24 | 2014-02-27 | Nikolaos V. Tsekos | Robotic Device and Systems for Image-Guided and Robot-Assisted Surgery |
US20160378861A1 (en) * | 2012-09-28 | 2016-12-29 | Sri International | Real-time human-machine collaboration using big data driven augmented reality technologies |
US20180168740A1 (en) * | 2016-08-16 | 2018-06-21 | Insight Medical Systems, Inc. | Systems and methods for sensory augmentation in medical procedures |
Non-Patent Citations (1)
Title |
---|
See also references of EP4135566A4 * |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11928834B2 (en) | 2021-05-24 | 2024-03-12 | Stryker Corporation | Systems and methods for generating three-dimensional measurements using endoscopic video data |
Also Published As
Publication number | Publication date |
---|---|
US20230245753A1 (en) | 2023-08-03 |
EP4135566A4 (en) | 2024-05-15 |
JP2023523560A (en) | 2023-06-06 |
EP4135566A1 (en) | 2023-02-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20230245753A1 (en) | Systems and methods for ai-assisted surgery | |
Saeidi et al. | Autonomous robotic laparoscopic surgery for intestinal anastomosis | |
US10878966B2 (en) | System and method for analysis and presentation of surgical procedure videos | |
US20210307841A1 (en) | Methods, systems, and computer readable media for generating and providing artificial intelligence assisted surgical guidance | |
US20230200928A1 (en) | Methods for Autoregistration of Arthroscopic Video Images to Preoperative Models and Devices Thereof | |
US20230363831A1 (en) | Markerless navigation system | |
US11810356B2 (en) | Methods for arthroscopic video analysis and devices therefor | |
WO2023095492A1 (en) | Surgery assisting system, surgery assisting method, and surgery assisting program | |
US20230352133A1 (en) | Systems and methods for processing medical data | |
CN115942913A (en) | Systems and methods for visually guided bone removal during joint surgical procedures | |
EP4135567A1 (en) | Systems and methods for computer-assisted shape measurements in video | |
US20240104733A1 (en) | Systems and methods to process electronic medical images for diagnostic or interventional use | |
US20240203567A1 (en) | Systems and methods for ai-assisted medical image annotation | |
WO2023168306A2 (en) | Arthroscopic surgery assistance apparatus and method | |
EP4193954A1 (en) | Tactile solutions integration for patient specific treatment | |
Nishio et al. | Real-time Orthopedic surgery procedure recognition method with video images from smart glasses using convolutional neural Network | |
JP7148193B1 (en) | Surgery support system, surgery support method, and surgery support program | |
WO2024004013A1 (en) | Program, information processing method, and information processing device | |
US20240000511A1 (en) | Visually positioned surgery | |
US12051245B2 (en) | Methods for arthroscopic video analysis and devices therefor | |
EP4391906A1 (en) | Mri-based pipeline to evaluate risk of connective tissue reinjury | |
WO2024030683A2 (en) | System and methods for surgical collaboration | |
WO2023144570A1 (en) | Detecting and distinguishing critical structures in surgical procedures using machine learning |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21788903 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2022562500 Country of ref document: JP Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
ENP | Entry into the national phase |
Ref document number: 2021788903 Country of ref document: EP Effective date: 20221114 |