WO2023131878A1

WO2023131878A1 - Automatic guiding for human performance of a medical procedure

Info

Publication number: WO2023131878A1
Application number: PCT/IB2023/050061
Authority: WO
Inventors: Shlomo Ben-Haim
Original assignee: Libra Science Ltd.
Priority date: 2022-01-05
Filing date: 2023-01-04
Publication date: 2023-07-13

Abstract

Methods and systems for guiding a performance of an ongoing medical procedure of a certain kind are described. A processor accesses a structure of a medical procedure of the certain kind, wherein the structure includes events and identification criteria, so that each event is associated with a respective one of the identification criteria. The processor compares each of a plurality of portions of an ongoing recording of the ongoing medical procedure with identification criteria included in the structure of the medical procedure. The processor associates each of a plurality of portions of the ongoing recording with a respective event based on the comparison, and displays, during the ongoing medical procedure, guidance for performing the medical procedure based on the association between events and recording portions.

Description

AUTOMATIC GUIDING FOR HUMAN PERFORMANCE OF A MEDICAL PROCEDURE

RELATED APPLICATIONS

This application claims the benefit of priority of U.S. Provisional Patent Application No. 63/296,507 filed on January 5, 2022, and U.S. Provisional Patent Application No. 63/303,518 filed on January 27, 2022 the contents of which are incorporated herein by reference in their entirety.

FIELD AND BACKGROUND OF THE INVENTION

The present invention, in some embodiments thereof, relates to the field of providing guidance to a surgeon performing a medical procedure.

During the performance of a medical procedure, when an emergency or other misfortune happens, a more experienced doctor may be called to help the surgeon. Such help usually requires the surgeon being helped to be able to recollect and admit his or her mistakes that caused the emergency.

SUMMARY OF THE INVENTION

An aspect of some embodiments of the invention includes a computer implemented method of guiding a performance of an ongoing medical procedure of a certain kind carried out, the method comprising using at least one hardware processor for: accessing a data storage device storing a structure of a medical procedure of the certain kind, the structure comprising a plurality of events and identification criteria, wherein each event is associated with a respective one of the identification criteria; comparing each of a plurality of portions of an ongoing recording of the ongoing medical procedure with the identification criteria of the structure of the medical procedure; associating each of a plurality of portions of the ongoing recording with a respective event based on the comparison; and displaying, during the ongoing medical procedure, guidance for performing the medical procedure based on the association between events and recording portions.

In some embodiments, the computer implemented method also includes receiving, during the ongoing medical procedure, a request to play a portion of the ongoing recording associated with a specified event, wherein the displaying the guidance comprises displaying the portion of the ongoing recording associated with the specified event. In some embodiments, the computer implemented method also includes determining that a problem occurred during the ongoing medical procedure, and displaying an indication to an event that occurred before the occurrence of the problem was determined.

In some embodiments, the structure comprises one or more events, each tagged with an interaction tag, and the displayed indication is an indication to an event tagged with an interaction tag.

In some embodiments, the receiving the request is from a remote user, and the displaying comprises displaying to the remote user.

In some embodiments, the structure of the medical procedure comprises an order of the events, and displaying the guidance comprises displaying an indication to a deviation between the order of the events in the ongoing medical procedure and the order of the events in the structure of the medical procedure.

In some embodiments, the structure of the medical procedure comprises associations between each of a plurality of the events and a respective change in a signal monitored from the patient, and displaying the guidance comprises displaying an indication that a change in the signal recorded in the ongoing recording is different from the change in the signal associated with the event in the structure of the medical procedure.

In some embodiments, the ongoing recording includes synchronized recordings of at least two of: actions of one or more members of an operating staff carrying out the operation; images of an inner body part of the patient being affected by the actions of the operating staff; and signals that are indicative to the reaction of the patient to the actions of the operating staff, including at least one of heart rate, blood pressure, ECG, EEG.

In some embodiments, the guidance comprises synchronized display of at least two of actions of one or more members of an operating staff carrying out the operation;; images of an inner body part of the patient; and signals that are indicative to the reaction of the patient to the actions of the operating staff, including at least one of heart rate, blood pressure, ECG, EEG.

In some such embodiments, the computer implemented method further comprises receiving a request to play a portion of a library recording of a medical procedure of the certain kind, associated with a specified event, wherein the displaying the guidance comprises displaying a portion of a library recording of amedical procedure of the first kind associated with the specified event.

In any of the above embodiments, the method further comprises receiving a request to play a portion of a library recording of a medical procedure of the first kind, the portion being associated with a specified event, wherein the request specifies the event in relation to the event last associated with a portion of the ongoing recording, wherein the displaying output the guidance comprises displaying a portion of a library recording of a medical procedure of the first kind associated with the specified event.

In some such embodiments, the request specifies the event as the current event, the preceding event, or the next event.

In any of the above embodiments, the events comprise at least one problematic event, associated with an identification criterion, and when a portion of the ongoing recording is associated with a problematic event, the guidance comprises an alert that the problematic event occurred.

In some embodiments, the guidance comprises a suggestion how to cope with the problematic event.

Optionally, the method further comprises receiving a request to play a library recording of the problematic event being successfully dealt with, and the displaying the guidance comprises displaying a portion of a library recording associated with a successful solution to the problematic event.

In some embodiments, the ongoing recording comprises recording of internal communication between an imaging device and a display thereof and/or a monitoring device and a display thereof.

In some embodiments, the ongoing recording comprises video recording of a display of a monitoring device or an imaging device.

Optionally, the comparing includes comparing image processed video recording of monitoring or imaging display with the identification criteria.

In some embodiments, the structure comprises a recommendation associated with a respective event, the recommendation being to act in a certain manner during the present or the following event, and the displaying of the guidance comprises displaying the recommendation.

An aspect of some embodiments of the present invention comprises a system for guiding a performance of an ongoing medical procedure of a given kind carried out, the system comprising: a processor; a digital memory storing a structure of a medical procedure of the given certain kind, the structure comprising a plurality of events and identification criteria, wherein each event is associated with a respective one of the identification criteria, and instructions, that when executed by the processor, causes the processor to compare each of a plurality of portions of an ongoing recording of the ongoing medical procedure with identification criteria of the structure of the medical procedure; associate each of a plurality of portions of the ongoing recording with a respective event based on the comparison; and provide for display, during the ongoing medical procedure, guidance for performing the medical procedure based on the association between events and recording portions.

In some embodiments, the instructions cause the processor to receive, during the ongoing medical procedure, a request to play a portion of the ongoing recording associated with a specified event; and provide for display the portion of the ongoing recording associated with the specified event.

In some embodiments, the system comprises an input configured to receive the request from a remote user, and provide the guidance for display to the remote user.

In some embodiments, the structure of the medical procedure comprises an order of the events, and the instructions cause the processor to provide for display an indication to a deviation between the order of the events in the ongoing medical procedure and the order of the events in the structure of the medical procedure.

In some embodiments, the structure of the medical procedure comprises associations between each of a plurality of the events and a respective change in a signal monitored from the patient, and the instructions cause the processor to provide for display an indication that a change in the signal recorded in the ongoing recording is different from the change in the signal associated with the event in the structure of the medical procedure.

In some embodiments, the ongoing recording includes synchronized recordings of at least two of actions of one or more members of an operating staff carrying out the operation; images of an inner body part of the patient being affected by the actions of the operating staff; and signals that are indicative to the reaction of the patient to the actions of the operating staff, including at least one of heart rate, blood pressure, ECG, EEG.

In some embodiments, the instructions cause the processor to provide for display synchronized display of at least two of actions of one or more members of an operating staff carrying out the operation; images of an inner body part of the patient; and signals that are indicative to the reaction of the patient to the actions of the operating staff, such as heart rate, blood pressure, ECG, EEG.

In some embodiments, the memory stores a library of portions of recordings of medical procedures of the certain kind, each associated with an event, and the instructions cause the processor to provide for display a portion of a library recording of a medical procedure of the certain kind associated with a specified event, in response to receiving a request to play a portion of a library recording of a medical procedure of the certain kind, associated with the specified event.

In some embodiments, the memory stores a library of portions of recordings of medical procedures of the certain kind, each associated with an event, and the instructions cause the processor to provide for display a portion of a library recording of a medical procedure of the certain kind associated with a specified event, in response to receiving a request to play a portion of a library recording of a medical procedure of the certain kind, associated with the specified even, wherein the request specifies the event in relation to the event last associated with a portion of the ongoing recording.

In some embodiments, the request specifies the event as the current event, the preceding event, or the next event.

In some embodiments, the events comprise at least one problematic event, associated with an identification criterion, and when a portion of the ongoing recording is associated with a problematic event, the guidance comprises an alert that the problematic event occurred.

In some embodiments, the instructions cause the processor to provide for display a library recording of the problematic event being successfully dealt with in response to a request to play such a library recording. In some embodiments, the system is configured to record internal communication between an imaging device and a display thereof and/or a monitoring device and a display thereof.

In some embodiments, the system is configured to record video recording of a display of a monitoring device or an imaging device.

In some embodiments, the instructions cause the processor to compare the identification criteria with data obtained by image processing of the video recording of the monitoring or imaging display.

In some embodiments, the structure comprises a recommendation associated with a respective event, the recommendation being to act in a certain manner during the present or the following event, and the instructions cause the processor to provide the recommendation for display together with a recording of the respective event.

In some embodiments, the system further includes a user interface configured to allow a user to send to the processor a request specifying a recording portion to be replayed.

In some embodiments, the system comprises a user interface configured to allow a user to generate a recommendation and associate the generated recommendation with an event in the structure of the medical procedure of the certain kind.

In some embodiments, the system further comprises a dedicated display, and the instructions cause the processor to provide the guidance for display on the dedicated display.

An aspect of some embodiments of the invention includes a system for replaying a portion of an ongoing recording of an ongoing medical procedure of a certain kind carried out in an operating room, the system comprising: a user interface; a display; a memory, storing a structure of a medical procedure of the certain kind, the structure comprising procedure steps, each associated with a respective unique identification criterion; and a processor, configured to execute the following instructions during the ongoing medical procedure: instructions to extract, from each of a plurality of portions of the ongoing recording, a respective identification criterion; instructions to associate, each of the plurality of portions of the ongoing recording, with a respective procedure step based on comparison between the extracted identification criterions and the unique identification criterions included in the stored structure; and instructions to cause the display, upon receiving from the user interface a request to replay a specified procedure step, to replay a portion of the ongoing recording associated with the procedure step specified in the request.

In some embodiments, the system further comprises inputs configured to receive digital data from medical equipment in the operating room, and wherein the ongoing recording comprises recordings of data received from the medical equipment via said inputs during the ongoing medical procedure.

In some embodiments, the processor is configured to control recording of data received via the inputs while causing the display to replay the requested portion of the ongoing procedure.

In some embodiments, the system further comprises inputs configured to receive data from one or more cameras in the operating room, and wherein the ongoing recording comprises recordings of data received from the one or more cameras via said inputs.

In some embodiments, the processor is further configured to time-stamp data received via the inputs according to the time of their receipt, and cause the display to replay simultaneously data received via different inputs and time- stamped as being received at the same time.

In some embodiments, the medical equipment includes at least one member of the group consisting of endoscopic tower, ultrasound system, trans -esophageal echography (TEE) system, electrophysiological mapping system, fluoroscopy system, intracardiac echography (ICE) system, hemodynamics monitoring system, and anesthesia monitoring system

In some embodiments, the structure associates with each of a plurality of procedure steps a respective alert criterion, and the processor is configured to extract safety criterions from the ongoing recording of one of the plurality of procedure steps when said one of the plurality of procedure steps is being performed, compare the extracted safety criterions with the alert criterion stored in association with said one of the plurality of procedure steps, and produce an alert based on the comparison.

In some embodiments, the memory stores an order of the procedure steps, and the processor is configured to execute instruction to extract, from a recording of a procedure step being performed, an identification criterion; identify the procedure step being performed by comparing the extracted identification criterion to the stored identification criterions; and cause an alert if the step identified as being performed is being performed out of the stored order. In some embodiments, the digital data from the medical equipment in the operating room include data sent from a sensor of a monitoring or imaging system to a display of the monitoring or imaging system

An aspect of some embodiments of the invention includes a computer implemented method of parsing a performance of an ongoing medical procedure of a certain kind carried out, the computer implemented method comprising using at least one processor for: accessing a structure of a medical procedure of the certain kind, the structure comprising events and identification criteria, wherein each event is associated with a respective one of the identification criteria; comparing each of a plurality of portions of an ongoing recording of the ongoing medical procedure with identification criteria comprised in the structure of the medical procedure; associating each of a plurality of portions of the ongoing recording with a respective event based on the comparison; and creating an index to a time that a respective event happened during the medical procedure.

In some embodiments, the computer implemented method also includes displaying, during the ongoing medical procedure, the time that a respective event happened during the medical procedure.

Optionally, a plurality of times, each with its respective events are displayed.

In some embodiments, the computer implemented method also includes guidance for performing the medical procedure based on the time that a respective event happened.

In some embodiments, the computer implemented method also includes guidance for performing the medical procedure based on a temporal relationship between the time that a respective event happened and the respective event.

In some embodiments, the computer implemented method also includes determining that a problem occurred during the ongoing medical procedure, and displaying an indication to an event that occurred before the occurrence of the problem was determined.

An aspect of some embodiments of the invention includes a system for parsing a performance of an ongoing medical procedure of a certain kind carried out, the system comprising: a processor; a digital memory storing a structure of a medical procedure of the certain kind, the structure comprising events and identification criteria, wherein each event is associated with a respective one of the identification criteria, and instructions, that when executed by the processor, causes the processor to compare each of a plurality of portions of an ongoing recording of the ongoing medical procedure with identification criteria comprised in the structure of the medical procedure; associate each of a plurality of portions of the ongoing recording with a respective event based on the comparison; and create an index to the time that a respective event happened during the medical procedure.

In some embodiments, the instructions cause the processor to provide for display, during the ongoing medical procedure, guidance for performing the medical procedure.

In some embodiments, the instructions cause the processor to display during the ongoing medical procedure, the time that a respective event happened during the medical procedure.

In some embodiments, the instructions cause the processor to provide for display, during the ongoing medical procedure, a plurality of times, each with its respective events.

In some embodiments, the instructions cause the processor to provide guidance for performing the medical procedure based on the time that a respective event happened.

In some embodiments, the instructions cause the processor to provide guidance for performing the medical procedure based on temporal relationship between the time that a respective event happened and the respective event.

In some embodiments, the instructions cause the processor to determine that a problem occurred during the ongoing medical procedure, and to display an indication to an event that occurred before the occurrence of the problem was determined.

An aspect of some embodiments of the invention includes a computer implemented method of parsing a performance of an ongoing medical procedure of a certain kind carried out, the computer implemented method comprising using at least one processor for: accessing a structure of a medical procedure of the certain kind, the structure comprising events and identification criteria, wherein each event is associated with a respective one of the identification criteria; comparing each of a plurality of portions of an ongoing recording of the ongoing medical procedure with identification criteria comprised in the structure of the medical procedure; associating each of a plurality of portions of the ongoing recording with a respective event based on the comparison; and counting the number of events that happened during the medical procedure.

In some embodiments, the computer implemented method also includes displaying, during the ongoing medical procedure, the number of events that happened during the medical procedure.

Parsing optionally includes breaking down a medical procedure into a plurality of events.

An aspect of some embodiments of the invention includes a system for parsing a performance of an ongoing medical procedure of a certain kind carried out, the system comprising: a processor; a digital memory storing a structure of a medical procedure of the certain kind, the structure comprising events and identification criteria, wherein each event is associated with a respective one of the identification criteria, and instructions, that when executed by the processor, causes the processor to compare each of a plurality of portions of an ongoing recording of the ongoing medical procedure with identification criteria comprised in the structure of the medical procedure; associate each of a plurality of portions of the ongoing recording with a respective event based on the comparison; and count the number of events that happened during the medical procedure.

In some embodiments, the instructions cause the processor to provide for display, the number of events that happened during the medical procedure.

An aspect of some embodiments includes a computer implemented method of guiding a performance of an ongoing medical procedure of a certain kind carried out, the computer implemented method comprising using at least one processor for: feeding each of a plurality of portions of an ongoing recording of the ongoing medical procedure of the certain kind into a machine learning model; obtaining a respective event for each portion as an outcome of the machine learning (ML) model; associating each of the plurality of portions of the ongoing recording with the respective event based on the comparison; and displaying, during the ongoing medical procedure, guidance for performing the medical procedure based on the association between events and recording portions. In some embodiments, the computer implemented method further comprises extracting a plurality of feature from each of the plurality of portions, wherein the plurality of feature represent identification criteria, wherein the plurality of features are fed into the ML model.

In some embodiments, the ML model is trained on a training dataset comprising a plurality of records, each record including a respective portion of a plurality of portions of a historical recording of a medical procedure of the certain kind and a ground truth label of an event.

In some embodiments the historical recording is divided into the plurality of portions

In some embodiments, the record includes a plurality of feature extracted from the respective portion, wherein the plurality of feature represent identification criteria.

Unless otherwise defined, all technical and/or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure pertains. Although methods and materials similar or equivalent to those described herein can be used in the practice or testing of embodiments of the present disclosure, exemplary methods and/or materials are described below. In case of conflict, the patent specification, including definitions, will control. In addition, the materials, methods, and examples are illustrative only and are not intended to be necessarily limiting.

As will be appreciated by one skilled in the art, aspects of the present disclosure may be embodied as a system, method or computer program product. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system” (e.g., a method may be implemented using “computer circuitry”). Furthermore, some embodiments of the present disclosure may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon. Implementation of the method and/or system of some embodiments of the present disclosure can involve performing and/or completing selected tasks manually, automatically, or a combination thereof. Moreover, according to actual instrumentation and equipment of some embodiments of the method and/or system of the present disclosure, several selected tasks could be implemented by hardware, by software or by firmware and/or by a combination thereof, e.g., using an operating system

For example, hardware for performing selected tasks according to some embodiments of the present disclosure could be implemented as a chip or a circuit. As software, selected tasks according to some embodiments of the present disclosure could be implemented as a plurality of software instructions being executed by a computer using any suitable operating system. In some embodiments of the present disclosure, one or more tasks performed in method and/or by system are performed by a data processor (also referred to herein as a “digital processor”, in reference to data processors which operate using groups of digital bits), such as a computing platform for executing a plurality of instructions. Optionally, the data processor includes a volatile memory for storing instructions and/or data and/or a non-volatile storage, for example, a magnetic hard-disk and/or removable media, for storing instructions and/or data. Optionally, a network connection is provided as well. A display and/or a user input device such as a keyboard or mouse are optionally provided as well. Any of these implementations are referred to herein more generally as instances of computer circuitry.

Any combination of one or more computer readable medium(s) may be utilized for some embodiments of the present disclosure. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may also contain or store information for use by such a program, for example, data structured in the way it is recorded by the computer readable storage medium so that a computer program can access it as, for example, one or more tables, lists, arrays, data trees, and/or another data structure. Herein a computer readable storage medium which records data in a form retrievable as groups of digital bits is also referred to as a digital memory. It should be understood that a computer readable storage medium, in some embodiments, is optionally also used as a computer writable storage medium, in the case of a computer readable storage medium which is not read-only in nature, and/or in a read-only state.

Herein, a data processor is said to be "configured" to perform data processing actions insofar as it is coupled to a computer readable memory to receive instructions and/or data therefrom, process them, and/or store processing results in the same or another computer readable storage memory. The processing performed (optionally on the data) is specified by the instructions . The act of processing may be referred to additionally or alternatively by one or more other terms; for example: comparing, estimating, determining, calculating, identifying, associating, storing, analyzing, selecting, and/or transforming. For example, in some embodiments, a digital processor receives instructions and data from a digital memory, processes the data according to the instructions, and/or stores processing results in the digital memory. In some embodiments, "providing" processing results comprises one or more of transmitting, storing and/or presenting processing results. Presenting optionally comprises showing on a display, indicating by sound, printing on a printout, or otherwise giving results in a form accessible to human sensory capabilities.

A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a computer readable medium and/or data used thereby may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations for some embodiments of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). Some embodiments of the present disclosure may be described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

Some embodiments of the present disclosure are herein described, by way of example only, with reference to the accompanying drawings. With specific reference now to the drawings in detail, it is stressed that the particulars shown are by way of example, and for purposes of illustrative discussion of embodiments of the present disclosure. In this regard, the description taken with the drawings makes apparent to those skilled in the art how embodiments of the present disclosure may be practiced.

In the drawings:

FIG. 1 is a schematic, pictorial illustration of an operating room equipped with a system for guiding medical procedure, in accordance with an embodiment of the present invention; FIG. 2 is a flowchart of a method of guiding a performance of an ongoing medical procedure of a given kind, in accordance with some embodiments of the present invention; and

FIG. 3 is a simplified block diagram of a system for guiding performance of an ongoing recording of a medical procedure carried out in an operating room, in accordance with some embodiments of the present invention.

DESCRIPTION OF SPECIFIC EMBODIMENTS OF THE INVENTION

The present invention, in some embodiments thereof, relates to the field of guiding the performance of a medical procedure by a computer program that provides guidance to a surgeon.

An aspect of some embodiments of the invention relates to systems, methods, computing devices, and/or code instructions (stored on a data storage device and executable by one or more processors) for guiding a performance of an ongoing medical procedure, optionally of a certain kind which may be selected from multiple candidate kinds. A processor (optionally of a system that include the processor) may access a structure of a medical procedure of the certain kind. The structure may include events and identification criteria, wherein each event is associated with one of the respective identification criteria. The structure may be implemented as, for example, a set of historical records, a mapping dataset that maps between events and identification criteria, a trained machine learning (ML) model, and the like. The events may be, for example, unique identifications such as text tags, classification categories, and the like. The identification criteria may be one or more features extracted from one or a combination of data, such as images, audio files, sensor measurements, and feeds from medical devices. The processor may compare each of multiple portions of an ongoing recording of the ongoing medical procedure with identification criteria of the structure of the medical procedure. The processor may associate each of the portions of the ongoing recording with a respective event based on the comparison. During the ongoing medical procedure, the processor may display on a screen guidance for performing the medical procedure based on the association between events and recording portions. The guidance may be, for example, determining that a problem occurred during the ongoing medical procedure, and displaying an indication to an event that occurred before the occurrence of the problem was determined. For example, determining excessive bleeding, and displaying a video of a surgeon threading a catheter prior to the detected bleeding. In another example, the guidance may be detecting that an order of events of the ongoing medical procedure deviates from an order of events defined by the structure of the medical procedure, and displaying on a screen an indication of the deviation. In another example, the guidance may be detecting that a change in a signal from a medical device recorded in the ongoing recording that is different from a change in the signal associated with the event in the structure of the medical procedure. For example, the heart rate of the patient exceeded a threshold when the heart rate as defined by the structure is to remain below the threshold.

An aspect of some embodiments of the invention relates to systems, methods, computing devices, and/or code instructions (stored on a data storage device and executable by one or more processors) for replaying a portion of an ongoing recording of an ongoing medical procedure (e.g., of a certain kind) carried out in an operating room The processor may extract, from each of multiple of portions of the ongoing recording, a respective identification criterion. The processor may associate, each of the portions of the ongoing recording, with a respective procedure step based on a comparison between the extracted identification criterions and the unique identification criterions included in the stored structure. Upon receiving from the user interface a request to replay a specified procedure step, a portion of the ongoing recording associated with the procedure step specified in the request is replayed on the display.

An aspect of some embodiments of the invention relates to systems, methods, computing devices, and/or code instructions (stored on a data storage device and executable by one or more processors) for parsing a performance of an ongoing medical procedure, optionally, of a certain kind, being carried out. Parsing optionally includes breaking down a medical procedure into a plurality of events. The processor may access a structure of a medical procedure, which may be of the certain kind. The structure may include events and identification criteria, where each event is associated with a respective identification criteria. Each of the portions of an ongoing recording of the ongoing medical procedure may be compared with identification criteria of the structure of the medical procedure. Each of the portions of the ongoing recording may be associated with a respective event based on the comparison. An index to the time that a respective event happened during the medical procedure may be created and/or the number of events that happened during the medical procedure may be counted.

An aspect of some embodiments of the invention relates to systems, methods, computing devices, and/or code instructions (stored on a data storage device and executable by one or more processors) for guiding the performance of a medical procedure. The guidance may be provided during the procedure, e.g., by suggesting solutions to problematic events when they take place, and/or after the procedure, e.g., in preparing a case summary. In some embodiments, the systems, methods, computing devices, and/or code instructions may indicate past events (prior to problematic events) as candidates for the source of the problematic events. An aspect of some embodiments of the invention relates to systems, methods, computing devices, and/or code instructions (stored on a data storage device and executable by one or more processors) for parsing the performance of a medical procedure. Parsing optionally includes breaking down a medical procedure into a plurality of events. Parsing may include creating an index to the time that a respective event happened during the medical procedure. Parsing may include counting the number of events that happened during the medical procedure.

In some embodiments, the systems, methods, computing device, and/or code instructions also include guidance for performing the medical procedure based on temporal relationship between the time that a respective event happened and the respective event.

At least some embodiments described herein address the technical problem and/or the medical problem of automatically providing guidance for performing a medical procedure, for example, by suggesting solutions to problematic events when they take place, or after the procedure, e.g., in preparing a case summary. At least some embodiments described herein improve the technical of automated guidance, and/or improve the medical field of performing a medical procedure, by automatically providing guidance for performing the medical procedure. At least some embodiments described herein improve upon prior approaches of automatically providing guidance for performing a medical procedure, which relied upon manual guidance provided by a more senior physician. Such approaches are subjective, and require the availability of the more senior physician. At least some embodiments described herein improve upon technical problems and/or medical problems described herein, and/or improve upon the technical fields and/or medical fields described herein, and/or improve upon prior approaches, for providing automated guidance for performing a medical procedure, by accessing a structure of a medical procedure which may be of a certain kind. The structure may include events and/or identification criteria, where each event is associated with a respective one of the identification criteria. Each of multiple portions of an ongoing recording of the ongoing medical procedure may be with identification criteria of the structure of the medical procedure. Each of the portions of the ongoing recording may be associated with a respective event based on the comparison. During the ongoing medical procedure, guidance for performing the medical procedure based on the association between events and recording portions may be displayed on a screen.

In at least some embodiments described herein, the automated guidance for performing the medical procedure based on the association between events and recording portions may increase safety of the patient by preventing or reduce adverse effects, and/or when an adverse effect occurs an approach to minimize the damage to the patient. The automated guidance may improve utilization of scarce trained medical personnel, by avoiding or reducing using the trained medical personnel where the automated guidance is sufficient.

Medical procedures may be of different kinds. Procedures of different kinds may differ from each other, for example, in the goal of the procedure (e.g., replacing a mitral valve, replacing an aortic valve (TAVI - transcatheter aortic valve implantation), occluding an atrial appendage, etc.), characteristics of the patient (e.g., age, sex, body mass index), a protocol to be followed (among various protocols for performing procedures of the same goal for the same patient population), invasiveness (e.g., open surgery, minimally invasive surgery), body part being operated on (e.g., heart, lungs, blood vessels, bones, colon), etc.

The systems, methods, computing devices, and/or code instructions according to the present invention are described herein as guiding and/or parsing a procedure of a given kind. In some embodiments, the systems, methods, computing devices, and/or code instructions include, optionally as a first step, selecting a kind of procedure, e.g., from a list of available kinds. The kind of medical procedure may be determined, for example, automatically (e.g., by code that analyzes images and/or other data of the procedure), obtained from storage (e.g., field stored on a memory indicating the kind), and/or manually entered by a user. The structure may be selected according to the kind of medical procedure, where each kind of medical procedure is associated with a respective structure.

The guidance and/or parsing may be based on knowledge of the structure of the medical procedure. The structure may include events that compose a gold standard performance of the procedure (may also be referred to as steps of the procedure) and/or problematic events that may take place during the procedure. Some examples of problematic events are bleeding, new onset of conduction disturbance (e.g., left bundle branch block), sudden drop in blood pressure, etc. Some examples of the gold standard performance may include the steps in the procedure, for example, introduce guidewire into femoral artery, thread guidewire to left ventricle, pass sheath, position percutaneous valve at target anatomical location, deploy valve by retracting sheath, etc... The events may be manually entered by a user, extracted from guidelines, and/or learned by a machine learning model such as based on key extracted features.

In some embodiments, the structure may also associate one or more of the events with an interaction tag. The structure may include one or more events, each tagged with an interaction tag. The displayed indication is an indication to an event tagged with an interaction tag. Interaction tags are associated, in some embodiments, to events relating to interactions between the medical staff and the patient, known to have a potential for causing problems after the interaction. For example, some medications may have side effects that might adversely affect the procedure, so the event of administering medication may be tagged with an interaction tag. In another example, during cutting tissue of the patient, a blood vessel may inadvertently be cut leading to blood loos, so the event of cutting tissue of the patient may tagged with the interaction tag. The interaction tags may be manually entered by a user, extracted from guidelines, and/or learned by a machine learning model such as based on key extracted features.

The steps may be ordered in the order they are expected to appear in the procedure. In some embodiments, the order is linear. But in other embodiments, depending on the specific kind of procedure, the order may be tree-like, when the procedure includes different steps to take depending on details of the anatomy of the patient, patient’s response to actions made by the operating staff, etc. In some embodiments, events may be included in the structure, but not be part of any order. These may include events related to unintended consequences of actions of the operating staff, whether these actions were according to the protocol or departed from the protocol. These events may include mainly problematic events that may occur at different stages of the procedure. In some embodiments, steps may be performed in parallel, for example, when there are two physicians working on different parts of the body. The order of steps may be manually entered by a user, extracted from guidelines, and/or learned by a machine learning model such as based on key extracted features.

The structure may also include an association of each event (or at least each of a plurality of the events) with a respective identification criterion. The identification criterion is a set of one or more occurrences that occur together during the event. These occurrences may include actions being taken by the operating staff (e.g., starting or stopping X-ray streaming; taking a guidewire, pushing or pulling a catheter into or out of the patient’s body, starting or stopping ultrasound imaging, etc.). Such occurrences are optionally recorded using video cameras, filming the operating staff (e.g., the hands of the operating surgeon) in the operating room. Alternatively or additionally, such occurrences may be recorded using other media, for example, audio recordings of the operating staff talking. The identification criterion may be manually entered by a user, extracted from guidelines, and/or learned by a machine learning model such as based on key features extracted from data (e.g., images, audio, sensor measurements) obtained during the event.

Other occurrences that may make part of identification criterion are changes in images of an inner part of the patient. For example, appearance of a catheter in an X-ray of the heart, movement of a catheter in an ultrasound stream, changes in images made by an endoscope, etc. Such occurrences are optionally recorded using imaging equipment. Changes in images may be detected, for example, by an automated comparison of successive images, detecting motion of objects such as using optical flow, and the like.

A third kind of occurrences that may make part of an identification criterion is change in vital or other signs of the patient, for example, heart rate, blood pressure, ECG, EEG, blood saturation, body temperature, etc. Such occurrences are optionally recorded using monitoring equipment and/or sensors. For example, obtained from output feeds of medical devices monitoring the patient and/or the procedure (e.g., screen captures of a control center, outputs of an ECG machine, etc..) and/or measurements made by sensors such as thermometer, ECG electrodes, automated blood pressure cuff, etc.

Some identification criteria include two or more occurrences, each of a different kind. For example, a feature(s) extracted from a video film of the hands of the operating surgeon, together with a feature(s) extracted from an image of an internal part of the patient’ s body and/or a feature(s) extracted from one or more of the monitored signals, in a combination may form an identification criterion of an event in the structure of the medical procedure.

Extracted features described herein may be defined, for example, manually as hand crafted features, automatically by data analysis code (e.g., edge detection, segmentation of an object, detected motion of an object, keypoints, etc.) and/or automatically learned by a ML model such as a neural network that learns to extract the features according to prelearned weights.

The systems, methods, computing devices, and/or code instructions for guiding and/or parsing the performance of the medical procedure may include comparing each of a plurality of portions of an ongoing recording of the ongoing medical procedure with identification criteria included in the structure. In some embodiments, the ongoing recording may include synchronized recordings of various sources in the operating room, including video cameras, audio sensors, monitoring equipment, sensors, and/or imaging equipment. A computer program may compare these recordings with the available identification criteria, to identify the event now taking place in the operation. The comparison may be done, for example, by a statistical analysis of similarity between the recordings and identification criteria (e.g., above a threshold), by a trained ML model, and the like.

The ongoing recording includes at least two kinds of recordings out of the following three: (i) recordings of actions by the operating surgeon (or other staff members), recorded for example by video cameras, still cameras, audio sensors, and the like; (ii) recordings of images or video streams of an inner part of the patient’ s body, recorded for example, using imaging equipment, such as X-ray, Ultrasound, or electrical field imaging or mapping modality; (iii) recordings of parameters characterizing the physiological state of the patient, recorded for example from monitoring equipment. The latter may include recordings of sensors, such as ECG (body- surface and/or intra-cardiac) by ECG electrodes, EEG by EEG electrodes, heart rate by a hear rate sensor, respiration rate by a respiration rate sensor, blood pressure by a blood pressure sensor, body temperature by a thermometer, oxygen saturation by an oxygen saturation sensor, etc.

The ongoing recording may include video recordings of displays of various imaging and/or monitoring devices. In some embodiments, the data represented on the displays is extracted from the video using image processing and/or trained ML models (e.g., neural networks), and the data obtained from the image processing is compared to the identification criteria, for example, by computing a similarity metric and/or implicitly by the trained ML model. For example, a screen of an ECG monitor may be video recorded, the recording may be processed to digital data representing the ECG readings, which may then be compared to identification of a LBBB to see if it occurred.

Additionally or alternatively, the ongoing recording may include recording of internal communication from a monitoring or imaging unit to a corresponding display, so that the data used to generate the display (e.g., the ECG trace or X-ray image) is captured in favor of the guidance.

In some embodiments, the systems, methods, computing devices, and/or code instructions further include associating each of a plurality of portions of the ongoing recording with a respective event based on the comparison. The association may be done, for example, using a mapping dataset, a key-value set, by creating a record for training the ML model, and/or by the trained ML model. As a result of this association, the ongoing recording is being tagged on the fly, with different parts of the recording being tagged as belonging to different events in the structure of the medical procedure.

Based on this tagging, guidance is provided, by displaying information relevant to the current event in the procedure on a screen and/or playing audio messages on speakers.

For example, when a portion of the ongoing recording is associated with a certain (e.g., first) event, while according to the structure of the procedure, another (e.g., a second or a third) event should have taken place, the guidance may include a warning to this effect. For example, the guidance may include displaying a warning that the certain event is not in order, optionally together with a suggestion that the surgeon should consider taking the second or the third events, instead. The other event may be detected as a suggestion instead of the certain event, for example, by analyzing the order of events in the structure. In another example, when a portion of the ongoing recording is associated with a problematic event, the display may indicate (e.g., in text, video, haptic, or by an audio warning) that the problematic event (e.g., bleeding or left bundle branch block) is occurring, optionally, together with a suggestion how to cope with the problematic event. In some embodiments, the surgeon may request (or the processor and/or system that includes the processor may suggest) replaying a recording of another procedure of the same kind, in which the surgeon successfully dealt with the same problematic event. The portion of the recording may be obtained from another recording of another procedure using the tag indicating the problematic event.

In another example, the guidance may include indicating an interaction event that took place at earlier stages of the medical procedure, and might be the cause of a current problem, or might require extra caution currently or when performing the following actions in the procedure. The interaction event may be identified, for example, by code that detects sequences of events, by code that learns correlations between events, by an ML model that learns from historical records of events, and/or manually defined by a user.

In some embodiments, the guidance may include replaying a portion of the ongoing recording preceding and following the detected problematic event.

In another example, the guidance may take the form of replaying a requested portion of the ongoing recording or of a library recording of an operation of the first kind, while the procedure (and its recording) continue. For example, a user may request replaying a recording of an event that took place in the procedure, by specifying the event name or its place in events order (e.g., the present event, the preceding event, the next event, etc.). The requested recording may be identified according to the tag associated with it, and replayed. Optionally, such replay is on a dedicated display, so the displays of the medical equipment may keep functioning as usual, while the dedicated display shows a past event from the same procedure. The request to replay the past event may be in response to problems encountered later in the procedure. The surgeon may find in the past event a mistake and decide how to correct this mistake to solve the encountered problem. The request may come from the operating surgeon, or from a more experienced surgeon called to help solve the problem. The more experienced surgeon may be in the operating room or in a remote site (e.g., at a different room, building, city, or country).

Display of a recording may include a display of temporal development of video and/or audio recordings of staff in the operating room, intra-body imaging, and/or physiological parameters obtained by sensors and/or other monitoring devices. In some embodiments, two or more types of the aforementioned three types are displayed, in synchronized manner. The synchronization is expressed in that all the displayed views (received from cameras, imaging, and/or monitoring equipment) are of occurrences that occurred at substantially the very same time (e.g., within a time window). This way, a review of the display allows areviewerto see what was presented to the doctor in real time. The synchronization may be achieved by timestamping the ongoing recordings of the various signals from the various sources, according to the time at which the data was collected. The time resolution may be, for example, of 10, 20, 50, or 100 msec.

Display of a recording may include overlay annotations on the source signals (e.g. on Fluro images, echo images etc.). Optionally, the overlay annotations may include indications and/or tagging of anatomical structures within the image.

Display of a recording may be used as teaching material, e.g., to be used in training. For example, the recording may be used in training material for teaching the problematic event circumstances or the problematic event consequences. Recording may include the recording preceding and following a detected problematic event.

In some embodiments, the structure of the medical procedure associates one or more of the events with occurrences that don’t make part of the identification criterion characterizing the respective event. In other words, not all occurrences associated with an event should make part of an identification criterion. These occurrences may include, for example, a characteristic movement of the operating surgeon (or other staff members), a characteristic change in a parameter monitored from the patient, and/or a characteristic movement in an image of the inner part of the patient.

Occurrences associated with an event may be characterized by respective ranges, for example, in some procedures, there may be an event associated in the structure with a heart rate increase of up to 10%, blood pressure increase for 1-5 minutes, etc. In some embodiments, if the occurrence does not occur, or occurs out of range, a warning is displayed. In some embodiments, the warning is accompanied with a display of possible causes to the deviation from the expected occurrence and/or suggestions for action to resolve a problem that may be the cause for the deviation of the occurrence from the occurrence expected according to the structure of the medical procedure. The respective ranges may be determined, for example, manually entered by a user, automatically determined by code according to a statistical analysis of historical values, and/or automatically learned by a ML model.

An aspect of some embodiments of the invention includes a system configured to carry out methods described herein. Such a system may include a processor and a digital memory, wherein the processor is configured to execute instructions stored on the digital memory. In some embodiments, such system may be used for training and/or proctoring physicians and/or for other teaching purposes (e.g., clinical education). Such system may be used for remote training and/or proctoring. Optionally, such system may be used for remote clinical support.

In some embodiments, such system may be used for industry support purposes, e.g., may assist in installing new device and/or educating the hospital personal on the new devices.

The digital memory of a system according to an aspect of the present invention may store the structure of the medical procedure of the given kind, and/or code instructions, that when executed by the processor, causes the processor to compare each of a plurality of portions of an ongoing recording of the ongoing medical procedure with identification criteria comprised in the structure of the medical procedure, associate each of a plurality of portions of the ongoing recording with a respective event based on the comparison; and provide for display, during the ongoing medical procedure, guidance for performing the medical procedure based on the association between events and recording portions.

In some embodiments, further instructions and/or recordings may be stored on the digital memory, as required for carrying out the methods described herein.

As used herein, the term system sometimes refers to the processor within the system The term system and processor are sometimes used interchangeably.

In some embodiments, the system connects to the medical equipment in the operating room For example, the system may have an input for receiving data on its way towards the display of an X-ray imaging device (or any other imaging or monitoring device). This way, the system receives the same data received by the X-ray screen, and can record it directly, rather than decipher the greyscale value of each pixel in each frame of a video recording of that screen. Nevertheless, in some embodiments, video-recording of screens or displays is used (rather than or in addition to connecting to the internal communication of the medical equipment), and these recordings are image-processed for obtaining data that may be used in the comparison to the stored structure of the procedure.

In some embodiments of the invention, the structure includes associations between events, and respective recommendations on how to proceed with the operation, once a certain event has occurred. These recommendations may be displayed on a screen and/or played on speakers automatically upon the occurrence of the event, or, in some embodiments, only if a recommendation is requested. In some embodiments, the system includes a user interface (e.g., graphical user interface (GUI) designed to enable a user to add recommendations. Optionally, the user interface may be designed to enable the user to decide if the recommendation appears automatically, or just in response to a request. Optionally, the user may use the user interface to have the recommendation to appear only to the user that introduced it, or to all the doctors that use the system for guidance in performing the same medical procedure.

In embodiments, there may be more than one user using the system in favor of the same procedure. For example, the surgeon in the operating room may ask a remote physician for help, and the remote physician may request to see a replay of an event in the procedure so far. In some embodiments, the remote user may also add recommendations to the structure of the procedure.

In some embodiments, the system includes a display, dedicated to displaying guidance provided for display by the processor. Such a dedicated display may allow replaying portions of recordings of the ongoing procedure (or any other recording) during the procedure, while the displays of the medical equipment in the operating room keep displaying the ongoing situation in the operation.

An aspect of some embodiments of the invention includes a system that enables its user to perform real-time, in-procedure, quality assessment of medical intervention, for example, by replaying a recording of a former part of the medical intervention, while a later part of the intervention is ongoing. In some embodiments, the user specifies an event to be replayed, and the system identifies a recording portion that records the specified event.

In some embodiments, the quality assessment is performed by the system by comparing recordings made during the medical interventions to pre-stored identification criteria.

The system may connect to digital feeds from existing medical equipment, including, for example, monitoring and imaging systems in the operating room (e.g. Endoscopic tower, ultrasound, TEE, EP mapping systems, Fluoroscopy, ICE, hemodynamics, anesthesia monitoring, etc.), sensors (e.g., thermometer, ECG, EEG, heart rate sensor, blood pressure, oxygen saturation), and video cameras. The system can work with any number of digital feeds from medical equipment, for example, between 2 and 8 feeds. The system can also work with any number of digital feeds from cameras, for example, feeds from between 1 and 4 cameras.

The system records and tags the real time feeds and provides the operator with means to review the recordings made at prior times or at prior events during the operation. Optionally, the system allows reviewing times or events that raise a quality question by the system user-defined rules or past experience.

The operator can review the medical procedure, during the procedure or at the end of it. During this review, the user can identify events as relating to unwanted outcomes and create rules for the system to enhance its real time monitoring abilities when used in future cases. For example, the system may include a user interface that allows the user to associate events with rules, recommendations, comments, or the like, that will be displayed in the next time a similar event (i.e., an event that meets the same identification criterion) happens during a following execution of a medical procedure of the same kind. Alternatively or additionally, the rules and/or associations may be learned by a ML model and/or by other code such as that computes correlations between the events and unwanted outcomes. The user may view the learned correlations, and remove errors, fix the errors, and/or confirm the learned correlations.

In some embodiments, a user can share their system enhancement with other users, so each user can benefit from the work of others. Alternatively or additionally, users may upload their enhancements to a server, and an experienced doctor may review the enhancement and decide if to include them in the systems of other users.

In some embodiments, the system is configured to replay a requested section of a procedure for exam and review while the procedure is still in progress. The section may be requested according to its time (e.g., please let me review the first 5 minutes of the procedure) or an event (e.g., please let me review the crossing of the fossa). In some embodiments, the clinician is able to "rewind" scenes for the exam and review while the various feeds keep recording the operating room and signals arriving from medical equipment in the operating room This function enables the operator to have a “second” look at all the signals (including those that don’t have self- recording features) to support a therapeutic decision. This is a major improvement to current practice where most of the signals are available only in real time and cannot be reviewed in retrospect.

For example, in some embodiments, an operator simultaneously uses hemodynamic monitoring, anesthesia monitoring, and an endoscope. While the endoscope video is conventionally stored, the currently disclosed system is designed to present endoscope video in synchronization with recordings of the other inputs, which conventionally are not stored, or stored without synchronization with each other or with the endoscope view recording. Therefore, the system allows a user to find, for example, what happened during the endoscopic procedure that caused a drop in blood pressure, a finding that would not be possible if only the endoscopic images were available for review, or if the timing of the hemodynamic change were not synchronized with the endoscope video. The system may automatically analyze correlations between the different input data relative to historical correlations between different input data to automatically determine the cause of the change.

There are a variety of uses for this functionality involving reviewing specific procedure steps or clinical events while the procedure is still in progress, including, but not limited to taking immediate corrective action, improving the effectiveness and quality of the treatment, reducing the need for additional follow up procedures and ultimately improving patient outcomes.

In some embodiments, the performance quality of an ongoing medical procedure is assessed by the system by comparing, during the procedure, recordings of the ongoing medical intervention to identification criteria identifying a gold standard performance of a medical intervention of the same kind.

Recording of a medical procedure optionally includes synchronous recordings of one or more video cameras, recording the crew in the operating room The video cameras may include a ceiling-mounted camera, filming the operating table from above, head-mount cameras, filming the operating table from the viewpoint of the surgeon, etc.

Recording of the medical procedure optionally includes synchronous data recording from medical equipment that generates digital data and routinely used in the operating room when a procedure like the ongoing medical procedure takes place. The medical equipment that generates the data may include, for example, endoscopic tower, ultrasound, TEE, EP mapping systems, Fluoroscopy, ICE, hemodynamics, and/or anesthesia monitoring. Alternatively or additionally, the medical equipment includes one or more sensors, for example, thermometers, EEG, ECG, heart rate, oxygen saturation, and the like.

In some embodiments, display screens of such medical equipment are filmed, and these films are replayed by the system as requested. In some embodiments, the films are image- processed to reconstruct the data they displayed, and this data is analyzed, for example, for purpose of comparing with pre-stored identification criteria. Preferably, instead of filming the display screens, in some embodiments, the system connects to the input to the display and collects the data input going to the display of the imaging or monitoring equipment, rather than the output of the display, so that reconstructing the data from films of the display may be omitted.

The system itself may then produce a display of the collected data when required to replay a portion of the operation. The display produced by the system may be shown on another screen (e.g., a screen dedicated for replay by the system), rather than by the display of the recorded medical equipment. This may allow replaying a portion of the procedure that had already been performed, while displaying current stage of the procedure on the displays of the medical equipment.

The recordings may be synchronous in the sense that data generated together by different sources (e.g., cameras and/or pieces of medical equipment) are recorded in a way permitting to replay the recordings of all the sources together, according to the times at which the data was collected. In some embodiments, this is accomplished by timestamping the recordings from all the sources according to the times at which the recorded data was collected. The time may be in resolution of seconds or parts thereof, e.g., 1, 10, 20, 50, or 100 milliseconds.

In some embodiments, the system is trained to identify events in an ongoing recording, for example, by feeding a portion of the recording and/or feeding feature extracted from the recording into the ML model described herein. The events may include, for example, actions taken by the surgeon or another crewmember (e.g., the anesthesiologist), beginning (or end) of stages in operation, common complications, etc.

Optionally, such identification comprises comparing the recording of the ongoing operation with identification criteria, defining how to identify each of the events. The comparison may be done, for example, using a correlation function, the ML model, and/or other approaches described herein.

An aspect of some embodiments of the invention is a method of determining identification criteria for events in recordings of a medical procedure. The method includes receiving a structure of the medical procedure, and a sample of recordings of the medical procedure. The structure includes events composing the medical procedure, and each of the recordings of the medical procedure includes portions tagged as recordings of specific events from the structure. The method further includes extracting features from the recordings, optionally using machine learning models, and/or one or more identification criteria for each of the events.

In some embodiments, the processor generates the sample of the recordings, for example, by recording medical procedures, and identifying each event when it happens. The identification may be by an operating doctor that describes what he is doing. For example, the operating doctor may say, “now I am going to enter the femoral artery at the groin”, “now I am going to do the grasping and clip closure”, etc. These sayings are recorded in sync with the recording of the procedure, and transformed into tags, e.g., using speech to text application. In some embodiments, the system expects the events based on the structure, and asks the operating doctor for reporting (e.g., using a text to speech tool). For example, the system may ask the doctor: “when are you crossing the fossa”, and the operating doctor answers, for example “now” or “in about 5 minutes”. If the answer is “now”, the system tags the current recording as belonging to crossing the fossa. If the answer is “in about 5 minutes”, the system may ask again 5 minutes later.

In some embodiments, rather than being tagged by audio recording of the operating doctor stating in real time what he is doing, the recording of the medical procedure is tagged off line by an expert (or a group of experts), that tag the recording according to their understanding of where in the recording each event starts and where it ends.

In some embodiments, the same system may be used for collecting data for generating the identification criteria and applying the identification criteria for replaying an ongoing procedure. In some embodiments, the system also has machine model learning capabilities that allow extracting identification criteria from recordings.

In some embodiments, there is provided a dedicated system for generating identification criteria for procedures, and a separate system for obtaining identification criteria, recording an ongoing procedure, and replaying parts of the ongoing procedure based on comparison between the ongoing procedure and the identification criteria.

In some embodiments, a group of features that happen together may be considered as an identification criterion. The group of features may include, for example, typical movement of the surgeon hands, an appearance of a typical feature in an ultrasound image, a typical change in blood pressure, etc. This way, the system may automatically tag the ongoing procedure, so replay of the ongoing procedure according to events is enabled without manual analysis of the ongoing recording. Thus, the recording may produce, in real time, a “table of contents” of the recording, wherein each event is listed, and can be replayed, e.g., by selecting it from the list. For example, in some embodiments, the system includes a user interface, allowing a user to request display of the table of contents, and select for replay an item from the table of contents. The system then finds automatically the part of the ongoing recording associated with the item in the table of contents, and displays it. In some embodiments, the operator may also take the recording fast backwards and forward, for example, if the operator wants to see some seconds before and/or after an identified event.

In some embodiments, the events are associated with quality criteria. The quality criteria define a quality level at which an event is performed. In some embodiments, there are two quality levels: acceptable, and problematic. Alternatively, there are more quality levels, e.g., perfect, good, fair, and wrong. The quality levels may depend, for example, on various parameters recorded during the event, such as heart rate change, blood pressure range, a range of time durations, along which the event is expected to end, etc. In some embodiments, these ranges may be expressed as an average and a standard deviation (or other indicator of distribution width) around it. Such quality criteria may allow an operator to search for problematic events or performance failures, and not only for specific events. In some embodiments, the quality criteria are assigned by a trained ML model, trained on a training dataset of data captured during the event labelled with an indication of the quality criteria.

In some embodiments, the system may be used for producing a case summary. In some such embodiments, after a case is done, the user is asking for a case summary, and the system shows him the main events, according to a predetermined event list that should appear in the summary, and addresses the user’s attention to events identified as a being performed at certain quality levels (e.g., problematic, less than good, etc.). The user can replay questionable events, and write in the summary what happened, based on his view of the replay, rather than based on his memory. If the system shows no questionable events, the doctor may summarize that all the stages were properly performed.

Before explaining at least one embodiment of the present disclosure in detail, it is to be understood that the present disclosure is not necessarily limited in its application to the details of construction and the arrangement of the components and/or methods set forth in the following description and/or illustrated in the drawings. Features described in the current disclosure, including features of the invention, are capable of other embodiments or of being practiced or carried out in various ways.

The present invention may be a system, a method, and/or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.

Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction- set- architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks . These computer readable program instructions may also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.

Reference is now made to FIG. 1, which is a schematic, pictorial illustration of an operating room equipped with a system 300 for guiding and/or parsing medical procedure, in accordance with some embodiments of the present invention. Reference is also made to FIG. 2, which is a flowchart 200 of a method of guiding a performance of an ongoing medical procedure of a given kind, in accordance with some embodiments of the invention. Reference is also made to FIG. 3, which is a simplified block diagram of a system for guiding an ongoing recording of a medical procedure carried out in an operating room, in accordance with some embodiments of the present invention.

Referring now back to Fig. 1, system 300 is depicted as being used for guiding and/or parsing a trans-catheter intervention, for example, a minimally invasive structural heart disease intervention (MISHDI), but the system is not limited for use in trans-catheter procedures. A catheter 15 is percutaneously inserted into a living body 17 of a patient lying on a gurney 19. Catheter 15 is controlled and manipulated by operator 70. Catheter 15 is connected to an electro- anatomical navigation unit 20 as the catheter is navigated into a heart 18 of the patient. Examples of electro-anatomical catheter navigation system 20 include the CARTO system (Biosense Webster, Diamond Bar, Calif.) and systems described in International patent application publications Nos. WO2018/ 130974, WO2019/034944, WO2019/035023, and WG/2020/008418.

Electro-cardiac signal (ECS) probe sensors 22 are attached to the surface of patient body 17 near heart 18 in order to receive electro-cardiac signals. Probe sensors 22 exemplify monitoring sensors that monitor physiological state of patient body 17, e.g., by monitoring its ECG (body surface and/or intra-cardiac).

Video camera 75 records the hands of operator 70, and system 300 receives the recording.

In some embodiments, e.g., in embodiments using the CARTO™ system for mapping and navigation, one or more magnetic field generators 26 create a magnetic field through the body of the patient, for use by the CARTO system to track the position of catheter 15 in heart 18.

Some tracking systems may not need magnetic field generators 26, and they may be replaced by other field generators (e.g., electrical field generators and electrodes, as described, for example in the above mentioned International patent application publications).

The local electro-cardiac signals are sampled when the distal tip of catheter 15 locally contacts the heart tissue. The position of the catheter distal tip during tracking is displayed to operator 70 on an output display 60 on an operator terminal 50, and recorded by system 300 in sync with a recording of the local electro-cardiac signals.

In some embodiments, an imaging system 30 is used to obtain an image of the heart. Imaging system 30 is shown to include an imaging source 32, which may use magnetic resonance imaging (MRI), X-ray computed tomography (CT), fluoroscopy or any suitable imaging technique to obtain the heart image. The image of the heart is then digitized and stored in imaging system 30.

System 300 receives a digitized heart image from imaging system 30, ECG signals (and/or other physiological parameters of the patient body) from the local electro-cardiac signals obtained from sensors 22, the catheter location obtained from navigating system 20, and the video recording of the surgeon hands from camera 75. System 300 stamps the heart image data, the local electrocardiac signals data, the catheter location data and the video recording data with synchronized time stamps, and save the obtained recording in its memory (see Fig. 3), so that upon request all recorded data may be replayed in sync.

System 300 is configured to receive a user input from operator 70, indicating a recording of a particular event is to be replayed. Operator terminal 50 comprises display 60 and a user input device (304 in Fig. 3), such as a keyboard, mouse, touchscreen, a microphone connected to a speech to text application, etc. In some embodiments, display 60 is a touch-screen that allows the operator to input requests or other data to system 300. In the figure, display 60 is shown to display heart 18 and a signal 65 received from sensors 22. The operator can choose the desired event to be replayed using a user input device of operator terminal 50. System 300 compares the recording comprising the imaging data, tracking data, and electro-anatomical data, to pre- stored identification criteria, and tags the recording based on that comparison. The operating room also includes one or more displays 90 (one is shown) that display the output from the monitoring, tracking, and imaging systems directly, i.e., not via system 300. One or more displays 90 display the signals received in real time, even when display 60 shows a replay of an event that took place earlier in the procedure. In some embodiments, system 300 keeps recording and tagging the events in the procedure in parallel to replaying an earlier event in request from the surgeon.

Referring now back to Fig. 2, is a flowchart 200 of a method of guiding performance of an ongoing medical procedure of a given kind according to some embodiments of the invention. Flowchart 200 includes block 202, at which a structure of the medical procedure of the given kind may be accessed, e.g., by processor 308 of Fig. 3. The structure includes events and identification criteria, and each event is associated with a respective one of the identification criteria.

Exemplary architectures of the structure, and/or exemplary approaches for creating the structure are described herein.

Optionally, the structure is implemented as one or more ME models. Different ME models may be trained for different kinds of medical procedures. An exemplary approach for training the ML model(s) is now described: Historical recordings of multiple medical procedures of a certain kind are obtained. The medical procedures may be on different subjects (or the same subject), and/or performed by different physicians (or the same physician) and/or in different medical facilities (or the same facility). The historical recordings may be from one or multiple data-streams, which may be synchronized, as described herein.

Each recording (or set of synchronized recordings) may be divided into portions that correspond to events of the medical procedure. The division into portions may be done, for example, manually by a physician such as by marking a beginning and/or an end of each portion such as by selecting frames of a video. The division into portions may be done automatically, for example, by a scene division process that divides videos into scenes, for example, by grouping similar images and/or similar other data into the same scene.

Features, which may represent identification criteria, may be extracted from each recording. The features may be defined manually (e.g., hand crafted feature), may be explicit features (e.g., extracted by image processing code), and/or may be implicit features (e.g., extracted by hidden layers of a neural network).

A training dataset of multiple records may be created. A set of records may be created for each historical medical procedure, where each record includes a respective portion of a historical medical procedure of the certain kind, optionally the features extracted from the respective portion, and a ground truth label of an event.

The records of a set may be arranged in sequence and/or linked to each other such as by pointers. The ML model may learn the sequence, for example, a graph neural network may be fed the sequence of records to learn the sequence of events.

Each record may include additional data which may be ground truth, for example, whether the event is a problem (as described herein), interaction tags, an indication of guidance to provide, and the like, as described herein.

The ML model(s) may be trained on the training dataset.

At block 204, data-streams from one or more imaging and/or monitoring devices and/or other sensors may be received, and may be stored as one or more recordings of the ongoing procedure.

As used herein, the term sensor(s) may refer to imaging and/or navigation devices and/or monitoring devices and/or other sensors. The terms sensor(s), imaging devices, and/or monitoring devices, are sometimes used interchangeably, and/or may represent examples of sources of datastreams. The data-stream includes, for example, one or more of: values of physiological parameters of the patient, monitored by the monitoring device, images displayed on display devices, and/or videos from video cameras recording the medical staff in the operating room, optionally the hands of the operating surgeon, video streams of imaging and/or navigation devices used in the medical procedure, and other data-streams as described herein. The video recording may keep record of actions of the operating staff, the monitoring device(s) may monitor the systemic response of the patient’s body to the actions of the operating staff, and/or the imaging device may record the internal, local, effects of the actions taken by the operating staff. In some embodiments, at least two of the three or more data-streams described herein are included in the received data-stream, and recorded for further use as an ongoing recording of the ongoing procedure. The data-stream may include data captured from internal communications sent from a monitoring device to its display, from an imaging device to its display, or from any other piece of medical equipment to its own display. Alternatively (or additionally), the data-stream may include of video recording data of the displays of the various devices themselves.

As used herein, the terms data-stream and recordings are sometimes used interchangeable to refer to the data obtained from one or more sensors.

Optionally, at block 206 the data-stream may be recorded into an ongoing recording that includes (e.g., all) the data, in formats allowing selecting data for replay and/or replaying the data in sync. Data selected for replay may include data received from all the sources, or data from specified source(s).

At block 208, one or more portions of the ongoing recording of the ongoing medical procedure may be each compared with identification criteria comprised in the structure of the medical procedure. This step may be carried out repeatedly as new data-stream(s) are received.

The comparison may be done, for example, by a statistical analysis of similarity between the recordings and identification criteria (e.g., above a threshold). For example, a correlation between features extracted from a certain portion of the recording(s) and features of the identification criteria is computed by a correlation function. When the correlation value is above the threshold, the certain portion of the recording(s) is associated with the event of the identification criteria. When there are multiple identification criteria, the closest identification criteria to the features extracted from the portion of the recording may be found, for example, by computing a Euclidean distance from the portion of the recording to each identification criteria, and locating the identification criteria with smallest Euclidean distance. The event of the identification criteria with smallest Euclidean distance is assigned to the portion of the recording. The comparison may be done by the trained ML model described with reference to 202. The portion of the recording(s) and/or features extracted from the portion of the recording(s) are fed into the ML model. The ML model generates an outcome of the event and/or identification criteria mapped to the event. Other mapping datasets that map between portions of recording(s) and/or features extracted from the portion of the recording(s), and events (and/or identification criteria), may be used. In some embodiments, features 208 and 210 may be performed in two steps by the ML model that generates an outcome of the identification criteria. The portions of the ongoing recording is then associated with a respective event associated with the identification criteria outcome of the ML model. Alternatively, features 208 and 210 may be performed in a single step by the ML model that generates an outcome of the event. The comparing step of 208 and the associating step of 210 are performed implicitly by the ML model.

At 210, when the compared portion is found to meet an identification criterion for an event in the procedure as defined by the structure, the event may be associated with the compared portion. This way the portion of the recording may be associated with the event. The process of associating portions of a recording (e.g., the ongoing recording) with an event is sometimes referred to herein as tagging the recording, for example, a metadata tag indicating the event is assigned to the portion. In another example, a mapping dataset maps between events (e.g., defined by the structure) and portions.

At block 212, an index to the time that a respective event happened during the medical procedure may be created. The index may provide rapid access to events for which guidance is being provided.

The performance of the ongoing medical procedure may be parsed using the index. The parsing breaks down the medical procedure into multiple events, each of which may be indexed according to time.

During the ongoing medical procedure, the time that a respective event happened during the medical procedure may be displayed on a display. Multiple times may be displayed during the ongoing medical procedure, where each time is displayed with its respective event(s).

At block 214, guidance for performing the medical procedure is displayed based on the tagging, i.e., based on the association between events and recording portions. For example, a recommendation on how to carry out the identified event may be displayed, a warning may be displayed (textually, visually, by audio, etc.) if the current event is identified as a problematic event (e.g., bleeding) or an event that should take place only later in the procedure, etc. Optionally, guidance for performing the medical procedure is based on the time that a respective event happened. The guidance for performing the medical procedure may be based on a temporal relationship between the time that a respective event happened and the respective event. For example, a certain event for which guidance is provided is identified. The time of the certain event is determined. Recordings at the time of the certain event may be presented on a display. For example, when a problem event is detected to have occurred at 9:45 AM, images and/or sensor data captured at 9:45 AM may be presented. Optionally, images and/or sensor data captured around 9:45 AM may be presented; e.g., 1, 2, 3 minutes before and/or after the problem event is detected.

A problematic event may be identified explicitly and/or implicitly. An explicit problematic event may refer to a certain event associated with one or more portions of the recording, where the certain event is defined as the problematic event. For example, a portion of video depicts excessive bleeding from tissue of the patient, and is associated with a problematic event tag. An implicit problematic event may refer to a certain event which is expected, but has not been found. For example, during the medical procedure, a transeptal puncture from the left atrium to the right atrium is expected to be performed i.e., expected event. However, another different event is detected, for example, the physician did something else. The problematic event may be detected as the lack of the expected event occurring.

Optionally, the processor detects that a problem occurred during the ongoing medical procedure, and may display an indication to an event that occurred before the occurrence of the problem was determined. For example, the processor detects that the problem occurred during a transeptal puncture, and display and indication to the event that occurred prior to the transeptal puncture.

In case of the problematic event, a suggested solution may also be provided for display. In some embodiments, the surgeon may ask for the system to play a library recording of a successful solution to the occurrence of the problematic event in the context of the same medical procedure and/or the recording of the successful solution may be played automatically. The system may replay a recording portion from a library of tagged recordings. Optionally, a portion is tagged as a successful solution to the problematic event.

An event identified to occur out of order might be a sign that the surgeon skipped some actions that form part of the gold standard performance of the operation of the given kind. In response to the warning, the surgeon may carry out the skipped steps, for example, after completing the present step, or at any other way that may be suitable for the specific circumstances . The order may be defined, for example, by links and/or maps between events, which may be stored as metadata associated with the structure and/or as a mapping dataset and/or graph associated with the structure. The graph may be traversed as events are identified. When a next node of the graph is not reached by the next event of the next portion, an indication of out of order might be generated.

Optionally, the processor may count the number of events that happened during the medical procedure. The processor may count the number of problematic events that happened during the medical procedure. The processor may count the number of each type of event, where each event is of a certain type (e.g., category). The number of counted events, number of problematic events, and/or number of each type of event may be displayed on a display.

In another example, the system may receive a request (e.g., from the surgeon or from a supervisor, optionally via a GUI) to replay a recording portion associated with an event that already took place. The event may be specified by name (e.g., crossing the fossa) or order in respect to the present event (e.g., preceding event). In some embodiments, the request may beta play a recording of the next event, and the system records in response the following event as have been recorded in the past and saved, for example, in a recording library. Requests to replay events from a library of recorded events in a procedure of the same kind may include specification of the event by name or order, among others.

The request may be sent in response to some unexpected problem that the surgeon is facing. The system then may identify the recording portion based on the tagging, and replay the requested portion. The replay may include at least two kinds of recordings from among recordings of actions of the surgeon and/or other member(s) of the operating staff, (e.g., as reflected in the video recorded data from camera 75) recordings of systemic effects of the actions of the operating staff (e.g., as recorded by sensors 22), and/or recordings indicative to internal effects of the actions of the staff (e.g., as recorded by imaging system 30). The replay may allow, for example, the surgeon to have a second look at how the replayed steps were carried out (e.g., based on video of the surgeon hands), how the body of the patient reacted (e.g., based on monitored signals), what exactly took place at the local of the intervention (e.g., based on images of the intra-body region being intervened), to find a misdeed that might have brought to an unexpected problem

In some embodiments, the system may guide the procedure by selecting past events and displaying to the doctor indications thereto. For example, the system may indicate to the doctor that a problem being faced might be related to an action taken before. For example, if the patient’ s blood pressure is dropping in a manner that deviates from what the structure defines as being allowed, the system may identify this as a problem Additionally or alternatively, the doctor may recognize that some feedback received from the imaging and/or monitoring devices and/or sensors is indicative to a problem, and ask the system, via the user interface, to deal with the present situation as with a problem In response to such a request, or to identification of a problem by the system automatically, the system retrieves from a lookup table and/or mapping data stored in memory 302 (cf. Fig. 3) actions that may have occurred in the medical procedure so far and cause problems of different kinds or the specific problem identified by the system automatically or based on the request. The actions may be treated as identification criteria, and mapped to another event. The other event may be found in a historical record of previous recordings, and/or in the current recording. In some embodiments, the system may open automatically a window showing the recording of the suspected event and/or a slide from that recording. Optionally, the doctor can request seeing the entire recording, for example, based on an impression from the slide. This way, the system may find events that may have caused the present problem, by looking in the lookup table for events associated with that problem, and looking at the ongoing recording of the procedure for one or more of these events. If found, the system may indicate to the user what events are suspected as causing the current problem (e.g., the blood pressure drop). The doctor may select an event from the suspected events, and ask the system to show the recording of a selected event, to see if it indeed might have caused the current problem, and/or should other reasons be looked for. In some embodiments, the system may find a single event and/or select automatically from a plurality of events, and shows this event, optionally, regardless of a request from the doctor.

At 216, features described with reference to 204-214 may be dynamically iterated, optionally during real time, as the data-streams are being received. Iterations may analyze sequential portions of the data-stream, for providing real-time guidance.

Referring now back to Fig. 3, is a simplified block diagram of system 300 for guiding and/or parsing an ongoing recording of a medical procedure carried out in an operating room is presented. System 300 may be designed for replaying portions of the ongoing recordings, and/or displaying recommendations, and/or other indications of guidance, as described herein. In some embodiments, system 300 may be used for training and/or proctoring physicians and/or for other teaching purposes (e.g., clinical education). System 300 may be used for remote training and/or proctoring. Optionally, system 300 may be used for remote clinical support.

System 300 is shown to include a computing device 350, a memory 302, a user interface 304, a display 306, and a processor 308. System 300 may implement the features of the method described with reference to FIG. 2, by one or more processors 308 of a computing device 350 executing code instructions 312 stored on memory 302 (also referred to as a program store).

Computing device 350 may receive data-streams from sensor(s) 356 which may be stored as recordings. Each sensor or set of sensors is installed for monitoring the medical procedure and/or the patient during the ongoing medical procedure.

Examples of data-streams and sensors(s) 356 that generate the data-streams include, for example, videos from video cameras (and/or still images from still cameras), audio data from audio sensors that record sounds, physiological sensors that monitor physiological parameters of the subject(s) (e.g., heart rate, blood pressure, ECG, and the like), and/or recordings from one or more medical devices used for the medical procedure (e.g., anesthesia device delivering anesthesia to the subject, intravenous (IV) pump delivering intravenous, images displayed on a screen of a CT workstation, and the like), and/or imaging sensors (e.g., fluoroscope, CT, Ultrasound, TEE, mapping and/or navigation systems or other imaging devices used during a medical procedure, e.g., in an operating room or imaging room).

Computing device 350 may receive data-streams from sensors 356, for example, directly over a network 358, and/or via a client terminal 360 in local communication with the sensor(s) 356.

Multiple architectures of system 300 based on computing device 350 may be implemented, for example:

In an exemplary centralized architecture, computing device 350 may be implemented as one or more servers (e.g., network server, web server, a computing cloud, a virtual server) that provides services to one or multiple clinical locations, for example, multiple operating rooms, and/or multiple clinics. In such architecture, computing device 350 may receive data- streams from sensor(s) 356 located in different operating rooms for monitoring different ongoing medical procedures, as described herein. Computing device 350 may centrally provide guidance during the ongoing medical procedure to each of the operating rooms. Computing device 350 may be in communication with different client terminals 360 each located in a different operating room, for presenting images on local displays and/or receiving input from different user interfaces used by different users.

In another example, computing device 350 may be implemented as an exemplary localized architecture, for example, for locally providing guidance for an ongoing medical procedure in a certain operating room and/or clinic. Computing device 350 may be implemented as, for example, code running on a local workstation (e.g., catheterization laboratory control station, surgical control station), and/or code running on an external device (e.g., mobile device, laptop, desktop, smartphone, tablet, and the like). Computing device 350 may be in local communication with sensor(s) 356. Computing device 350 may be locally connected to sensor(s) 356, for example, by a cable (e.g., USB) and/or short-range wireless connection and/or via network 358. In the localized implementation, the local computing device 350 may locally analyze the recordings from local sensor(s) 356, and locally provide guidance. Computing device 356 may be installed, for example, in an operating room, ER, ambulances, Cath lab, or any other space in which a medical procedure is taking place.

Computing device 350 may be implemented as, for example, a client terminal, a server, a virtual machine, a virtual server, a computing cloud, a single computer, a group of connected computers, a mobile device, a desktop computer, a thin client, a Smartphone, a Tablet computer, a laptop computer, a wearable computer, glasses computer, and a watch computer.

Processor(s) 308 may be implemented, for example, as a central processing unit(s) (CPU), a graphics processing unit(s) (GPU), field programmable gate array(s) (FPGA), digital signal processor(s) (DSP), and application specific integrated circuits) (ASIC). Processor(s) 308 may include one or more processors (homogenous or heterogeneous), which may be arranged for parallel processing, as clusters and/or as one or more multi core processing units.

Memory 302 may be a digital memory that stores code instructions executable by hardware processor(s) 308. Exemplary memories 302 include a random-access memory (RAM), read-only memory (ROM), a storage device, non-volatile memory, magnetic media, semiconductor memory devices, hard drive, removable storage, and optical media (e.g., DVD, CD-ROM). Memory 302 may store code instructions 312 which implement one or more features (e.g., of methods) described herein.

Computing device 350 may include a data storage device 352 for storing data, for example, a procedure structure repository 310 that stores structures of medical procedures of one or more different kids, an ongoing recording repository 314 that stores ongoing recordings of one or more medical procedures, procedure structure 310, and other data such as trained ML model(s) 354 and/or training dataset for training ML model(s) 354. Data storage device 352 may be implemented as, for example, a memory, a local hard-drive, a removable storage device, an optical disk, a storage device, and/or as a remote server and/or computing cloud (e.g., accessed over network 358). It is noted that code may be stored in data storage device 352, with executing portions loaded into memory 302 for execution by processor(s) 308. Machine learning model(s) 354 may be implemented, for example, as one or combination of: a classifier, a statistical classifier, one or more neural networks of various architectures (e.g., convolutional, fully connected, deep, encoder-decoder, recurrent, graph, combination of multiple architectures), support vector machines (SVM), logistic regression, k-nearest neighbor, decision trees, boosting, random forest, a regressor and the like. ML model(s) 354 may be trained using supervised approaches and/or unsupervised approaches on training dataset(s), for example, as described herein.

Procedure structure repository 310 may be a data structure, comprising procedure steps, each associated with a respective unique identification criterion. The procedure structure may also store information indicative to the order by which the steps take place, for example, as a directional graph of nodes where each node is an event, connected by directed edges indicating the order between events, and associated with metadata indicating the identification criterion of the event, links between events, and/or pointers between events. In some embodiments, executing all the steps stored in structure 310 in the indicated order sums to executing the entire procedure according to a standard protocol, for example, by traversing the nodes of the graph according to the directed edges. In some embodiments, memory 302 may store several structures for executing the procedure according to several respective protocols, and the user may select a protocol by which the user wants to carry out the procedure. The user may indicate their choice of protocol, for example, using user interface 304. In some embodiments, memory 302 stores multiple procedure structures, each for carrying out a different kind of procedure (e.g., one for atrial valve replacement, one for mitral valve repair, one for left atrium occlusion, etc.). In some embodiments, several structures are stored for the same procedure, e.g., for carrying out according to different protocols. The procedure and/or protocol may be selected by a user, e.g., via user interface 304.

In some embodiments, the structure may include optional steps, to be executed only in specific cases. For example, some steps are to be executed only if the patient has a rare anatomy, or some steps are replaced with others according to the anatomy of the patient. In another example, a step is meant to deal with a complication and makes part of the procedure only if the complication occurred. The complication itself may be stored in the structure as an optional step.

In some embodiments, each step is associated with a respective unique identification criterion, according to which a portion of a recording (e.g., a portion of the ongoing recording) may be identified as a recording of the respective step. The criterion may comprise a plurality of characteristics (also referred to herein as features) that the recording portion should have in order to meet the criterion. One of the characteristics may be the identification of the preceding step. For example, an identification criterion may include a requirement that the step to be identified is executed immediately after another specified step. Additionally or alternatively, the criterion may comprise one or more events that should be recorded on the portion of the recording to be identified. These may include, for example, the activation of specified medical equipment (e.g., taking an X-ray scan, an echo scan, etc.), a change in a monitored signal (e.g., raise or drop in blood pressure, change in ECG or EEG, etc.), a particular movement of the surgeon’s hands, or combinations of such events.

In some embodiments, code instructions 312 include instructions to extract, from each of a plurality of portions of the ongoing recording, respective identification characteristics, for example, extracted features. These characteristics may be of the same nature as the characteristics included in the unique identification cri terions stored in the procedure structure. For example, the extracted identification characteristics may include events, such as activation of a specified piece of medical equipment, change in one or more monitored signals, a particular movement of the surgeon hands, etc. Additional exemplary details of extracting features are described herein.

In some embodiments, code instructions 312 also include instructions to associate, each of the plurality of portions of the ongoing recording, with a respective procedure step based on comparison between the extracted identification characteristics and the unique identification criterions included in the stored structure. For example, the identification criterion may be met when a series of events took place in a given order. In such embodiments, processor 308, when executing the instructions, associates the portion of the ongoing recording with the respective step when (e.g., all, or a selected subset of) the events making up the unique identification criterion are found also among the extracted characteristics, and appear in the recording portion in the order specified by the identification criterion. In some embodiments, a recording portion is identified once it is found to include a certain number of the events, even if not all of them. This may allow identifying recordings in cases in which the surgeon skipped an action, or replaced an action with an action not stored in association with the step in the procedure structure.

Finally, code instructions 312 may also include instructions to cause the display to replay a portion of the ongoing recording, optionally upon receiving an appropriate request. The request may be received from a user via user interface 304, and specify a procedure step to be replayed. In response to such a request, processor 308 may search for a portion of the ongoing recording associated with the procedure step specified in the response, and cause the display to replay that portion, provided it was found. In some embodiments, if no recording portion is associated with the step specified in the request (e.g., because the surgeon skipped the specified step), the processor may send a notification that the specified procedure step has not been identified yet.

As may be evident from the above description, processor 308 may associates portions of ongoing recording 314 to procedure steps based on similarity or commonality between identification characteristics extracted from a portion of the ongoing recording to unique identification criterions or the events composing them. In some embodiments, this association takes place in real time, that is, while the procedure is being performed.

In some embodiments, system 300 further includes data interface(s) 320. Data interface(s) 320 are optionally designed to receive digital data from sensors 356 of medical equipment in the operating room, for example, from endoscopic tower, ultrasound system, trans-esophageal echography (TEE) system, electrophysiological mapping system, fluoroscopy system, intracardiac echography (ICE) system, hemodynamics monitoring system, and/or anesthesia monitoring system. As used herein, the terms medical equipment and sensors 356 may be used interchangeably, for example, as sources of data fed into data interface(s) 320. Processor 308 is optionally designed to generate the ongoing recording, by recording data received from the medical equipment via data interface 320.

Optionally, system 300 further includes data interface(s) 322 designed to receive data from one or more cameras (not shown) that films (or snapshots) the operating room, e.g., the hands of the surgeon or other crew members. Processor 308 is optionally designed to generate the ongoing recording, by recording data received from the cameras via data interface(s) 322.

Data interface(s) 320 and/or 322 may be implemented as, for example, one or more of, a wire connection (e.g., physical port), a wireless connection (e.g., antenna), a network interface card, a wireless interface to connect to a wireless network, a physical interface for connecting to a cable for network connectivity, and/or virtual interfaces (e.g., software interface, application programming interface (API), software development kit (SDK), virtual network connection, a virtual interface implemented in software, network communication software providing higher layers of network connectivity).

Computing device 350 may include a network interface 362 for connecting to network 358, for example, one or more of, a wire connection (e.g., physical port), a wireless connection (e.g., antenna), a network interface card, a wireless interface to connect to a wireless network, a physical interface for connecting to a cable for network connectivity, and/or virtual interfaces (e.g., software interface, application programming interface (API), software development kit (SDK), virtual network connection, a virtual interface implemented in software, network communication software providing higher layers of network connectivity).

Network 358 may be implemented as, for example, the internet, a local area network, a virtual network, a wireless network, a cellular network, a local bus, a point-to-point link (e.g., wired), and/or combinations of the aforementioned.

It is noted that data interface(s) 320, data interface(s) 322, and network interface 362 may be implemented as different individual interfaces, and/or one or more combined interfaces.

Computing device 350 may communicate with one or more server(s) 364 over network 358, for example, to obtain other recordings from other sensor(s) via another server, to obtain updated versions of the structure, and the like.

Computing device 350 may include and/or be in communication with one or more physical user interfaces 304 that include provide a mechanism to enter data (e.g., annotation of training dataset, availability of medical resources) and/or view data (e.g., scheduled video stream, scheduling of medical resources) for example, one or more of, a touchscreen, a display, gesture activation devices, a keyboard, a mouse, and voice activated software using speakers and microphone. Display 306 may be integrated with user interface 304, or be a separate device. Display 306 presents displayed data, such as of the guidance, as described herein.

In some embodiments, code instructions 312 may further cause processor 308 to timestamp data received via interfaces 320 and/or 322, so that data recorded together are associated with the same time stamp. For example, the timestamping may be according to the time of data receipt at the processor. With timestamped recordings, processor 308 may be further configured to cause display 306 to replay simultaneously data received via different interfaces (320, 322), when said data are stamped with the same time.

In some embodiments, code instructions 312 may cause processor 308 to generate the ongoing recording by recording data received from sensors 356 (e.g., medical equipment) via interface 320, and optionally also from the camera/s via interface 322. Optionally, instructions 312 may cause processor 308 to keep recording data received via interface 320 while causing the display to replay a requested portion of the ongoing procedure. In other words, according to the aforementioned option, the replay does not require stopping or pausing recording of the procedure.

In some embodiments, structure 310 further includes alert criteria, that when met, the system (optionally, via display 306) generates an alert, e.g. in the form of sound, and/or text and/or visual (e.g., images, video, animation). To that end, structure 310 may associate with each of a plurality of procedure steps a respective alert criterion. To use them, processor 308 may be configured to extract alerting features from the ongoing recording of one of the plurality of procedure steps composing the medical procedure. Optionally, the one of the plurality of procedure steps is being performed, while alerting features are extracted from recording portions already made.

The processor may be further configured to compare the extracted alerting features with the alert criterion stored in association with the procedure steps being performed, and generate an alert when an alerting criterion is met. The comparison of alerting features may be done as described herein with reference to features.

In some embodiments, memory 302 stores an order of the procedure steps, so as to allow identifying a step performed out of order, e.g., too late or too early. Processor 308 may be accordingly configured to execute instruction to extract, from a recording of a procedure step being performed, an identification criterion so as to identify a procedure step; and cause an alert if the step identified as being performed is being performed out of the stored order (e.g., too early, before it had to be performed, or too late), after steps ordered to be performed afterwards are already performed.

In some embodiments, display 306 may be divided to a plurality of display portions (a/k/a windows), and user interface 304 may allow a user (e.g., the surgeon) to control what is shown in each window. For example, the user may control one window to show his hands, another window to show Ultrasound imaging, a third window to show fluoro images, and a fourth window to show the patient’s ECG. In some embodiments, the views shown on the various windows are synchronized, e.g., they all show current data, or they all show data collected at the same time earlier in the procedure. In some embodiments, the user can ask to show at different windows data collected at different times, for example, to review how things developed over time, or to compare current state with earlier state, for example, to visualize results of actions taken after the earlier state. In some embodiments, different windows show recordings that differ from each other by event and modality, for example, showing simultaneously in one window - ultrasound that is being currently recorded, and in another window - ECG recorded during an earlier event.

Thus, in some embodiments, the system determines that a problem occurred during the ongoing medical procedure, and displays an indication to an event that occurred before the occurrence of the problem was determined. In some embodiments, this is accomplished using interaction tags. An interaction tag is a tag associated to an event, during which interaction between staff members and the patient occurred, e.g., when an artery (e.g., the femoral artery) was punctured, when medication was administered, etc. In some embodiments, once a problem is identified (automatically by the system, or by the doctor) the system looks for past events tagged as interaction events, and displays indications to them as suspected events. In some embodiments, there are different kinds of interaction tags, each is identified as being possibly connected to another problem. In some embodiments, an event may be associated with more than one interaction tag, if the event is associated with more than one problem. Optionally, interaction tags may be temporary. For example, if it is known that an event might cause a problem in the first 10 minutes, but not after, the tag associating the event with the problem may be removed by the system 10 minutes after the event took place.

Usage Examples

Actions review by a surgeon

During an Interventional Cardiology TMVR (transcatheter mitral valve replacement) procedure, the doctor feels that the leaflet insertion is not ideal and is not sure that he followed the correct sequence of clip closure steps. The doctor also feels that the grasping has been difficult, and does not want to risk missing the leaflets. To resolve the problem, the surgeon selects at the user interface of the system an event called “grasping and clip closure”. The system finds the corresponding portion of the ongoing recording, and displays to the doctor the views that were on the different screens at the time of leaflet grasping and clip closure. The doctor sees in the Echo images being replayed on a screen that the leaflets were well incorporated in the clip arms and that during closure they remained well embedded. The doctor may then ask to replay the same segment again, this time, concentrating on the head camera input. In this replay, the physician observes that the sequence of pushing the lock line was missed, and concludes that there is a risk of clip detachment. Based on these details revealed by reviewing the ongoing procedure in real time, the physician decides how to proceed, in this example, the physician may decide to redo the clip closure sequence, in order to avoid future complications.

In some embodiments, after the operation is ended, the recording of the ongoing procedure that just ended is automatically deleted, so there will be no possibility to use the recording as evidence against (or in favor) of the physician.

Catch up on procedure steps

During a TAVI (trans-catheter aortic valve implantation) procedure, the system alerts that a questionable event has taken place, for example, an event poorly performed, or an event that is out of its expected order, or an event that should not happen during the procedure. The operating physician, who has very little experience, does not understand the reason for the alert, so the operating physician calls a more experienced doctor to the operating room, or talk to the more experienced doctor remotely. By the time the experienced doctor is in the room or in contact, the patient is hypotensive. The experienced physician asks for the system to replay the questionable event. The system finds, and replays an event, in which during pullback of the pigtail catheter, the sheath was withdrawn, and the physician pushed it without the dilator. The hypotension started a minute later, and became severe 5 minutes later, when the experienced physician came to the operating room The replay may be on a display in the operating room and optionally also in the remote location where the experienced physician is. In the latter case, the portion of the ongoing recording may be sent to a display in the remote location, e.g., via the Internet. The experienced physician diagnoses an intramural hematoma. Protamine is given to the patient, and the patient is stabilized.

In one example, a conduction disturbance event may be detected during the procedure; and a recording of the fluoroscopy video before, during and right after the conduction disturbance event may be displayed to the doctor.

Review of a specific step in the procedure

During a mitral valve repair, after a triangular resection, the valve is not competent at the hydrodynamic test. The operating physician knows that this might happen if the resection was too extensive. Although the physician does not recall any problem with the resection, the physician decides to review it with the help of the system The physician then requests the system to replay the recording of the resection, and finds out and that the resection was not too extensive but left an area of prolapse. As the surgery progressed, one suture was too deep and took a secondary chorda. The physician decides to look under the leaflets to see if there is a suture pulling on a chorda. As a suture pulling on a chorda is found, the surgeon cuts it, and the valve returns to normal function.

Review of an entire procedure

After the procedure ends, the surgeon fills a procedure report using the system to show highlights from the procedure. The report may require the physician to specify if certain events were carried out routinely, and mention any problem or deviation from the protocol that occurred during the procedure. The system may show the surgeon the recording portions of the certain events that must be referred to in the report, and the physician may review, and report that the event was carried out properly (or not, as the case may be). The surgeon may also request the system to show any problematic event that the system identified during the procedure, including events out of order, events accompanied with unexpected changes in patient’s position, or any other event of interest, as the surgeon may define. The surgeon fills in the report based on what the surgeon sees in the review.

General

As used herein with reference to quantity or value, the term “about” means “within ±10% of’.

The terms “comprises”, “comprising”, “includes”, “including”, “having” and their conjugates mean: “including but not limited to”.

The term “consisting of’ means: “including and limited to”.

The term “consisting essentially of’ means that the composition, method or structure may include additional ingredients, steps and/or parts, but only if the additional ingredients, steps and/or parts do not materially alter the basic and novel characteristics of the claimed composition, method or structure.

As used herein, the singular form “a”, “an” and “the” include plural references unless the context clearly dictates otherwise. For example, the term “a compound” or “at least one compound” may include a plurality of compounds, including mixtures thereof.

The words “example” and “exemplary” are used herein to mean “serving as an example, instance or illustration”. Any embodiment described as an “example” or “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments and/or to exclude the incorporation of features from other embodiments.

The word “optionally” is used herein to mean “is provided in some embodiments and not provided in other embodiments”. Any particular embodiment of the present disclosure may include a plurality of “optional” features except insofar as such features conflict.

As used herein the term “method” refers to manners, means, techniques and procedures for accomplishing a given task including, but not limited to, those manners, means, techniques and procedures either known to, or readily developed from known manners, means, techniques and procedures by practitioners of the chemical, pharmacological, biological, biochemical and medical arts.

As used herein, the term “treating” includes abrogating, substantially inhibiting, slowing or reversing the progression of a condition, substantially ameliorating clinical or aesthetical symptoms of a condition or substantially preventing the appearance of clinical or aesthetical symptoms of a condition.

Throughout this application, embodiments may be presented with reference to a range format. It should be understood that the description in range format is merely for convenience and brevity and should not be construed as an inflexible limitation on the scope of descriptions of the present disclosure. Accordingly, the description of a range should be considered to have specifically disclosed all the possible subranges as well as individual numerical values within that range. For example, description of a range such as “from 1 to 6” should be considered to have specifically disclosed subranges such as “from 1 to 3”, “from 1 to 4”, “from 1 to 5”, “from 2 to 4”, “from 2 to 6”, “from 3 to 6”, etc.', as well as individual numbers within that range, for example, 1, 2, 3, 4, 5, and 6. This applies regardless of the breadth of the range.

Whenever a numerical range is indicated herein (for example “10-15”, “10 to 15”, or any pair of numbers linked by these another such range indication), it is meant to include any number (fractional or integral) within the indicated range limits, including the range limits, unless the context clearly dictates otherwise. The phrases “range/ranging/ranges between” a first indicate number and a second indicate number and “range/ranging/ranges from” a first indicate number “to”, “up to”, “until” or “through” (or another such range- indicating term) a second indicate number are used herein interchangeably and are meant to include the first and second indicated numbers and all the fractional and integral numbers therebetween.

Although descriptions of the present disclosure are provided in conjunction with specific embodiments, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art. Accordingly, it is intended to embrace all such alternatives, modifications and variations that fall within the spirit and broad scope of the appended claims.

It is appreciated that certain features which are, for clarity, described in the present disclosure in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features, which are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable subcombination or as suitable in any other described embodiment of the present disclosure. Certain features described in the context of various embodiments are not to be considered essential features of those embodiments, unless the embodiment is inoperative without those elements.

All publications, patents and patent applications mentioned in this specification are herein incorporated in their entirety by reference into the specification, to the same extent as if each individual publication, patent or patent application was specifically and individually indicated to be incorporated herein by reference. In addition, citation or identification of any reference in this application shall not be construed as an admission that such reference is available as prior art to the present invention. To the extent that section headings are used, they should not be construed as necessarily limiting. In addition, any priority documents) of this application is/are hereby incorporated herein by reference in its/their entirety.

Claims

53 WHAT IS CLAIMED IS:

1. A computer implemented method of guiding a performance of an ongoing medical procedure of a certain kind carried out, the method comprising using at least one processor for: accessing a data storage device storing a structure of a medical procedure of the certain kind, the structure comprising a plurality of events and identification criteria, wherein each event is associated with a respective one of the identification criteria; comparing each of a plurality of portions of an ongoing recording of the ongoing medical procedure with the identification criteria of the structure of the medical procedure; associating each of a plurality of portions of the ongoing recording with a respective event based on the comparison; and displaying, during the ongoing medical procedure, guidance for performing the medical procedure based on the association between events and recording portions.

2. The computer implemented method of claim 1, further comprising determining that a problem occurred during the ongoing medical procedure, and displaying an indication to an event that occurred before the occurrence of the problem was determined.

3. The computer implemented method of claim 2, wherein the structure comprises one or more events, each tagged with an interaction tag, and the displayed indication is an indication to an event tagged with an interaction tag.

4. The computer implemented method of any preceding claim, further comprising receiving, during the ongoing medical procedure, a request to play a portion of the ongoing recording associated with a specified event, and the guidance comprising the portion of the ongoing recording associated with the specified event.

5. The computer implemented method of claim 4, wherein the receiving the request is from a remote user, and the displaying comprises displaying to the remote user.

6. The computer implemented method of any preceding claim, wherein the structure of the medical procedure comprises an order of the events, and displaying the guidance comprises 54 displaying an indication to a deviation between the order of the events in the ongoing medical procedure and the order of the events in the structure of the medical procedure.

7. The computer implemented method of any preceding claim, wherein the structure of the medical procedure comprises associations between each of a plurality of the events and a respective change in a signal monitored from the patient, and displaying the guidance comprises displaying an indication that a change in the signal recorded in the ongoing recording is different from the change in the signal associated with the event in the structure of the medical procedure.

8. The computer implemented method of any preceding claim, wherein the ongoing recording includes synchronized recordings of at least two of: actions of one or more members of an operating staff carrying out the operation; images of an inner body part of the patient being affected by the actions of the operating staff; and signals that are indicative to the reaction of the patient to the actions of the operating staff, including at least one of heart rate, blood pressure, ECG, EEG.

9. The computer implemented method of any preceding claim, wherein the guidance comprises synchronized display of at least two of: actions of one or more members of an operating staff carrying out the operation; images of an inner body part of the patient; and signals that are indicative to the reaction of the patient to the actions of the operating staff, including at least one of heart rate, blood pressure, ECG, EEG.

10. The computer implemented method of claim 9, further comprising receiving a request to play a portion of a library recording of a medical procedure of the certain kind, associated with a specified event, wherein the displaying the guidance comprises displaying a portion of a library recording of a medical procedure of the certain kind associated with the specified event.

11. The computer implemented method of any preceding claim, further comprising receiving a request to play a portion of a library recording of a medical procedure of the certain kind, the portion being associated with a specified event, wherein the request specifies the event in relation 55 to the event last associated with a portion of the ongoing recording, wherein the displaying output the guidance comprises displaying a portion of a library recording of a medical procedure of the certain kind associated with the specified event.

12. The computer implemented method of claim 11, wherein the request specifies the event as the current event, the preceding event, or the next event.

13. The computer implemented method of any preceding claim, wherein the events comprise at least one problematic event, associated with an identification criterion, and when a portion of the ongoing recording is associated with a problematic event, the guidance comprises an alert that the problematic event occurred.

14. The computer implemented method of claim 13, wherein the guidance comprises a suggestion how to cope with the problematic event.

15. The computer implemented method of claim 13, further comprising receiving a request to play a library recording of the problematic event being successfully dealt with, and the displaying the guidance comprises displaying a portion of a library recording associated with a successful solution to the problematic event.

16. The computer implemented method of any preceding claim, wherein the ongoing recording comprises recording of internal communication between an imaging device and a display thereof and/or a monitoring device and a display thereof.

17. The computer implemented method of any preceding claim, wherein the ongoing recording comprises video recording of a display of a monitoring device or an imaging device.

18. The computer implemented method of claim 17, wherein the comparing includes comparing image processed video recording of monitoring or imaging display with the identification criteria.

19. The computer implemented method of any preceding claim, wherein the structure comprises a recommendation associated with a respective event, the recommendation being to act 56 in a certain manner during the present or the following event, and the displaying of the guidance comprises displaying the recommendation.

20. A system for guiding a performance of an ongoing medical procedure of a certain kind carried out, the system comprising: a processor; a digital memory storing a structure of a medical procedure of the certain kind, the structure comprising a plurality of events and identification criteria, wherein each event is associated with a respective one of the identification criteria, and instructions, that when executed by the processor, causes the processor to compare each of a plurality of portions of an ongoing recording of the ongoing medical procedure with identification criteria of the structure of the medical procedure; associate each of a plurality of portions of the ongoing recording with a respective event based on the comparison; and provide for display, during the ongoing medical procedure, guidance for performing the medical procedure.

21. The system of claim 20, wherein the instructions further cause the processor to determine that a problem occurred during the ongoing medical procedure, and provide for display an indication to an event that occurred before the occurrence of the problem was determined.

22. The system of claim 21, wherein the structure comprises one or more events, each tagged with an interaction tag, and the displayed indication is an indication to an event tagged with an interaction tag.

23. The system of any of claims 20 to 22, wherein the instructions cause the processor to receive, during the ongoing medical procedure, a request to play a portion of the ongoing recording associated with a specified event, and the guidance comprising providing for display the portion of the ongoing recording associated with the specified event.

24. The system of claim 23, comprising an input configured to receive the request form a remote user, and provide the guidance for display to the remote user.

25. The system of any of claims 23 to 24, wherein the structure of the medical procedure comprises an order of the events, and the instructions cause the processor to provide for display an indication to a deviation between the order of the events in the ongoing medical procedure and the order of the events in the structure of the medical procedure.

26. The system of any of claims 20 to 25, wherein the structure of the medical procedure comprises associations between each of a plurality of the events and a respective change in a signal monitored from the patient, and the instructions cause the processor to provide for display an indication that a change in the signal recorded in the ongoing recording is different from the change in the signal associated with the event in the structure of the medical procedure.

27. The system of any of claims 20 to 26, wherein the ongoing recording includes synchronized recordings of at least two of actions of one or more members of an operating staff carrying out the operation; images of an inner body part of the patient being affected by the actions of the operating staff; and signals that are indicative to the reaction of the patient to the actions of the operating staff, including at least one of heart rate, blood pressure, ECG, EEG.

28. The system of any of claims 20 to 27, wherein the instructions cause the processor to provide for display synchronized display of at least two of actions of one or more members of an operating staff carrying out the operation; images of an inner body part of the patient; and signals that are indicative to the reaction of the patient to the actions of the operating staff, such as heart rate, blood pressure, ECG, EEG.

29. The system of claim 28, wherein the memory stores a library of portions of recordings of medical procedures of the certain kind, each associated with an event, and the instructions cause the processor to provide for display a portion of a library recording of a medical procedure of the certain kind associated with a specified event, in response to receiving a request to play a portion of a library recording of a medical procedure of the certain kind, associated with the specified event.

30. The system of any of claims 20 to 29, wherein the memory stores a library of portions of recordings of medical procedures of the certain kind, each associated with an event, and the instructions cause the processor to provide for display a portion of a library recording of a medical procedure of the certain kind associated with a specified event, in response to receiving a request to play a portion of a library recording of a medical procedure of the certain kind, associated with the specified even, wherein the request specifies the event in relation to the event last associated with a portion of the ongoing recording.

31. The system of claim 30, wherein the request specifies the event as the current event, the preceding event, or the next event.

32. The system of any of claims 20 to 31, wherein the events comprise at least one problematic event, associated with an identification criterion, and when a portion of the ongoing recording is associated with a problematic event, the guidance comprises an alert that the problematic event occurred.

33. The system of claim 32, wherein the guidance comprises a suggestion how to cope with the problematic event.

34. The system of claim 32, wherein the instructions cause the processor to provide for display a library recording of the problematic event being successfully dealt with in response to a request to play such a library recording.

35. The system of any of claims 20 to 34, configured to record internal communication between an imaging device and a display thereof and/or a monitoring device and a display thereof.

36. The system of any of claims 20 to 35, configured to record video recording of a display of a monitoring device or an imaging device. 59

37. The system of claim 36, wherein the instructions cause the processor to compare the identification criteria with data obtained by image processing of the video recording of the monitoring or imaging display.

38. The system of any of claims 20 to 37, wherein the structure comprises a recommendation associated with a respective event, the recommendation being to act in a certain manner during the present or the following event, and the instructions cause the processor to provide the recommendation for display together with a recording of the respective event.

39. The system of any of claims 20 to 38, further comprising a user interface configured to allow a user to send to the processor a request specifying a recording portion to be replayed.

40. The system of any of claims 20 to 39, comprising a user interface configured to allow a user to generate a recommendation and associate the generated recommendation with an event in the structure of the medical procedure of the certain kind.

41. The system of any of claims 20 to 40, further comprising a dedicated display, and the instructions cause the processor to provide the guidance for display on the dedicated display.

42. A system for replaying a portion of an ongoing recording of an ongoing medical procedure of a certain kind carried out in an operating room, the system comprising: a user interface; a display; a memory, storing a structure of a medical procedure of the certain kind, the structure comprising procedure steps, each associated with a respective unique identification criterion; and a processor, configured to execute the following instructions during the ongoing medical procedure: instructions to extract, from each of a plurality of portions of the ongoing recording, a respective identification criterion; instructions to associate, each of the plurality of portions of the ongoing recording, with a respective procedure step based on comparison between the extracted identification criterions and the unique identification criterions included in the stored structure; and 60 instructions to cause the display, upon receiving from the user interface a request to replay a specified procedure step, to replay a portion of the ongoing recording associated with the procedure step specified in the request.

43. The system of claim 42, further comprising: inputs configured to receive digital data from medical equipment in the operating room, and wherein the ongoing recording comprises recordings of data received from the medical equipment via said inputs during the ongoing medical procedure.

44. The system of claim 43, wherein the processor is configured to control recording of data received via the inputs while causing the display to replay the requested portion of the ongoing procedure.

45. The system of any of claims 42 to 44, further comprising inputs configured to receive data from one or more cameras in the operating room, and wherein the ongoing recording comprises recordings of data received from the one or more cameras via said inputs.

46. The system of any one of claims 43 to 45, wherein the processor is further configured to time-stamp data received via the inputs according to the time of their receipt, and cause the display to replay simultaneously data received via different inputs and time- stamped as being received at the same time.

47. The system of any one of claims 43 to 46, wherein the medical equipment includes at least one member of the group consisting of endoscopic tower, ultrasound system, trans-esophageal echography (TEE) system, electrophysiological mapping system, fluoroscopy system, intracardiac echography (ICE) system, hemodynamics monitoring system, and anesthesia monitoring system

48. The system of any one of claims 42 to 47, wherein the structure associates with each of a plurality of procedure steps a respective alert criterion, and the processor is configured to extract safety criterions from the ongoing recording of one of the plurality of procedure steps when said one of the plurality of procedure steps is being performed, compare the extracted safety criterions with the alert criterion stored in association with said one of the plurality of procedure steps, and 61 produce an alert based on the comparison.

49. The system of any of claims 42 to 48, wherein the memory stores an order of the procedure steps, and the processor is configured to execute instruction to extract, from a recording of a procedure step being performed, an identification criterion; identify the procedure step being performed by comparing the extracted identification criterion to the stored identification cri terions; and cause an alert if the step identified as being performed is being performed out of the stored order.

50. The system of any one of claims 44 to 49, wherein the digital data from the medical equipment in the operating room include data sent from a sensor of a monitoring or imaging system to a display of the monitoring or imaging system.

51. A computer implemented method of parsing a performance of an ongoing medical procedure of a certain kind carried out, the method comprising using at least one processor for: accessing a structure of a medical procedure of the certain kind, the structure comprising events and identification criteria, wherein each event is associated with a respective one of the identification criteria; comparing each of a plurality of portions of an ongoing recording of the ongoing medical procedure with identification criteria comprised in the structure of the medical procedure; associating each of a plurality of portions of the ongoing recording with a respective event based on the comparison; and creating an index to a time that a respective event happened during the medical procedure.

52. The computer implemented method of claim 51, further comprising displaying, during the ongoing medical procedure, the time that a respective event happened during the medical procedure.

53. The computer implemented method of claims 51 or 52, further comprising displaying, during the ongoing medical procedure, a plurality of times, each with its respective events. 62

54. The computer implemented method of any of claims 51-53, further comprising providing guidance for performing the medical procedure based on the time that a respective event happened.

55. The computer implemented method of any of claims 51-54, further comprising providing guidance for performing the medical procedure based on a temporal relationship between the time that a respective event happened and the respective event.

56. The computer implemented method of any of claims 51-55, further comprising determining that a problem occurred during the ongoing medical procedure, and displaying an indication to an event that occurred before the occurrence of the problem was determined.

57. The computer implemented method of any of claims 51-56, wherein the structure comprises one or more events, each tagged with an interaction tag, and the displayed indication is an indication to an event tagged with an interaction tag.

58. A system for parsing a performance of an ongoing medical procedure of a certain kind carried out, the system comprising: a processor; a digital memory storing a structure of a medical procedure of the certain kind, the structure comprising events and identification criteria, wherein each event is associated with a respective one of the identification criteria, and instructions, that when executed by the processor, causes the processor to compare each of a plurality of portions of an ongoing recording of the ongoing medical procedure with identification criteria comprised in the structure of the medical procedure; associate each of a plurality of portions of the ongoing recording with a respective event based on the comparison; and create an index to the time that a respective event happened during the medical procedure.

59. The system of claim 58, wherein the instructions cause the processor to provide for display, during the ongoing medical procedure, guidance for performing the medical procedure. 63

60. The system of claims 58 or 50, wherein the instructions cause the processor to display during the ongoing medical procedure, the time that a respective event happened during the medical procedure.

61. The system of any of claims 58-60, wherein the instructions cause the processor to provide for display, during the ongoing medical procedure, a plurality of times, each with its respective events.

62. The system of any of claims 58-61, wherein the instructions cause the processor to provide guidance for performing the medical procedure based on the time that a respective event happened.

63. The system of any of claims 58-61, wherein the instructions cause the processor to provide guidance for performing the medical procedure based on temporal relationship between the time that a respective event happened and the respective event.

64. The system of any of claims 58-63, wherein the instructions cause the processor to determine that a problem occurred during the ongoing medical procedure, and to display an indication to an event that occurred before the occurrence of the problem was determined.

65. A computer implemented method of parsing a performance of an ongoing medical procedure of a certain kind carried out, the method comprising using at least one processor for: accessing a structure of a medical procedure of the certain kind, the structure comprising events and identification criteria, wherein each event is associated with a respective one of the identification criteria; comparing each of a plurality of portions of an ongoing recording of the ongoing medical procedure with identification criteria comprised in the structure of the medical procedure; associating each of a plurality of portions of the ongoing recording with a respective event based on the comparison; and counting the number of events that happened during the medical procedure.

66. The computer implemented method of claim 65, further comprises displaying, during the ongoing medical procedure, the number of events that happened during the medical procedure. 64

67. A system for parsing a performance of an ongoing medical procedure of a certain kind carried out, the system comprising: a processor; a digital memory storing a structure of a medical procedure of the certain kind, the structure comprising events and identification criteria, wherein each event is associated with a respective one of the identification criteria, and instructions, that when executed by the processor, causes the processor to compare each of a plurality of portions of an ongoing recording of the ongoing medical procedure with identification criteria comprised in the structure of the medical procedure; associate each of a plurality of portions of the ongoing recording with a respective event based on the comparison; and count the number of events that happened during the medical procedure.

68. The system of claim 67, wherein the instructions cause the processor to provide for display, the number of events that happened during the medical procedure.

69. A computer implemented method of guiding a performance of an ongoing medical procedure of a certain kind carried out, the method comprising using at least one processor for: feeding each of a plurality of portions of an ongoing recording of the ongoing medical procedure of the certain kind into a machine learning model; obtaining a respective event for each portion as an outcome of the machine learning (ML) model; associating each of the plurality of portions of the ongoing recording with the respective event based on the comparison; and displaying, during the ongoing medical procedure, guidance for performing the medical procedure based on the association between events and recording portions.

70. The computer implemented method of claim 69, further comprising extracting a plurality of feature from each of the plurality of portions, wherein the plurality of feature represent identification criteria, wherein the plurality of features are fed into the ML model. 65

71. The computer implemented method of claim 69, wherein the ML model is trained on a training dataset comprising a plurality of records, each record including a respective portion of a plurality of portions of a historical recording of a medical procedure of the certain kind and a ground truth label of an event.

72. The computer implemented method of claim 71, wherein the historical recording is automatically divided into the plurality of portions.

73. The computer implemented method of claim 71, wherein the record includes a plurality of feature extracted from the respective portion, wherein the plurality of feature represent identification criteria.