ES2761248T3 - Contactless handling of the ablation instrument workstation using depth sensors - Google Patents

Abstract

A system for non-contact management of an ablation instrument workstation comprising: a depth sensor (10) for detecting movement; a motion software module (18) operable to receive the detected motion from the depth sensor (10), derive a gesture based on the detected motion; and a client software module (20) operable to receive the applicable gesture on a client computer (16) at the ablation instrument workstation to perform a task according to the client's logic based on the applicable gesture, characterized because the client software module is operable to allow a specific user to control said task only after pressing a pedal connected to the client's computer, after receiving a voice command, or after receiving a focus gesture, and further operable to filter the gesture to accept an applicable gesture by determining whether said gesture is performed by said specific user by tracking the body to determine whether the movement is within a predicted body location and / or performing facial recognition.

Description

DESCRIPTION

Contactless handling of the ablation instrument workstation using depth sensors FIELD OF THE INVENTION

The present invention relates generally to the contactless handling of operating room devices through the use of depth sensors and, in some situations, voice recognition.

BACKGROUND OF THE INVENTION

In an operating room, the doctor needs to remain sterile; therefore, he or she cannot physically access a computer for assistance during surgery. Accordingly, any tasks the physician wants to perform using computer applications on operating room devices, such as an ablation instrument workstation, must be performed by another. It is necessary for the doctor to be able to operate these devices without physically touching the device or its accessories, for example, its keyboard, screen, mouse, etc. WO 2009/137688 A2, US 2010/013764 A1, US 2010/231509 A1 and WO 2011/123669 A1 disclose the related prior art.

SUMMARY OF THE INVENTION

An inventive system and a computer readable storage device are presented which stores a program of instructions executable by a machine to perform a method for non-contact handling of a work station of the ablation instrument. The system comprises a depth sensor to detect a movement, a motion software module operable to receive the detected movement from the depth sensor, deduce a gesture based on the detected movement and filter the gesture to accept an applicable gesture, and a Manageable client software module to receive the applicable gesture on a client computer at or associated with the ablation instrument workstation to perform a task according to the client's logic based on the applicable gesture.

In one aspect, the task is to change the power settings of one or more electrodes, change the maximum allowable temperature for one or more electrodes, change the ablation mode between unipolar and bipolar, select and deselect electrodes for ablation. In one aspect, the movement is at least one of a head movement, a hand movement, and a body movement, and the gesture is further deduced based on a short history. In one aspect, the system further comprises hardware to make the detected movement an applicable gesture. In one aspect, the applicable gesture is one or more of the detected movement made by a user who is determined to be authorized using facial recognition, and a gesture recognized in accordance with customer logic. In one aspect, the filtering of the gesture is performed by determining whether the gesture is authorized and valid. In one aspect, the system further comprises a client communication module for receiving the application gesture on the client computer. In one aspect, the system further comprises voice recognition that provides voice input to allow the customer to perform the task based on voice input along with the applicable gesture.

A computer-readable storage device that stores a program of instructions executable by a machine to perform a method for contactless handling of an ablation instrument workstation comprises detecting movement using a depth sensor, deducing, using a CPU , a gesture based on the detected movement, filter the gesture to accept an applicable gesture, receive the applicable gesture on a client at the ablation instrument workstation and perform a task according to the client logic based on the applicable gesture .

The invention is defined by the scope of the attached independent claims 1 and 5. Other preferred embodiments are disclosed in the dependent claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be more fully understood from the following detailed description of the embodiments thereof, taken in conjunction with the drawings. Further features, as well as the structure and operation of various embodiments, are described in detail below with reference to the accompanying drawings. In the drawings, similar reference numbers indicate identical or functionally similar elements.

Figure 1 is a block diagram of an embodiment of the system of the invention.

Figure 2 is a block diagram of another embodiment of the system of the invention.

Figure 3 is a flow chart of the method of the invention, stored in a computer readable storage device.

DETAILED DESCRIPTION

An inventive technique for contactless handling of an ablation workstation is presented. According to this new technology, a person can perform the functions of the ablation workstation without touching any computing device, for example a monitor, keyboard, mouse, etc.

As shown in Figure 1, the system in one embodiment comprises various components including a depth sensor 10, a computer or CPU 12, a communication medium 14, and a client computer 16 that is part of the ablation workstation . A motion detection software module 18 may reside on CPU 12. Client computer 16 may control a target software module 20. A communication client module 22 may reside on client computer 16. In one embodiment, the The system may also include voice recognition, such as a microphone or an array of 28 microphones .

The depth sensor 10 typically includes a depth recognition camera that not only records images but also records depth, allowing 3D representation of the recorded images. As a person skilled in the art knows, a depth sensor is connected to a computer that uses the sensor's depth map to deduce movement, for example, a user's hand gestures, postures, and positions. In a medical environment, the user may be the operator of the system, generally a physician. In one embodiment of the present invention, the depth sensor tracks head and hand movements. In another embodiment, the depth sensor tracks not only the hands and head but also the general location of the body, such as movement to and / or away from the computer screen or monitor. In one embodiment, depth sensor 10 can be implemented using modules or middleware such as Microsoft® Kinect, ASUS® Xtion PRO or ASUS® Xtion PRO LIVE. More than one depth sensor can be used in parallel to extend occlusion handling, field of view, and precision. Any sensor, such as a camera that can provide a depth map with sufficient detail in real time, can be adjusted for use with the system of the invention. The depth sensor obtains location information and sends this information to CPU 12 for processing as described below. As is known to one of skill in the art, software and a controller provided by depth sensor 10 can be used to obtain this location information.

The motion detection software module or motion software 18 on CPU 12 obtains location information, such as the locations of the operator's relevant body parts, for example, hands, head, feet, etc., a few times per second, from depth sensor 10. Every movement received and / or location in the body is tracked and saved to a brief history of movement file or data set.

The movement software may include software that uses the detected data as movements of the body parts of the movement software 18 along with the brief history of movement data to deduce more complex movements and process the data in meaningful gestures, such as movement. forward, a circular movement of the hand, or a wink with the head. For example, linear and / or circular movements of the hand are detected and parameters are calculated to quantify these movements. Movement software associates particular attributes, such as the extent of movement, the moment it was captured, its speed and / or duration, etc., with each gesture.

The system can be associated with a specific user by tracking the body, that is, from an initial configuration, tracking the individual movements of a user and ensuring that only the movements of that specific user are responded to. For example, body locations and movements are tracked, and as long as the differences from the locations in the chart above are small enough, the movements are assumed to be from the same person. The system can also be associated with a specific user using face detection and / or recognition with the camera on the depth sensor. The system can use face detection and / or recognition to verify that the body is the one associated with the recognized face in each state or position. These techniques can be combined, for example, using body tracking while movements are small and, when there is uncertainty, using face detection and / or recognition to find the correct user, and then continue as before.

The motion detection software module 18 may also include a filter that can decide which gestures or movements to accept and which should be ignored or filtered as not applicable, eg invalid and / or unauthorized. An invalid movement, for example irrelevant, may be a movement that is not acceptable to the Motion Detection Software Module 18. For example, if a movement is too small or slow to filter out uncontrollable natural movements of the user, then this movement it would be invalid or irrelevant. Unauthorized movement is movement that is not performed by the appropriate user and / or not performed in accordance with predetermined conditions. For example, for safety and / or security, certain movements may require physical input and / or interaction with a corresponding hardware device, such as pressing a pedal during a particular hand movement and / or at any point during the procedure. In one embodiment, when multiple cameras are used and one camera has a full body view of the physician, it could be determined if the specific physician provided the appropriate physical input. This security measure, for example, which requires physical entry in addition to the detected movements, trusts that the people in the room are not trying to alter the system; instead, the security measure provides extra security to prevent critical functions from happening inadvertently, in case the doctor / user makes any gesture by mistake.

Connecting a hardware device to the system can prevent unwanted actions by the system, as stated above. This can be critical in medical software. The hardware device (not shown), for example a foot pedal, could be connected to the client's computer 16 and / or the ablation instrument workstation, which is responsible for receiving and interpreting the actions of the device, for example, pressing the pedal, etc., and use these actions for the client's internal logic. The hardware device could also be connected to the CPU 12 doing motion detection and its software, in which case the status of the device could be sent with the detected movements and / or some movements could be filtered according to the state of the device.

The unfiltered movements are sent through the communication medium 14 to the communication client module 20 on the client computer 16. The communication medium 14 can be a serial, local area network (LAN), wireless LAN (WLAN) , Bluetooth, wireless, etc.

The client computer 16 receives the interpreted movements, for example, the gesture. The target software module 20 determines if the received motion is valid. An invalid move can be a move that is not defined in the target target software module 20. For example, when a client software module is active, a move to the screen can be defined to change the mode to unipolar, but if another client were active, this movement to the screen might not be defined in such a way that this movement would be inappropriate or irrelevant. In addition, an invalid movement may be one that is not allowed unless certain preconditions are met. For example, when a hardware device is connected to the customer's computer, it can allow or validate a move only when performed in conjunction with an action by the hardware device; for example, a wave motion can only be valid when performed while pressing the pedal of a hardware device.

Client logic, for example, Target 20 software module, determines how to respond to each recognized, for example, valid gesture. The target software module 20 is associated with the ablation instrument workstation and customer logic can perform tasks such as changing the power settings, selecting and deselecting electrodes, changing the maximum temperature allowed per electrode, etc. Consequently, the client software can determine, using its logic, that if the gesture is a fist, the client's logic can perform the task of selecting an electrode, and if the gesture is an upward movement of the hand, the logic Customer can perform the task of increasing the temperature of the selected electrode.

To operate the system of the invention, the user must first obtain control and activate contactless operation first. To obtain control, the user makes the system acknowledge that it will send instructions. The system then responds to instructions that, as discussed above, are in the form of different hand gestures and / or head movements and / or other body movements. The control can be released when it is no longer needed or after a period of immobility is detected. Control can be achieved using a focus gesture, for example, a wave or a nod of the head, or using hardware, such as a foot pedal that allows control while pressing, or via a voice command.

The movements detected by the system have no fixed functions. The system associates the meaning to each gesture, pose or movement according to its context according to the function and the way in which the user is currently operating the client computer 16. For example, while the user is operating the client in selection mode , a head movement can change the mode from unipolar to bipolar and vice versa.

During the time contactless control is activated, a visual indication may be shown to the user. Optionally, an audible indication can be provided upon activation and deactivation of the contactless control.

In one embodiment, hands-free operation can be increased with voice input that could be used separately or in conjunction with detected movements. Microsoft® Kinect, for example, has peripheral microphones and can detect the direction where the recognized voice came from. This and / or other voice recognition techniques can be used to capture voice commands and selectively filter user voice commands. Voice commands can be combined with gestures. For example, the user might say "Ablation Instrument Power" and then move their hand up and down to control the power. In another example, the user could say "ablation instrument temperature" and then move their hand up and down to control the temperature settings. Another example could be the user who says "electrodes of the instrument ablation ", and the movement of the hand to the left and to the right would highlight the electrode controls on the screen and a" press forward "will toggle the selection. In another example, the user might say" electrodes of the ablation instrument none / even / odd / all "which will select or deselect the electrodes appropriately. The client computer performs all of the logic in deciding what each move means along with the most recently recognized voice command. In one version of this embodiment, some commands may be performed using voice commands or gestures only, for example, setting the ablation mode can be done simply by saying "bipolar ablation instrument" or "unipolar ablation instrument" without the need for a gesture.

Figure 2 shows an embodiment having both CPU and client computer modules in a workstation 26. The system in the embodiment shown in Figure 2 has depth sensor 10 , voice recognition 24 , software module for motion detection software module 18 and target 20, which have the same functionality as those components in Figure 1. However, in this embodiment, only one workstation 26 and software module 20 is used target contains a customer communication module. The invention is not limited to the configuration of Figure 1 or Figure 2. Multiple CPUs may also be used, each with one or more modules.

Figure 3 is a flow chart of the inventive method. In step S1, the user gains control of the contactless control system, typically by making a movement, a voice command, or pressing a pedal. In step S2, depth sensor 10 detects movement, including change of location, and transmits it to CPU 12. In step S3, the CPU tracks and keeps a brief history of movements and / or changes location. In step S4, the CPU 12 uses the motion software to deduce, from the received motion according to the brief history, a more complex motion or gesture. In steps S5 and S6, filtering is performed. In step S5, it is determined whether the gesture has been performed by the authorized or appropriate user by determining whether the movement is within a intended location and / or by performing facial recognition. If the gesture is determined to be from the authorized user (S5 = YES), processing continues in step S6, where the CPU validates the gesture. If the gesture is valid (S6 = YES), it is sent to the client computer and the additional motion tracking is resumed in step S3. In step S7, the gesture is received in the communication client module on the client computer. If the active target software module recognizes the gesture (S8 = YES), then, in step S9, the client computer performs a function or operation of the device, for example a task, on the ablation instrument workstation according to the logic of the client or the target software module, based on the deduced gesture. These tasks include, but are not limited to, changing the power settings of one or more electrodes, changing the maximum allowable temperature for one or more electrodes, changing the ablation mode (unipolar or bipolar), selecting and deselecting electrodes for the instrument ablation, etc. Once the motion software sends the gesture to the client computer (when S6 = YES), the motion software responds again to the next detected motion and / or body location.

In one embodiment, performing an authorized move also requires determining whether the move is performed in conjunction with hardware like another device. In this embodiment, an authorized movement must include both a complex movement deduced by the software and an additional action, such as pressing a pedal. This authorization can be made after step S5, when S5 = YES, and before step S6.

If the detected movement is not authorized (S5 = NO) or the detected movement is invalid (S6 = NO), processing continues in step S3. If the gesture is not recognized by the client software (S8 = NO), then no task is performed.

The system of the invention advantageously allows tasks to be performed by an operating room device using head tracking and / or other motion detection techniques that allow the doctor or other user of the system to perform certain actions while still using at least one of its hands for catheter navigation. Beneficially, the system can filter and forward gestures only to authorized personnel. This ensures that the system is only controlled by the intended people. Hand gestures and head movements and other actions of other people can be ignored.

Various aspects of the present disclosure may be performed as a program, software or computer instructions incorporated or stored in a computer or usable or machine-readable medium, which causes the computer or machine to perform the steps of the method when executed on the computer processor and / or machine. Also provided is a machine-readable program storage device, for example, a computer-readable medium, which tangibly incorporates a machine-executable instruction program for performing various functionalities and methods described in the present disclosure.

The system and method of the present disclosure can be implemented and executed on a general-purpose computer or a special-purpose computer system. The computer system may be any type of system known or to be known, and may typically include a processor, memory device, storage device, input / output devices, internal buses, and / or a communication interface to communicate with other systems. computer along with communication hardware and software, etc. The system It can also be implemented in a virtual computing system, colloquially known as a cloud.

The computer readable medium is a computer readable storage device, which may be, for example, a magnetic, optical, electronic, electromagnetic, infrared or semiconductor system, apparatus or device, or any suitable combination of the foregoing; however, the computer readable storage device is not limited to these examples. Additional particular examples of the computer readable storage device may include: a laptop floppy disk, a hard drive, a magnetic storage device, a portable compact disk (CD-ROM) read-only memory, a random access memory ( RAM), a read-only memory (ROM), a programmable and erasable read-only memory (EPROM or Flash memory), an electrical connection that has one or more cables, an optical fiber, an optical storage device, or any combination appropriate of the above; however, the computer-readable storage medium is also not limited to these examples. Any tangible medium that may contain or store a program for use by or in connection with an instruction execution execution system, apparatus, or device could be a computer-readable storage device.

The terms "computer system" and "computer network" that may be used in the present application may include a variety of combinations of fixed and / or portable hardware, software, peripherals, and storage computing devices. The computer system may include a plurality of individual components that are networked or otherwise linked to work collaboratively, or may include one or more independent components. The hardware and software components of the computer system of the present application can include and can be included within fixed and portable devices such as desktops, laptops and / or servers, and server networks (cloud). A module can be a component of a device, software, program or system that implements some "functionality", which can be incorporated as software, hardware, firmware, electronic circuits, etc.

The embodiments described above are illustrative examples and the present invention should not be construed as being limited to these particular embodiments. Thus, one skilled in the art can make various changes and modifications without departing from the scope of the invention as defined in the appended claims.

Claims (13)

1. A system for contactless handling of an ablation instrument workstation comprising: a depth sensor (10) to detect movement; a movement software module (18) operable to receive the detected movement from the depth sensor (10), deduce a gesture based on the detected movement; and a client software module (20) operable to receive the applicable gesture on a client computer (16) at the ablation instrument workstation to perform a task according to the client's logic based on the applicable gesture, characterized in that the client software module is operable to allow a specific user to control said task only after pressing a pedal connected to the client computer, after receiving a voice command, or after receiving a focus gesture, and further operable to filter the gesture to accept an applicable gesture by determining whether said gesture is performed by said specific user by tracking the body to determine if the movement is within a intended body location and / or by performing facial recognition. 2. The system according to claim 1, further comprising hardware to make the detected movement the applicable gesture. 3. The system according to claim 1, further comprising a client communication module for receiving the application gesture on the client computer. The system according to claim 1, further comprising voice recognition that provides voice input to allow the client to perform the task based on voice input along with the applicable gesture. 5. A computer readable storage device that stores a program of instructions executable by a machine to perform a method for contactless handling of an ablation instrument workstation, the method comprising: receive an indication that a pedal has been depressed, a voice command has been received, or a focus gesture has been detected to associate the workstation with a specific user; detect movement using a depth sensor; deduce a gesture based on motion detected using a CPU; characterized in that filter the gesture to accept an applicable gesture by determining whether said gesture is performed by said specific user using body tracking to determine if the movement is within a intended body location and / or by performing facial recognition; receive the appropriate gesture on a client at the ablation instrument workstation; and perform a task according to the client's logic based on the applicable gesture. 6. The system according to claim 1 or the computer readable storage device according to claim 5, wherein the task is one of changing the power configuration of one or more electrodes, changing the maximum temperature allowed for one or more electrodes, change the ablation mode, select the one or more electrodes for ablation, and deselect the one or more electrodes for ablation. 7. The system according to claim 1 or the computer readable storage device according to claim 5, wherein the movement is at least one of a movement of the head, a movement of the hand and a movement from the body and the gesture is deduced additionally based on a brief history. 8. The computer readable storage device according to claim 5, wherein the applicable gesture is performed while the hardware is activated. The computer readable storage device according to claim 5, further comprising a step of providing voice input using voice recognition, wherein the embodiment step is made based on voice input along with the gesture applicable. The computer readable storage device according to claim 5, wherein the step of filtering the gesture comprises determining whether the gesture is authorized and valid. The computer readable storage device according to claim 5, wherein the movement is at least one of a head movement, a hand movement and a body movement. 12. The computer readable storage device according to claim 5, wherein the gesture Applicable is one of a movement made while the hardware is activated, and a movement by a user who is determined to be authorized using facial recognition. 13. The computer readable storage device according to claim 5, wherein filtering the gesture comprises determining whether the gesture is authorized and valid.